Videos → The Art of Crafting Codemods

Introduction and Event Details0:00

Okay, ladies and gentlemen, let me tell you a little bit

Welcome and Remote Speaker Setup0:45

after this section, for this section, our speaker, he has his own mission, so he's not present here today with us. Anyway, we will use the technology for everyone of you that he's going to do the live skype from India

to over us, over here at the javascript bangkok.

Event Hashtag and Social Media Information1:07

So right now we're setting up the system a little bit. So let me tell you a little bit about our hashtag for this event, which is jsbangkok. All the photos in our event, please, once you post it on your social network, do not forget to hashtag to jsbangkok. And for more information, update, any information about our events, our next event, our promotions or something like that, please do not forget to add our line, line application,

which is @jsbangkok, the same one,

@, just add the @ sign in the front, @jsbangkok.

Sponsor Acknowledgements3:34

I'd like to say thank you so much for all the sponsors. Without them, this event would never happen in Thailand, in Bangkok, for a JavaScript Bangkok over here. The first one is KBTG, beyond tech capabilities, OOZOU, Bangkok's best digital agency,

ExxonMobil Limited, we are ExxonMobil, ODDS, AppMan, the number one insurance digital partner. We have expanded to three countries and more to come in the future. NEXMO, Event Pop, Next Hop, HotNow, Nextzy,

Skooldio, Thailand's leading tech school, and Seven Peaks Software, international software and design agency in Bangkok.

Thank you for all the sponsors for your great support.

Introduction to the Next Talk: Codemods4:29

Alright, now we are ready for the next topic. It will be an introduction to writing codemods. Why they are good, and what techniques are there. The speaker will share some great experiences with us. Let's listen together. With the topic "The Art of Crafting Codemods". By Mr. Rajasegar Chandran.

A front-end developer from Freshworks Inc. From India. Next section is The Art of Crafting Codemods. Please welcome Mr. Rajasegar Chandran, front-end developer from Freshworks Inc. from India, on the screen!

Remote Speaker Introduction and Greetings5:25

Hi. Hello. Namaskar.

Welcome from Chennai, India. I am Rajasekar Chandran. I'm sorry I couldn't make it to Bangkok. So the organizers were kind enough to let me connect from remote. I'm very honored and privileged to be here with you all, javascript bangkok.

Talk Start5:47

Okay, so let's get started. I'm going to share my screen.

Are you guys able to hear my screen?

Thank you.

What are Codemods and Their Purpose?6:11

So in this talk, I'm going to talk about codemods. I'm going to put an idea into your minds, an idea about codemods, an idea about how you can make large scale refactoring tasks less painful and more effective.

It is said that once the human mind is completely exposed to an idea, it will never regain its original dimensions. At the end of this talk, I hope you all feel confident making large scale changes with your code base.

The tooling and the techniques which we are going to discuss will enable you to do that.

I posted my slides here in this URL tinyurl.com/jsbkcodemods. You can follow along if you want. I'll tweet the URL.

You can follow along if you want.

Speaker Introduction and Background7:16

This is about me. I'm a front-end developer from Freshworks. Freshworks enables platform for small and medium sized businesses to support customers through various channels like chat, caller, etc. If you want to know more about Freshworks, you can message me on Twitter. My DMs are open.

I'm rajasegar on GitHub and rajasegar_c on Twitter. I also write occasionally some blog articles in hangaroundtheweb.com, my personal blog. I'm based out of Chennai, India.

The Problems Codemods Solve7:51

Let's discuss the problems first. We'll discuss little bit about the problems codemods solve for us before diving into the central theme of the discussion. Let's start with the more familiar problem, making changes in code. I think we change code more often as developers. Instead of adding and deleting code, it's part of a daily job.

Example: Simple Code Changes and Regex Issues8:13

Let's start with a small example. A little disclaimer here, I'm going to be showing a lot of code snippets.

This is a simple example. We have a variable declaration, let hello equal to Bangkok, and a function which says hello Bangkok. So I'm going to replace hello with Bangkok. How do I make this change? This is a simple change, actually. Usually I can do it with my find and replace in my code editor. If you want to do it in multiple places or files, you need to write a script for automating. Let's say you take this tool called sed, which is stream editor. This is one of the oldest Unix tools which works with regular expressions. You give regular expressions to identify the pattern, source and destination, and you replace with it. After running the script, this is what I got. As you can see here, the word hello has been changed to Bangkok everywhere.

Limitations of Regular Expressions9:18

Instead of just the variable name, this is the possible side effect that could happen if you're not careful with the regex replacements throughout your code base. This is what happened. I started with the problem and I ended with multiple problems using regular expressions.

The main problem here is with regular expressions, it doesn't work like with context. It doesn't know what context we are going to make the change. The original requirement of replacing hello with Bangkok becomes more specific like I want to replace only the variable name hello with Bangkok. This is a drawback and limitation with regular expressions.

Because regular expressions are low level primitives, doesn't able to capture the semantic information associated with your code like which one is a variable name, which one is a function name, which one is a string literal.

Upgrading Codebases: Types and Examples10:20

Then the next problem I'm going to talk about is upgrading code bases. At a very high level, there are two types of upgrades. First type of upgrade is upgrading to new language syntax and features, like you can upgrade to ES6 arrow function, you can upgrade from callbacks to promises to async/await, those kind of upgrades. The second type of upgrade is upgrading your library or framework to a newer version, like you can upgrade React to the latest version, Vue.js to the latest version, something like that. Let's see an example of these things. The first type of upgrade, upgrading to new language syntax features. Before arrow functions are part of the JavaScript specification, we used to do this. Store a reference to the variable self.

And if you want to use it inside a function,

we use that reference for the context.

Now with arrow functions, we don't have to do this anymore. The parent context, the arrow functions can make use of the parent context. This is an example from Vue.js. If you want to upgrade from Vue 1.x to 2.0,

there is a deprecation with the attached lifecycle. Attached lifecycle hook has been renamed to mounted. Whatever you have to do inside the attached hook, you have to wrap it with the next tick function. These kind of changes are not simply possible with regular expressions because regular expressions are low level primitives.

The Need for Better Tooling11:49

They assume that your source code is just a stream of characters. But instead, we need a better tooling to work with the higher level abstraction than regular expressions. We'll see what the tooling is shortly. So now you know that making changes to code is not just find and replace, is a lot other stuff like you add code, you delete code, you do more, actually.

Reflecting on Code Changes and Efficiency12:15

So it's time to stop and reflect. As it said earlier, regular expressions are low level primitives for just replacing textual representation of your source code. It is not suited for making large scale refactoring or code upgrades or making changes with context.

So on a side note, if you want to do large scale refactoring tasks or a major overhaul of your codebases, okay? The goal of this would be like, how can I do this efficiently? How can I do this with less friction? That is the goal of every project manager or team lead when carrying out a large migration of source code.

This is a book by Chad Fowler called "My Job Went to India: 52 Ways to Save Your Job." Chad Fowler is an acclaimed software developer, musician, and author. He worked with some of the world's largest companies like Facebook, Google, Microsoft. This book is his experience of training and mentoring an offshore development center in Bangalore, India. And later, for some reason, the same book got renamed to "The Passionate Programmer." I think 52 is not a very lucky number for Chad. In this book, Chad discusses three ways to increase the throughput. The first is to get faster people to do the work. The second, get more people to do the work. And the third, automate the work.

But in reality, it is not possible to simply get faster people or add more people to do the work. It seems like a classic mistake in software development. So the best sensible thing to do here is to automate the work.

Introducing Codemods: Automated Large-Scale Refactoring14:05

This is where codemods comes into the picture. Codemods is a tool or library to refactor large scale codebases. You do it with partial automation, with a little human oversight and occasional intervention. So codemods is basically to make large scale changes automated with better tooling than regular expressions.

Applications of Codemods14:28

So what can codemods do? These are some of the major applications of codemods, but they are actually quite capable of doing other things. You can easily update your codebase to fit your coding conventions. Let's say you want to adopt a style guide or coding convention for your codebase. You can do very well with codemods. And if you want to make widespread changes when an API is modified, or if you want to automate large scale refactoring tasks. So most of the time, the codemods will be applicable in this area. Frameworks like ReactJS, VueJS are making use of

codemods for their migration path. Especially EmberJS, which I am working with in my

workplace. Usually we work with large EmberJS codebases. Ember automatically provides migration paths from one version of the framework to other version using codemods. Can use automated tools to automatically update your codebase from one version to the other version by running codemods. And you also take advantage of newer language features, which we discussed earlier, upgrading your codebases to make use of new language features. And if you want to know what codemods are out there and what they are capable of doing, I put together an awesome list. So please check it out. There are a lot of codemods available for many purposes. So please have a look at it.

How Codemods Work: AST Manipulation15:55

And this is a very high level overview of how codemods

work. So the first stage, you'll convert the source code into something called AST. AST is called Abstract Syntax Tree representation. And you manipulate the AST representation of your source code. Then you again convert the AST representation back to source code. This is at a very high level how the transformations take place or codemods work.

What is an AST (Abstract Syntax Tree)?16:21

So what is an AST? AST is basically a tree representation of your source code. Your source code is converted into a tree data structure, and that becomes your AST. Whenever I try to explain AST, people are not able to grasp the idea very clearly. So I'll come up with this analogy often. Even though this is an oversimplification, it works much better and clearer to understand the concept. So AST is basically the DOM for your code. The Document Object Model, which is a tree representation of your markup code, right? So how is the AST created in the first place? Your AST is created by the compiler systems.

How ASTs are Created17:01

Your code passes through stages like lexical analyzer, syntax analyzer.

And at the end of the syntax analysis stage, you get the AST. So this is how the lexical analysis stages work. You give the code, there is something called a scanner which parses the code and converts into a token stream of characters. And later, the tokens are converted into a tree representation using the parser, which is AST. A little disclaimer here, I'm leaving out a lot of details about compilers because they make use of other sophisticated stuff to create ASTs like symbol tables. So this is just a simple overview of how ASTs are created.

AST Representation and Visualization17:42

This is your source code representation, and this is your AST, Abstract Syntax Tree. Usually it is represented in object format. In JavaScript, it is generally represented using JSON notation. And this is the tool which you will be using to visualize or explore your ASTs. Let's have a quick look at the tool.

So let me resize it for you.

Okay. So this is the sample code I've shown you. And this is AST representation. So at the root, you have a file node. This is called a node. And these are children, so child nodes. Under file, you have program, and you have variable declaration, this particular statement. And this is function declaration, which is this one. So inside variable declaration, you have a variable declarator, which you have different type of child nodes like identifier and something which identifies a specific part of your code.

Okay, so we'll come back to this later.

Where are ASTs Used?18:50

So now you know ASTs, right? So where exactly your ASTs are used? ASTs are used in a lot of places, right from syntax highlighting, code completion, static analysis. So without ASTs, your JavaScript tooling is not at all possible. So if you want to build advanced tooling, you need to make use of your ASTs. Like Babel makes use of AST, TypeScript, everything makes use of ASTs. These are some of the tools widely known in the JavaScript ecosystem, making use of ASTs to do their job, like ESLint, Babel, Parcel, your builders, everything.

Codemods Workflow: Parse, Transform, Print19:26

So this is how the codemods work overall. Like as I said earlier, your code is converted into AST and then AST back to code. So these are the different tools coming into play. At the first stage, this is called parse, which is done by a tool called Recast, and then the transform, and then the print. So we'll see that in detail.

JSCodeShift: The De Facto Tool for Codemods19:50

So what is JSCodeShift? JSCodeShift is the de facto tool for running your codemods on your codebases. So you create a transform function and you run it with JSCodeShift. It's created by Facebook and later open-sourced it. This CodeShift is made up of two things. One is called the runner and the other one is called the wrapper. The runner is actually a CLI, command-line interface tool, and the wrapper is a wrapper on top of a library called Recast.

Recast: The AST Transformer20:21

So what you have to do, you have to install the JSCodeShift tool globally or you can install it local, and you pass something called the transforms to the JSCodeShift and the filenames or the file paths. You can also pass in some options. The transforms contains the manipulation logic of your ASTs, the node manipulation logic of your ASTs.

So you can also pass folders to that.

And this is how, why JSCodeShift is very popular or a standard tool for running codemods, because it spawns multiple processes parallelly to apply the transform function throughout your codebase. Like if you have hundreds of files, so JSCodeShift can be capable of spawning multiple processes and running the transform against the files. So and then Recast. Recast is the library which JSCodeShift uses to apply the transformations on the ASTs or abstract syntax trees. Recast is AST to AST transformer. It is also a pretty printer, which means it keeps your source code formatting very intact. It does not change the formatting, something like Prettier JS will do. And it also generates source map.

And this is how the overall architecture of code transformations work. So first stage, we'll parse the code with the Recast and then you get the AST, and then the AST can be manipulated using the JSCodeShift APIs. Like you can find AST nodes, you can create new AST nodes, you can update AST nodes. Once the manipulations are done, you can print them back to the source code. So this is the overall idea or bird's eye view of how the codemod transformations take place.

AST Analogy: DOM and jQuery for Code22:17

So if you don't understand all the stuff I've said previously, so as I said earlier, the AST is your DOM. So I'll come up with more analogies to reinforce the idea. So you inspect the DOM with your browser, right? You inspect the DOM with your browser and you manipulate it with something called jQuery. So similarly, you inspect your ASTs or abstract syntax trees with AST Explorer and you can manipulate the ASTs with JSCodeShift.

So that is the overall idea. If you understand this and then you are an expert in codemods.

Live Codemod Demo in AST Explorer22:50

Okay, so let's see a demo. So let's switch to AST Explorer tab.

Okay. So let me turn on the transform space. So this is the original source code and this is AST, and this is the transform function I was talking about. So the default transform comes with reversing your identifier names. But what we want to replace the variable names, right? So I'll delete this part.

So what we are going to do is we're going to change

the name to something like Bangkok.

See, again we are doing the same mistake, right? We have changed everything to Bangkok. So you have to be careful here. So this context is everything. So what we have find is, we are finding all the identifiers in the code. So you have identifiers here in your variable declarators, right? So and you have identifiers inside your function name also.

So as you can see here, call object.

We have identifiers here also. So what I'm going to do is I'm going to find all the variable declarators instead, instead of finding just identifiers.

So I'll say variable declarators.

And I'm going to find the variable declarators.

So here you can see, inspect the node, the ID and name hello. So I'm going to write Okay.

Let me try it again.

Find variable declarators with the name hello,

rename to Bangkok.

Yep, we got it. So, this is the right API to find your variable declarators. So, instead of finding all the identifiers, you can make use of this API with JSCodeShift. I hope this demo made sense. And then let's switch back to the presentation.

Tools for Easier Codemodding26:20

So, now you know that finding nodes and manipulating nodes is the core part of writing codemods. So, I'm going to introduce a couple of tools which will make your job easier in writing codemods. The first is finding nodes. For this, we're going to make use of tool called AST Finder. So, you can find it here. I'll show a demo of that.

This is how the code will work. Let me resize. Okay, so you give a code expression, source code expression, and it will automatically help you to figure out how you can find it in the AST node. So, this code is basically a call expression with a callee of type member expression, and then it is having parameters like object name and property name foo. Right? So, this will help you to write codemods, so you don't have to remember all these APIs for JSCodeShift because JSCodeShift has got very poor documentation. When you are starting with codemods, it's a lot of things to learn. So, the next tool I'm going to introduce is AST Builder. So, which will help you to build the AST nodes from your source code. So, usually, when we are replacing code, sometimes, most of the times, you need to build new nodes and add it to the AST for your code to work. So, instead, you can make use of this tool. So, this is called AST Builder. So, the same thing, you give the source code as your input, and you know how to build that particular AST node. So, this is an expression statement, and inside the expression statement, you have a call expression and member expression, which is of type identifier foo and bar. So, anything that comes with dot is a member expression. So, this is an identifier, and this is an identifier. foo.bar is a member expression within, inside a call

expression. And then the last tool I'm going to introduce. So far, we have built codemods, right? So, what we find it difficult for me when I started writing codemods is, a lot of the time, I didn't know how AST APIs are, I don't know how the node manipulation APIs work. So, when I tell my junior developers to do the same, they are finding it a little difficult to find the APIs. So, you have to read all the existing codemods and know about these APIs. So, instead, what if you have a tool which will automatically write codemods for you?

Jarvis: Automated Codemod Generation Tool29:00

Right? So, that is the tool called Jarvis. I built it recently. So, it is very beta quality, but it works.

It is a start. So, let me give you a demo. So, this is the tool I was talking about. So, this is the input code, and this is the output code, and automatically the tool will write the codemod for you.

So, as you can see, it automatically finds the call expression with the filter, and it'll just replace it. So, it works with the assumption that if you want to change the node, you have to replace the node. So, let me show the example which we were working,

like replacing the variable name. So, let hello to Bangkok.

I'm sorry.

Yeah, Bangkok.

Okay, so as you can see, it automatically figures out what you have to do. So, that's it.

How Jarvis Works30:23

So, how Jarvis works is it uses two npm packages, AST Node Finder, AST Node Builder, and you have a rule engine.

So, we also have a codemod CLI tool to generate and bootstrap your codemods. You can share it with the outside community by publishing as npm package. I'll just skip through these slides.

Key Takeaways and Conclusion30:45

So, what is the takeaway here is that when you are making large-scale changes, you need to build transformation tools that you can be confident in. So, without breaking things. So, with this tooling and the AST, you can do that. And also, tool-assisted code modifications will help

you to maintain large-scale codebases in a better way. So, that's all from me. Hope I made sense with codemods, and hope I can think about codemods when you are thinking about large-scale code migrations. Thank you. Thank you very much.

Thank you for Rajasegar Chandran from India.

Event Merchandise and Closing Remarks32:43

Before we move up to the next section from our beloved sponsor, I would like to introduce all of you to know more about our special products that we provide you for

this event only. For example, we have like hoodie or parka, we have black t-shirt with the logo of our event, and we have a cap, tumbler, water container, and a sock. And we provide you with a special set too in the special price, special discount. So, if you are interested in buying our products, please contact our product booth on the seventh floor, beside the registration desk.