Abstract syntax trees on Javascript
Maybe one of the most interesting things I did the last year is learn how to modify/transform a bunch of Javascript code parsing it. How it works internally a minifier, parser or compressors is something not common to find information about it.
What is an Abstract syntax tree (AST)?
Abstract syntax trees are data structures widely used in compilers, due to their property of representing the structure of program code. An AST is usually the result of the syntax analysis phase of a compiler.
Exactly, the result of a syntax analysis. So, how this can be apply to Javascript is what we are going to discover now. But AST is not only used strictly with Javascript environments such nodejs or the browser, everything started on Java world long time ago, with Netscape.
Javascript Compilers and Engines
Rhino (javascript engine) was born as part of Nestcape project as Javascript interpreter, then it becomes open source under the hood of Mozilla. But Rhino is mainly a converter from Javascript to Java classes, but aside of that it has different uses, as an interpreter, then is when AST becomes important in our story.
AST is the core of Rhino, and deeply used (forked mostly) Google Closure and YUI Compressor, both use the Java parser internally to analyse, parse and mangle the Javascript code.
Spidermonkey has his own AST implementation and is documented as Parse API, and also it seems Rhino has his own AST implementation as well and this changes based on the JS version and those specific javascript engines might change.
To solve these inconsistencies, not that far way, it started the confluence of these differences on the ESTree organisation, creating (finally) the first AST common specification. Now we are talking ;)
Now we have a small overview about how AST becomes and where is used, let see how to use for our own benefit and the future.
Static Analysis of Javascript Statements
The static analysis has multiple benefits, like replace double quotes by single quotes or display the coverage of your javascript files based on your unit test.
Let’s analyse a basic example:
The whole block represents a VariableDeclaration, it what it is. The VariableDeclarator is the “var” keyword, inside of this node we will found the Identifier which represents the “a”. The tree view visualize walk thought our small sentence in a syntax way.
In AST all is a Node, any element extend from it and search for a specific node we will need to loop the whole tree and search exactly what we need. For instance, if our objective were to replace “a” variable by “b” then we are going to need some tools to modify, travel through and remove or add our original code.
Javascript Tooling
AST can be malleable, we can transform, replace and even remove nodes
Parsers
The following parsers are fully compatible with the ESTree spec and probably the most two popular ones.
Esprima
Esprima is the most popular parsed, heavily used on thousands of projects, probably the most important I know is Istanbul code coverage.
Acorn
Acorn claims to be faster, smaller and prettiest than Esprima. Personally I haven’t tried with it, but it’s the core of ASTExplorer and it seems to be quite good, I haven’t worked with this one personally but I’ll leave here the reference.
Whether Esprima or acorn, both do the work, and they do it very well. Up to you to choose.
Estraverse
Once we have our code in AST mode, we can loop the whole tree an change whatever we want. Estraverse is a tool to iterate through the tree and modify, skip or remove nodes. In the following example we are going to replace and long name Identifier by a short one.
Escodegen
Convert back the AST to code it’s possible with escodegen, after any transformation if the AST still valid will become Javascript again like we saw in the previous example.
Escope
Takes an Esprima syntax tree and returns theanalyzed scopes. It allows us, eg: go to an specific scope and do some transformations, detect all variables in the scipe and their references to other parts of the code, detect child scopes, get function parameters, etc etc.
You can practice in this demo page.
Esrefactor
A tool to locate any identifier in your code, this is very useful if we would want to rename any variable or function.
The current version (on npm) it doesn’t have this patch, so, I managed only to rename variables.
Esmangle
Esmangle is a small minifier, one of many ECMAScript tools, this one in particular is has a interesting set of parameters to change the behaviour.
AST-Types
Esprima has actually an object accessible to compare node by Syntax type, this alternative is more suitable for Unit Testing.
Harmony (ES6)
Before Esprima supported ES6 and RxJS syntax in the after the version 2.0, there was a fork by Facebook called esprima-fb. Actually is not necessary.
There is a AST explorer where you can play around using different configurations.
Conclusion
This article is about to recap some of the most popular/useful tools I found in the learning process about AST, probably there is more (feel free to comment if you have more suggestions or changes), personally I think understand AST is one of the most important steps to comprehend the web these days, topics like WebAssembly, V8, Blink always mention AST as a main intermediary for parsing.
