Offer semantic services alongside AST

[uniqueiniquity]

General organization of changes:

• Added option to parser called project, which takes either a single project file path or an array of project file paths (top-level parser.js)
• If option is given, program(s) are created from their corresponding project files (tsconfig-parser.js)
  • Changes to programs over the lifetime of the ESLint process (i.e. when changes are made to the file being linted with --fix or when ESLint is used as a server, as it is in Visual Studio and Visual Studio Code) are tracked by the TS watch API.
• During the existing TS AST traversal to build the ESTree AST, we build up two maps for a two-way mapping between TS AST nodes and ESTree AST nodes. (maps are built in convert.js, passed up through ast-converter.js to top-level).
  • References to the whole AST are provided in many places in convert.js to avoid having to ensure parent pointers are set in the TS AST (that is to say, for efficiency).
• We return the program corresponding to the file being linted and the maps in addition to the AST (parser.js)

Note that the features added by this PR are entirely enabled or disabled (from the consumer's perspective) by the project option; the fallback behavior is the existing behavior.

I also added test coverage to ensure the two-way mapping is consistent (i.e. ts2es(es2ts(node)) = node) and that the exposed checker is usable as a rule might expect.

Map approach is based on eslint/typescript-eslint-parser#415

[uniqueiniquity]

Will fix the merge conflict on Monday (and push a couple more commits that I don't have access to right now), but the core approach will stay unchanged.

[JamesHenry]

Really exciting stuff, @uniqueiniquity! Thanks again for contributing, @DanielRosenwasser mentioned this might be coming up soon.

I am sorry to have made more work for you when it comes to reconciling conflicts, but I unfortunately don't have scope to work on this during working hours right now, and have to seize any free-time opportunity I get (which happened to be this weekend).

In #25 I have brute force (via a gazillion type assertions) converted the repo to finally use TypeScript for its own source. The idea being that I didn't want to make runtime changes/improvements along the way which might get lost within the sheer size of the PR. There can now be lots of smaller, more focused follow ups to improve the way type information flows through.

I am sure a number of your improvements in this PR can further utilize that now too.

Thanks so much again!

[uniqueiniquity]

That's totally fine! Of course I'm biased, but I'm more than happy to convert this to TypeScript. 😄

[JamesHenry]

@uniqueiniquity Is this ready for review?

[uniqueiniquity]

@JamesHenry yes please!

[JamesHenry]

I think it would be good to keep the parse() function as being as simple as possible:

parse(CODE, OPTIONS): AST

Maybe we could add an alternative method:

parseAndGenerateServices(CODE, OPTIONS): YOUR INTERFACE

(open to suggestions on the name)

Then the opt-in is even more explicit for consumers, and the parse function interface will match what other parsers in this space expose.

We did a similar thing for ESLint: we have parse and parseForESLint exposed in custom parsers.

WDYT?

[j-f1]

You could also put the parser services in the AST.

[JamesHenry]

That would make them harder to serialize though right?

[uniqueiniquity]

I'm fine with adding a second method for explicit opt in.

One thing that I'm not sure about in general is whether or not to always give the ts<->estree maps, since (as it stands) we always create them.
So this could give three levels:

• parse gives ast
• parseAndGenerate w/o project flag gives ast and maps
• parseAndGenerate w/ project flag gives ast, maps, and program

Does it make any sense to set it up that way? The TS AST does make it easier to do some things that don't involve semantic information, like finding the position of keywords within nodes.

[j-f1]

That would make them harder to serialize though right?

You’d just need to add a toJSON method:

JSON.stringify({ foo: { toJSON: () => undefined } })
// => "{}"

Or is that not what you meant @JamesHenry?

[JamesHenry]

I think for this initial PR we could start with just two: all or nothing.

• parse() will be as it is today, intended for consumers like prettier where we want an ESTree-compatible AST as quickly as possible (i.e. skip any work that isn't conversion related)

• parseAndGenerateServices() which exposes everything (ast, maps, and program), intended for consumers like ESLint who want to consume semantic/type information for advanced rule creation.

By doing it this way all the new stuff is additive and we don't have to worry about changing the experience for existing consumers. If the new stuff is not quite right for the advanced use-case, we can then further refine it in follow up PRs.

[uniqueiniquity]

Makes sense to me. I've exposed a separate top-level function called parseAndGenerateServices and made all of the additions (i.e. maps and program) only run if that new function is called and the project option is given.

[JamesHenry]

See: #24 (comment)

[uniqueiniquity]

@JamesHenry I've made the changes you requested, I think. Would love for you to take another look!

[JamesHenry]

Awesome, thanks so much again!

[JamesHenry]

🎉 This PR is included in version 5.1.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀