[Draft] - Tree Sitter Service (POC)

[jasonwilliams]

This PR creates a new contributionPoint called tsGrammars which can be used to load a new tree sitter grammar, similar to AnyCode contribution points.

There is a new service which picks up the grammar for whichever active language and then loads it into the parser, that tokenization instance can then handle tokenizing source code.

Some issues:

Native node tree sitter doesn't seem to work with electron due to this issue: tree-sitter/node-tree-sitter#90

So I've got as far as setting up a new service which instantiates and attempts to load web-tree-sitter, however it's unable to load due to a security policy:

vscode\node_modules\web-tree-sitter\tree-sitter.js:1 Refused to connect to 'vscode/node_modules/web-tree-sitter/tree-sitter.wasm' because it violates the following Content Security Policy directive: "connect-src 'self' https: ws:".

I don't know the codebase well enough to properly configure this and put web-tree-sitter in the right place.

This PR fixes #50140
This PR fixes #216

[emm312]

Any updates on this PR?

[jasonwilliams]

Hey @alexdima do you have any advice?

[alexdima]

@jasonwilliams We have had an exploration done by @aiday-mar for using Tree Sitter as a syntax highlighting engine. For the purpose of the exploration we have been mostly interested in finding out what would be the potential speed / memory savings that Tree Sitter could bring. Please see #161479 for the exploration outcome. To productize this and allow extensions to contribute .wasm files we will first need to figure out what to do with Tree Sitter's parser ABI versioning, which increases every now and then and which breaks our desire to never break vscode extensions.

[jasonwilliams]

In that case I will close this in favour of #161479 @alexdima i don't know too much about the ABI versioning, but are there breaking changes happening a lot?