Allow dynamic location of textmate grammar

[octref]

Ref: vuejs/vetur#210

In Vue files you can have <template>, <style> and <script> tag.

However, certain libraries can add top-level custom blocks that uses another language. For example:

<template></template>
<script></script>
<style></style>

<docs>
# This might be markdown or json, depending on user's build configuration
</docs>

Our approach was:

• Give a setting vetur.grammar.customBlocks
• Register a command to recompile the TextMate grammar with the custom blocks
• Write the file to extension directory, the same path as the one specified in package.json

This has been working, but I'm worried extension integrity check might kick in and ask for reinstall. Also @sandy081 mentioned updates remove the old folder, so it's bad to keep states in extension directory.

What I need is a path that I can:

• Statically determine in write-time (as the path is specified in package.json)
• Have write permission

So the paths in ExtensionContext wouldn't work for me.

After checking with @sandy081, we think offering an API like languages.setGrammar would be best. We considered allowing path interpolations like ${globalStoragePath}/grammars/vue.json in package.json, but that means the best Vetur can do is to write to that path on first activation, so no coloring before first activation/reload.

However, I'm not sure we should provide an API tied to TextMate grammar. Also the API means the dynamically generated TextMate grammar wouldn't be loaded until extension activation.

@alexandrudima What do you think?

[alexdima]

I find it very interesting, I wasn't aware of the tricks you are doing with rewriting your grammar!

I think referring in package.json to a file which doesn't exist until the first extension activation is just asking for trouble. We would need to install a watcher to monitor a specific file/folder and if the file appears or is updated, we would need to recreate the TM grammar... Otherwise coloring would never work until after a restart.

Adding imperative API seems like the way to go, since you could call the API whenever you are ready. But... this would have the disadvantage of not coloring those blocks each and every time until after your extension activates and uses the API...

To be honest, I find both proposals have flaws...

[octref]

coloring would never work until after a restart.

That's actually ok for me, most people don't change the custom blocks frequently.

a file which doesn't exist until the first extension activation

What do you think about adding a setting and putting the grammar to user repo? When no grammar is found, Vue files fallback to using grammar bundled in extension.

Also, custom blocks tie to specific dependencies. For example, people use <docs> if they use vue-18n, and people use <page-query> with GraphQL if they are using Gridsome. So I think it's natural to colocate the setting, grammar files and dependencies in user repos.

[danixeee]

@alexandrudima What is the plan for this issue?

I am working on a language server for domain specific languages. Once the language grammar is created, extension is generated and installed. It works very well, without need to reload a window.
After that, if language grammar is changed, I would need to update the existing extension with a new textmate grammar (which is generated from language grammar) which will require a window reload.

[danixeee]

@alexandrudima I was wondering if you have had a chance to look at this feature?

[itsxallwater]

I'm a contributor on an extension called MV Basic that provides language support for a language (PickBASIC) that, depending on the vendor/flavor in use, has different language definitions and syntax highlighting requirements.

The language definitions are readily solved for by loading them at runtime in the language server, leveraging a configuration setting we've added to the extension, but dynamically switching the grammar has been a bit tougher given the discussion above. Adding my vote into the equation for having a route to specifying the grammar via API. I concede the trade off in needing the extension to activate and call the API, but we're effectively there anyway with respect to the language definition itself.

[danixeee]

@itsxallwater I found a workaround using setDecorations method, although it requires much more work.
I need to "paint" keywords of a language that is created in a runtime. Also, grammar changes should repaint a document, here is the example how it works.

Here are some implementation details if you find this helpful.

[smridge]

Is there some sort of regex to determine if filename or directory path includes a certain match?

Context: I'm writing a Rails syntax ruby language injection extension and running into similar issues. For example, certain methods are only available for controllers or views but not models.

[jquense]

any chance semantic highlighting could hackily cover this? https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide

[smridge]

any chance semantic highlighting could hackily cover this? https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide

From my understanding, semantic highlighting gives you more color variation to match things such as parameters to variables etc. It seems to allow for modifying the current token type after looking at the reference https://github.com/microsoft/vscode-extension-samples/blob/master/semantic-tokens-sample/src/extension.ts . I haven't fully grasped the VS Code docs on the semantic highlighting if it would allow for a dynamic location.

[mechatroner]

I wanted to open a new ticket, but found the existing one! I also have a strong need for dynamic grammar generation.
As a developer of Rainbow CSV I need this API to implement the following features:

• Support comments in CSV files (depending on user selection of a comment prefix Rainbow CSV would be able to generate an appropriate TextMate grammar)
• Support all possible single-character and multi-character CSV delimiters (just the ASCII character set has over 30 non-alphanumeric characters that can be used as CSV separators, so providing 30 pre-generated static syntax files is not feasible, and there are also use-cases for multi-character separators)
  There are multiple issues in my project which are blocked by the lack of this mechanism in VSCode: Ignoring comment lines mechatroner/vscode_rainbow_csv#14, Add a preference entry to let user change the delimiter ( ; or tab or | etc...) mechatroner/vscode_rainbow_csv#1, Add an option for an independent color for delimiters mechatroner/vscode_rainbow_csv#31, How to comment a line  mechatroner/vscode_rainbow_csv#70, added $ as a separation character but it is not detecting it mechatroner/vscode_rainbow_csv#58, Add a preference to allow for custom colours mechatroner/vscode_rainbow_csv#33

Both Vim and Sublime Text 3 editors support dynamic syntax generation; this mechanism is widely used in Vim by parenthesis highlighting extensions.

[atscott]

Just wanted to voice my support of this issue as well as the maintainer of the Angular Language Service extension for VSCode.

Components can define inline styles and since version 12, styles other than CSS (Sass, Less, Styl) can be used inline.
Syntax highlighting cannot be adjusted accordingly, even via extension config.

I updated our style highlighting recently to assume SCSS for inline styles to support more use-cases, but it would be nice to be able to do the actual right thing and provide the correct language highlighting for inline styles, either by extension config or programmatically in the extension itself.