Support for textDocument/semanticTokens

[angelozerr]

See https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_semanticTokens

To implement this LSP textDocument/semanticTokens support, I think we should:

• use some classes from LSP4E https://github.com/eclipse/lsp4e/tree/main/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/semanticTokens like https://github.com/eclipse/lsp4e/blob/main/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/semanticTokens/SemanticTokensDataStreamProcessor.java
• use an IJ extension point by trying:
  • solution 1: (RainbowVisitor, my prefered) try to support IntelliJ semantic-highlighting by implementing RainbowVisitor. I think it is the better solution because if I understand we could mix several higlighting (one for syntax with custom lexer or tetmate) and the LSP textDocument/semanticTokens
  • solution 2 (SyntaxHighlighter) : if it doesn't work, implement a SyntaxHighlighter like TextMate does with TextMateSyntaxHighlighterFactory. which works without a PsiFile lexer.
  • solution 3: (Editor.getMarkupModel() ), see eplanation at What about using LSP4IJ ? FalsePattern/ZigBrains#53 (comment)

The implementation should use a new class LSPSemanticTokensSupport like we did for any LSP features, see for instance

lsp4ij/src/main/java/com/redhat/devtools/lsp4ij/features/foldingRange/LSPFoldingRangeBuilder.java

Line 59 in 69f8a0c

LSPFoldingRangeSupport foldingRangeSupport = LSPFileSupport.getSupport(file).getFoldingRangeSupport();

Those class LSP*Support are registered in an LSPFileSUpport which is registered with a key in a PsiFile. It give the capability for instance to cancel the LSP request when file is closed. LSP4IJ support too LSP cancel request to cancel the LSP request when file changed or Intellij throws a ProcessCanceledException to avoid using TimeOut.

[angelozerr]

@FalsePattern , as you have already implemented textDocument/semanticTokens in your project, are you interested to work on this issue?

I have tried to explain how to implement this LSP support inside LSP4IJ, I know you did the solution 3, but I think it should be cleaner to use an IJ extension point (see solution 1 (the best I think) or solution 2).

Please note that LSP4IJ support cancelRequest which I think is very important for this LSP support. I think too we should use some LSP4E classes https://github.com/eclipse/lsp4e/tree/main/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/semanticTokens like https://github.com/eclipse/lsp4e/blob/main/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/semanticTokens/SemanticTokensDataStreamProcessor.java because I know that several Eclipse contributor have worked hard to omplement this feature and fixes several bugs.

LSP4IJ contains too a lof tests and I have tried to write test for important features like completion, hover (see https://github.com/redhat-developer/lsp4ij/tree/main/src/test/java/com/redhat/devtools/lsp4ij/features)

We should have too tests for textDocument/semanticTokens.

[angelozerr]

@FalsePattern it seems it is a blocker issue to adopt LSP4IJ, I will try to investigate how to implement this feature in my spare time since it is not for us (Red Hat) our priority, but if you want to try to implement it, please tell me.

[angelozerr]

@CppCXY @FalsePattern I have merged the PR and you can install https://github.com/redhat-developer/lsp4ij?tab=readme-ov-file#testing-nightly-builds to play with the experimental semantic tokens.

Please read:

• https://github.com/redhat-developer/lsp4ij/blob/main/docs/UserGuide.md#semantic-tokens-support
• https://github.com/redhat-developer/lsp4ij/blob/main/docs/LSPSupport.md#semantic-tokens
• https://github.com/redhat-developer/lsp4ij/blob/main/docs/DeveloperGuide.md#semantic-tokens-colors-provider

It is an experimental support, so don't hesitate to create any PR to improve the support, change the API, etc, thanks!