Semantic tokens in HLS

[michaelpj]

I've got an open PR for semantic tokens support in lsp.

This is for tracking what we'd need to do to use it in HLS.

• Does it have to go through the hls-plugin-api model of everything being provided as plugin methods? I'm not sure how well that will work, because a) combining results from multiple plugins won't be nice, since the tokens will all be using relative positions (we could absolutize them, combine them, and then relativize them again, though), and b) if you want to do the delta support then you need state, result ids, all that fun stuff that is probably much easier with a single provider of highlighting.
• I have no idea where this should go generally. In ghcide?
• What do we want to highlight?
  • Normal Haskell syntax in doctest comments? Syntax highlighting in doctest comments? #1535
  • Type families? Support for Type Families #1637
  • Other kinds of "type-like things", e.g. constraints?
  • Deprecated things?
  • Class methods?
• What types and modifiers do we want to use for highlighting? The ones which are listed in the spec aren't great for Haskelly things, and I don't know how well custom types work in practice.

[pepeiborra]

• Entirely up to you, but it seems overkill to use a plugin system when there is one canonical implementation for Haskell. Sure, HLS can actually be used to target languages other than Haskell, but it is always possible to override the LSP handler directly at the lower ghcide plugin level and/or generalise later on.
• If in hls-plugin-api, you would add another provider and then the corresponding translation in D.IDE.Plugin.HLS. If in ghcide, then it would be registered as an LSP handler directly, as is done for e.g. document symbols (see D.IDE.LSP.Outline)
• Template Haskell?
• What are we using for document symbols?

[wz1000]

I'm not sure if an implementation in HLS would be worth it. Syntax highlighting already works quite well for most editors. Implementing it using an IDE sounds like a lot of work for little gain:

• Syntax highlighting has to be very low latency to present a good UX - it is unclear if this will be achievable
• It has to be tolerant of errors - a parse error means HLS has no access to the ParsedModule, so how would we return highlighting results?
• We can use stale ParsedModules, but this seems complicated to get right and very dubious to implement
• It will add a lot of traffic to the protocol (communication is all single-threaded) and potentially make everything else less responsive.
• The feature request from Support for Type Families #1637 doesn't seem to have anything to do with this. It is more about information about type families in hover popups. I'm pretty sure semantic highlighting doesn't let us color in tokens that occur in the hover popup.

[wz1000]

Another thing, we can't really distinguish between data types, type synonyms, type families and classes using the results of the parsing phase. If we want these to be highlighted differently, we would have to use results from the renaming phase, which would put a serious speed-bump in everything.

I suppose we could use stale renaming data and accept the occasional inaccuracy, but again this is tricky to implement correctly.

[michaelpj]

I'm not sure if an implementation in HLS would be worth it. Syntax highlighting already works quite well for most editors. Implementing it using an IDE sounds like a lot of work for little gain

Hmm, I think maybe I've missed something here. I thought that this was additional to the basic syntax highlighting, rather than replacing it entirely? So we could just, say, emit some tokens for deprecated items, and the rest would all follow the normal syntax highlighting. Of course, the spec doesn't really make it clear...

If it's an optional extra, then we can probably live with it being slower and waiting for renaming/typechecking in order to update it. I agree that wouldn't be acceptable if that was for the base level of highlighting.

[Ailrun]

I believe the feature is all about semantic highlighting, which may takes a bit longer than a normal syntax highlighting, but it's OK as far as there is normal syntax highlighting.

[jneira]

haskell lsp already has support for this

[GrayJack]

* What types and modifiers do we want to use for highlighting? The ones which are listed in the spec aren't great for Haskelly things, and I don't know how well custom types work in practice.

The good thing about using the standard tokens and modifiers as much as possible, is better support for themes, more themes that only support the standard tokens/modifiers will have good enough Haskell highlight out of the box

[david-christiansen]

I was asked to provide a list of things that seemed like useful semantic highlights for Haskell here. Generally, I think it would make sense to tell things apart that are not already distinguished by Haskell's syntax. By default, Haskell has two namespaces, used in value and type positions, and lexical features (capitalization) are used to distinguish various kinds of names. But in recent Haskell, some of these distinctions break down, as things cross from values to types and the interleavings become more complicated. Even in Haskell 98, type synonyms are written the same way as other types, but they behave a bit differently. Here are some examples of things that might be useful with semantic highlighting:

• Distinguish promoted data constructors used without ticks from type constructors, and likewise for promoted kinds
• Distinguish locally-bound names from module-level definitions
• Highlighting of type synonyms and type families, because their equality works so differently from newtypes or datatypes
• Tell type variables that are to be implicitly universally quantified in a local signature apart from those bound by the forall on a type signature from an enclosing definition in the presence of ScopedTypeVariables
• Highlight polymorphic vs monomorphic occurrences of literals (like 8 :: Num a => a vs 8 :: Integer)
• This should be off by default, but highlighting name occurrences that will be inlined in the resulting code could be useful for performance work
• Exported vs non-exported names from the current module
• Kind variables vs type variables in the presence of kind polymorphism (this seems less valuable to me than the others here)

Basically, the idea is to find things that can be confusing when writing code, and provide a means of communicating information to resolve that confusion without looking at docs or scrolling up to some enclosing code.

[soulomoon]

I can't wait but working on a draft of semantic tokens.

1. A lot of other languages server have already support it.
2. It would provide a much better experience reading haskell code given Haskell tendency to have duplicated names for various names spaces.
3. A lot of editors have great support about it, at least customizing highlighting of semantic tokens is quite easy in vscode.
4. It is not very hard to do a naive implementation about it. Details can be revise later.

Hope you guys could give me advices since I am quite new to hls.
#3892

[michaelpj]

Closing this since we have an implementation! Let's make specific tickets for further improvements.