Return the grammar state after tokenizing

[golok727]

Clear and concise description of the problem

codeToTokens to return the grammar state after tokenization.

Suggested solution

This could be beneficial for tokenization in the editor I'm currently developing

Alternative

No response

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guide.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Contributes

• If this feature request is accepted, I am willing to submit a PR to fix this issue

[antfu]

What would be the API looks like in your mind?

[golok727]

What would be the API looks like in your mind?

Including the GrammarState in the TokensResult would be ideal.

Sample usage

class EditorTokensProvider {
// ...
    tokenizeLine(line, state?: GrammarState) {
         const {tokens, grammarState} = highligher.codeToTokens(code, {theme, grammarState: state, lang))

         return {tokens: tokens[0], endState: grammarState }
    }
//...
}

Additionally, could you provide a default static method in the GrammarState to get an initial state?

class GrammarState 
{
 static default(lang, theme) 
{
    return new GrammarState(INITIAL, lang, theme)
}
} 

Thank you for this awesome project and for your time!

[antfu]

That would be a breaking change to the function unfortunately. We would need to find another non-breaking API for that