Add highlighting to code blocks in the built-in documentation (and more?)

[MewPurPur]

Describe the project you are working on

This affects my user experience in all projects

Describe the problem or limitation you are having in your project

Code blocks in the documentation are missing highlighting which makes them hard to read. They also use 4 spaces, whereas I like to use tabs with size 3, which makes them hard to copy. But I'll primarily discuss the highlighting problem in this proposal.

Here's how code blocks look on the website:

This looks awesome! The tab container is nicely convenient, there is a button to copy the code, but most of all, the highlighting makes it easier to read the code.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Here's how code blocks look in the built-in docs:

This could be better than the web docs, but it's far worse currently. The built-in docs can not only have the same functionalities, they can also be context-aware and react to things like your chosen tab size or syntax highlighting settings.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

So how can we do better? I have two approaches on my mind:

A) Break the pattern of documentation pages being one huge RichTextLabel

+ Accomplishing the highlighting itself is straightforward, just add a read-only TextEdit SyntaxHighlighter.
+ Breaking the pattern means that things like the Language tabs and the Copy Code button will be possible without hacks.
– A significant refactor will be needed for all the systems that assume the docs content is in a single node, such as the Search system. More on this below.
– RichTextLabel is threaded, so without the pages being entirely RichTextLabel, performance might take a hit.

Instead of everything going on a single RichTextLabel, a few nodes would go into a VBoxContainer. For a start, this VBoxContainer could be just RichTextLabels and TextEdits, but in the future we might come up with other things that different nodes could do. As far as I know, we always close BBCode tags before starting our codeblocks and if not, it would be a mistake, so we don't need to address BBCode tags when a RichTextLabel ends to give way to a TextEdit should this be implemented.

I am not sure how the navigation system works, but it may need to be reworked. The search system would definitely need to be reworked, as it scours the single RichTextLabel using its search() method, so now we will need to do this with all of the RichTextLabels an TextEdits in the page.

B) Keep the documentation pages as a single RichTextLabel

– Accomplishing the highlighting is tricky. More on this below.
– Things like the Language tabs could be added in the future, but would take absurd hacks.
+ No major refactors needed.
+ Performance stonks

How can we get the highlighting to work if we go with this approach?

Setup: Create a SyntaxHighlighter of the relevant type, usually a GDScriptHighlighter. Pass to it the user's editor settings.

Algorithm: Ask the SyntaxHighlighter how it would paint the code and translate it into BBCodes

SyntaxHighlighter doesn't have a method that allows you to pass arbitrary text to it and learn what its highlighting would be. This would require its own proposal.

Instead, we can use the get_line_syntax_highlighting() method, which is called on a SyntaxHighlighter resource in a TextEdit with the line number passed to it. This method returns a dictionary where the keys are columns number and the values are colors that start from those columns. So, we might need to do the following:

• Create a temporary TextEdit and equip it with our SyntaxHighlighter.
• Use get_line_syntax_highlighting() on each line and get its color dictionary.
• Iterate the dictionary. We'll be adding color tags, which will change the length of the line, so we'll take the difference between column numbers to add our color tags and close them.
• After we're done iterating, free the TextEdit.

This might actually throw a wrench on the whole idea that performance would be better this way, but I can't tell for sure.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

If we have built-in docs we might as well make them look better than a dog's breakfast.

[mieldepoche]

Respecting the user theme (color theme, but also code font and code font size, indentation preferences) would be nice too.
As well as styling standalone expressions, not just code blocks:

but I guess that's for a future proposal?

[MewPurPur]

@mieldepoche Of course, editor settings will be taken into consideration. Standalone expressions though probably won't be getting styled, highlighting is only used for code blocks. I won't get into why but I don't think we should break off from this convention.

[Mickeon]

As well as styling standalone expressions, not just code blocks:

Part of my original intent when I reworked the built-in docs colors was to do this, but the code surrounding syntax highlighting is so overly laborious that, unless a whole rework was made, I hardly see it being possible without some hacky implementations.

If #5299 were implemented, though, at least true and false could borrow their color from the theme.

[seshoumara]

The documentation code isn't the only syntax that needs some highlighting. Last time I used bbcode it was hard to distinguish tags from text, from tag parameters (especially when you create a table). Let me know if I should open a separate issue instead.

[Mickeon]

That's entirely unrelated to this proposal, so by all means, yes