Refactor ConsoleDrivers to support ANSI sequences natively (e.g. Virtual Terminal Sequences on Windows)

[tig]

• Build a new driver that does uses the new Console Virtual Terminal Sequences API.
• Refactor CursesDriver to not rely on Curses (as much?)

This may serve as the basis of a new Linux/Mac driver that does not so deeply depend on (and be constrainted by) curses.

More Useful Windows Terminal Issues:

• Extremely slow performance when processing virtual terminal sequences microsoft/terminal#10462
• ReadConsoleOutputW doesn't work with surrogate pairs (the buffer layers thread) microsoft/terminal#10810
• [Epic] Text Buffer rewrite  microsoft/terminal#8000
• ConPTY: Optimize SGR sequences by combining them into a single sequence microsoft/terminal#3016
• Give API to measure the space that a string occupies microsoft/terminal#218
• Strange behavior with ENABLE_VIRTUAL_TERMINAL_INPUT | ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT microsoft/terminal#4949

[dodexahedron]

So I've been doing a deep dive into the win32 console APIs and the frustratingly fractured/incomplete documentation available for virtual terminal sequences...

I think it's a very excellent goal for TG, and helps make things a bit more consistent, cross-platform.

But, a little voice stopped me and gave me a reality check that I just want to relay, for consideration:

Unless significant work that blocks other work has been done on it, perhaps that's something that might make sense for a VNext?

If done right, it's just internal implementation details, so we should be able to handle it in a way that doesn't affect the external API of TG. So, if following SemVer, that could even be acceptable as a point release, if desired.

Just a thought, since it's a pretty significant undertaking and there's value in a carefully thought-out approach to the implementation, so we don't have to iterate on it too much or work ourselves into avoidable pigeonholes. 🤷‍♂️

[Decimation]

@dodexahedron Spectre.Console has a robust cross-platform implementation with VTS support. It's similar to Terminal.Gui but is more focused on discrete shell controls. I think its implementation could be used as reference for implementing similar cross-platform VTS support for Terminal.Gui.

[dodexahedron]

Thanks. :)

I know @tig is aware of that library or may have even looked into it at least on some level, before.

I'm not sure what the current thoughts are on how to take advantage of that kind of thing, be it referencing, using code from parts of it, or drawing inspiration from it. I only came on board recently and have been working on other things, but this stuff is probably going to be at least incidentally relevant to what I'm working on now at some point, so it's good to keep this kind of thing in mind.

There are several other libraries out there on nuget, as well, that have potential application here in whole or in part for various bits of the application, but I'm pretty sure we do at least want to keep hard package dependencies to a minimum, if we can do so without it being too heinous and too much pointless duplication of essentially identical work.

[Decimation]

You're welcome. I've worked with VTS myself so I understand how it works to an extent.

Here is additional information:

• I wrote my own shell UI which uses VTS in both native and managed console functions.
• Pastel is another library to look into: it applies VTS to individual string objects and has a feature to automatically configure the console environment.

I can contribute to this feature if possible/feasible.

[tig]

There's no reason I can see that support for vts can't be isolated in just a driver.

And y'all are probably not considering a huge piece of all of this: rendering output is about 20% of the challenge. Spectre does not have mouse support and very limited keyboard support.

[tig]

Oh, and curses is Bad.

It is mostly in the way and my vision is we can fix a bunch of Linux/Mac issues by minimizing how much we use it. We'd use curses only for keyboard and mouse input and use the same rendering support we build for "vts" there.

Hence the name of the driver in this VTS branch being "ANSI Driver".

[dodexahedron]

I have a thought...

Might be a good idea to use a name not including ANSI because of Microsoft's historic decision to use that moniker for when they really meant ASCII in win32, just for sake of avoiding confusion, especially around native method calls, which typically shouldn't be the A versions.

[dodexahedron]

There's no reason I can see that support for vts can't be isolated in just a driver.

And y'all are probably not considering a huge piece of all of this: rendering output is about 20% of the challenge. Spectre does not have mouse support and very limited keyboard support.

Mostly agreed, here, at a high level, without having delved into these things in depth in this context yet.

My general thinking is just that we do our best to strike a balance between not reinventing as many wheels as possible and keeping dependencies (with all the caveats they can entail, too) to a reasonable minimum.

And remember: Any other project with a compatible license that has something useful but which we don't want or need the whole library for can be utilized via just lifting specific code, so long as the license is complied with (which is pretty simple with MIT). And then we can further modify any such code to fit better here, as well.

[dodexahedron]

Oh, and curses is Bad.

It is mostly in the way and my vision is we can fix a bunch of Linux/Mac issues by minimizing how much we use it. We'd use curses only for keyboard and mouse input and use the same rendering support we build for "vts" there.

Hence the name of the driver in this VTS branch being "ANSI Driver".

Ha. Yes. Curses has plenty of idiosyncrasies, legacy debt, etc., just like win32 does.

These libraries are old and grew up largely in a vacuum, with a lot of unfortunate results, from hacks to straight-up odd choices, especially for use in the context of a modern project like this one in 2024. It's also a dependency that isn't guaranteed to exist on a target system (though it's very likely to, since it's used by a ton of other software), so that's a small but relevant thing to keep in mind on that side.

[tig]

Relevant.

https://github.com/mintty/mintty/blob/master/wiki/CtrlSeqs.md