Squots (2x3 block drawing characters) are not sized properly

[PhMajerus]

Testing in Windows terminal 1.12.2931.0.

Unicode 14 includes symbols inherited from legacy computer systems (mostly 8-bit character sets), called Symbols for Legacy Computing (U+1FB00..U+1FBFF).
Squots are 2x3 block drawing characters that were originally on the TRS-80 and Teletext.
I know there is still no font in Windows that supports those, and even installing a font that does, such as UNSCII (http://viznut.fi/unscii/) does not make font-fallback work, but "unscii 16-full" can be used as the Windows Terminal font, and they are rendered.

The problem is, they are properly sized at 1 character cell when used by themselves, but as soon as there is a VT sequence to change their color, extra spaces get inserted between them.
This does not seem to be a problem with the UNSCII font, as when using Cascadia Mono, they are rendered as generic rectangles, but with the same extra spacing issue.

Reference rendering, using gnome-terminal from Ubuntu on WSL, displayed through WSLg:

The same files rendered in Windows Terminal using UNSCII (unscii 16-full) font:

And the same again, but rendered using Cascadia Mono, missing glyphs are expected, but notice the spacing still present:

Same but using CMD.exe instead of ActiveScript Shell, to show the problem isn't with ActiveScript Shell:

And here are my UTF-8 files to experiment:
Sonic (squots) (UTF-8).txt
Mario (squots) (UTF-8).txt

The Symbols for Legacy Computing are a new block of Unicode 14, but they might be important for Windows Terminal as their goal is precisely to bring into the modern character set all the legacy 8-bit text mode glyphs from historical computers. Having them properly supported might allow for example Commodore 64 (PETSCII) text screens to be displayed properly.
Also, this rendering issue might be related to a larger issue that could affect other characters blocks. Investigating it could improve Windows Terminal text rendering in other areas.

[elsaco]

@PhMajerus on Windows 11, after installing from the Microsoft Store, WT throws an error when launching axsh:

Majerus.net ActiveScript Shell [Version 2.0.9.0]
Copyright (C) 2005-2017 Majerus.net. All rights reserved.
Using JScript Language.

Error: This version of ActiveScript Shell is too old. Please get an updated version from http://www.majerus.net.

[process exited with code 10 (0x0000000a)]

And http://www.majerus.net/axsh/ gives 404 (not found) error.

[PhMajerus]

@elsaco Unfortunately, the Windows Store version doesn't work for now, it was built on the original Centennial container and was designed to refuse running if too old, but now it cannot be updated until I can turn it into an MSIX package.
The web site has been down for a while after a nearby construction destroyed most of our hardware, and I didn't have the time to rebuild that yet as I do that during my free time.

The best in the meantime is to grab the current build off my personal OneDrive public folder, it's updated whenever I have some new and stable feature: http://onedrive.phm.lu
I hope I can do a clean relaunch of my web site and Store apps, but probably not before late next year, sorry for the inconvenience 😟

[PhMajerus]

Some simpler tests to repro the issue without the test files:

echo -e '()'
echo -e '\e[32m(\e[31m)\e[m'
echo -e '\U0001FB24\U0001FB17'
echo -e '\e[32m\U0001FB24\e[31m\U0001FB17\e[m'

gnome-terminal:

Windows Terminal 1.12.2931.0:

Windows Terminal with UNSCII font:

Note how the last line has an extra space even though the only difference with the previous line is some VT color commands.

[zadjii-msft]

Weird. Thanks for the report! I don't believe I ever heard of that particular range of unicode before ☺️

[PhMajerus]

Maybe a hint, Notepad renders a single character of that range as two unknown character rectangles:

I suspect this means lower-level APIs are getting confused and measure those as two characters (probably two unhandled UTF-16 surrogates in NT's internal string format, or assumed to be double-width).
Windows Terminal probably renders a cluster of characters with same attributes as a single draw call, so they are properly rendered next to each other, but then measures the width to align the next cluster when attributes change, and gets a wrong width.

echo -e '#\U0001FB24\U0001FB17#\x1B[32m#' show that the next cluster after an attributes change isn't properly aligned, even if it isn't between U+1FBxx characters, but just if the previous cluster contains any of those characters.

[DHowett]

Windows Terminal probably renders a cluster of characters with same attributes as a single draw call,

You're totally right. WriteCharsLegacy (the as yet refactoring-defying function that powers all of the traditional windows console) measures surrogate pairs individually and then inserts them into the buffer individually. Each of them takes up "one column".

This has significant overlap with #8000 😄

[j4james]

Just FYI, there are a number of different character ranges effected by this. You can see a bunch of them in the notcurses-info app, which displays a kind of test pattern using a various block and line-drawing characters. This is what it currently looks like in the Windows console.

I've just been assuming all of these issues will be fixed by #8000 though.

[PhMajerus]

This is now fixed in Windows Terminal Preview 1.17.1023.
We still don't have support for the squots characters in Cascadia, but Terminal calculates characters sizes properly and renders them properly if the font supports them.

Here's the rendering in Terminal using UNSCII (unscii 16-full) font :

[DHowett]

Oh wow! I think I completely forgot that this was a "surrogate pairs are stored in two columns" issue. Thanks for checking it again. Fixed in #13626 :)

[DHowett]

Actually, I'm sorry, @j4james fixed this in #14640. With WriteCharsLegacy out of the way, we can measure strings properly. Missed the credit there. :)

[PhMajerus]

Thanks @j4james for fixing it!
Now I really want those squots characters in Cascadia.

Here are some examples of better looking squots ansi-arts using the 256-color palette:

Sonic (8-bit).txt (original sprite by SЄGΔ, Sonic 2 on Game Gear)
Shion (Monster World III).txt (original sprite by SЄGΔ, Wonder Boy V in Monster World III on Mega Drive/Genesis)

[j4james]

For the record, I think it was Leonard's #13626 PR that did most of the hard work necessary to fix this issue - I just stole the credit by hooking up his code to the VT PrintString routine. I'm thrilled my PR ended up fixing a bunch of stuff, but I have to admit it wasn't entirely intentional. 😊