Serial Monitor periodically corrupts post-ASCII output

[per1234]

Describe the problem

🐛 When Serial Monitor is used to display characters past the ASCII range printed by a non-native USB board, corruption occurs at various non-fixed positions in the output.

To reproduce

1. Upload a sketch that prints characters past the ASCII range to serial:

   void setup() {
     Serial.begin(115200);
   }
   void loop() {
     Serial.println("ASCII: !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~");
     Serial.println("> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ");
     delay(200);
   }

2. Open Serial Monitor.
   🐛 Some of the characters are replaced by �:

ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡��£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»��½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓ��ÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´��¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌ��ÎÏÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬��¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅ��ÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦��¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿��ÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×��ÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸��º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐ��ÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²��´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊ��ÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª��¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃ��ÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤��¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½��¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕ��×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶��¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎ��ÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯��±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈ��ÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨��ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁ��ÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙ��Û

Expected behavior

All characters to be printed correctly by Serial Monitor all the time:

ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ
ASCII: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_`abcdefghijklmnopqrstuvwxyz{|}~
> ASCII: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛ

Arduino IDE version

Original report

2.0.0-beta.12-nightly.20211028

Last verified with

d6a4b0f (with serial-monitor 0.11.0)

Operating system

Windows

Operating system version

10

Additional context

I originally suspected the corruption might be caused by #375, but this bug persists even though that one has been resolved.

---

The delay duration has an influence on the position of the corruption and the manifestation is different from one board to another (less corruption on Leonardo and MKR, more on Nano Every).

---

The issue does not occur when using the Arduino IDE 1.8.19 Serial Monitor, even when the board is running the binary that produced the corrupted output in Arduino IDE 2.x (i.e., it's not related to the sketch having been compiled in Arduino IDE 2.x).

---

Originally reported at https://forum.arduino.cc/t/serial-monitor-difficulties-with-german-umlauts/870541

---

Additional reports:

• Character encoding problem with Serial Monitor #1405
• https://forum.arduino.cc/t/serial-monitor-difficulties-with-german-umlauts/870541/7
• https://forum.arduino.cc/t/serial-read-and-serial-available-are-working-not-as-expected-in-the-code/925823
• https://forum.arduino.cc/t/using-v-2-0-serial-monitor-print/1034093
• https://forum.arduino.cc/t/question-marks-in-serial-monitor/1023531
• https://forum.arduino.cc/t/serial-monitor-detect-interferes-with-sketch-on-promicro/1017659
• https://forum.arduino.cc/t/degree-symbol-on-serial-monitor/1015370
• https://forum.arduino.cc/t/serial-monitor-produces-gibberish/950416/1
• https://forum.arduino.cc/t/serial-monitor-produces-gibberish/950416/15
• https://forum.arduino.cc/t/serial-monitor-issues-with-arduino-due/975247
• https://forum.arduino.cc/t/test-ide-2-0-rc2-the-serial-port-is-garbled/935697

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[RaytracerTim]

This problem still remains, at least for Arduino Due
IDE 2.0.2

[palazzol]

I'd like to try to fix this one. I can see serial-monitor.exe is providing the proper binary characters, and the corruption is already present by the time the data reaches the frontend. Can someone like @kittaakos maybe point me towards instructions on how to debug the middleware layer? I think something in the binaryblob/json/utf8 encoding is going wrong in there. Thanks!

[kittaakos]

instructions on how to debug

You must build and start the IDE2 from the sources.

For example:

git clone https://github.com/arduino/arduino-ide.git --depth 1 \
&& yarn --cwd ./arduino-ide \
&& yarn --cwd ./arduino-ide rebuild:browser \
&& yarn --cwd ./arduino-ide rebuild:electron \
&& code ./arduino-ide

If you use VS Code, you can start the app in debug mode via the App (Electron) launch entry. If you're not using Code, you have to figure it out on your own how to debug it.

the middleware layer

If you want to debug the backend, put a breakpoint here to see what is coming from the CLI's monitor, and here to see what is sent to the frontend. If you need additional input on how and why things work as they work, please ping the original authors of the feature: @AlbyIanna or @davegarthsimpson.

[palazzol]

Thanks so much @kittaakos. I was already building from github and using Code. I didn't realize those Code customizations were right there!

The example program demonstrating this issue is actually not sending an increasing set of characters from bytes 128 to 255. It is sending UTF8 encoded versions of those bytes, because the sketch editor defaults to UTF8 encoding. I believe this is correct, as it appears the Arduino 1 IDE did the same. These are multibyte characters in UTF8. Because incoming serial data is packetized, sometimes the first part of a UTF8 encoded character arrives without the rest, and therefore cannot be decoded correctly. In this case, there is a bug in the UTF8 processing which attempts to decode it anyway, resulting in corruption. I am putting in a pull request which seems to fix the issue.

Note: If the incoming serial data has any other encoding other than UTF8, it will never be displayed correctly in the serial monitor. If the serial monitor is meant to respect locales, then the hardcoded 'utf8' in the TextDecoder will not be sufficient.
I also recommend this issue be fixed soon as well, to avoid getting into this situation: #1393