Increase editor.maxTokenizationLineLength standard value to minimum 750

[CptHolzschnauz]

Describe the problem

Sketch with a long line leads to strange behaviour in the IDE.

• Wrong color parsing after col +/- 500

In a long line of code, after character +/- 500 the IDE stops coloring:

In some cases, following lines are also not highlighted:

To reproduce

Put that sketch snippet in the IDE, around col 500 the effect starts:

void setup() {
  String mqtt_msg = "('" + mqtt_device_id + "','" + String(longitude_tft, 10) + "','" + String(latitude_tft, 10) + "','" + mqtt_alarm + "','" + String(B1Volt, 1) + "','" + String(B2Volt, 1) + "','" + Batt_Temp11 + "','" + Batt_Temp12 + "','" + Batt_Temp13 + "','" + Batt_Temp14 + "','" + Batt_Temp15 + "','" + Batt_Temp16 + "','" + Batt_Temp21 + "','" + Batt_Temp22 + "','" + Batt_Temp23 + "','" + Batt_Temp24 + "','" + Batt_Temp25 + "','" + Batt_Temp26 + "','" + Batt_Temp31 + "','" + Batt_Temp32 + "','" + Batt_Temp33 + "','" + Batt_Temp34 + "','" + Batt_Temp35 + "','" + Batt_Temp36 + "','" + Batt_Temp41 + "','" + Batt_Temp42 + "','" + Batt_Temp43 + "','" + Batt_Temp44 + "','" + Batt_Temp45 + "','" + Batt_Temp46 + "','" + String(air_quality_score) + "','" + mqtt_status1 + "','" + mqtt_status2 + "','" + mqtt_status3 + "','" + mqtt_status4 + "','" + (t - 3) + "','" + h + "','" + (p / 100) + "','" + SIV + "','" + cog + "','" + sog + "')";
}
void loop() {}

Expected behavior

• Correct color parsing for sketches that contain reasonably long lines

Arduino IDE version

2.0.1

Operating system

macOS

Operating system version

12

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[per1234]

Hi @CptHolzschnauz. Thanks for your report.

Wrong color parsing after col +/- 500

This is an intentional limit. The line length at which this occurs was recently reduced:

#1450

It is possible to configure this via the advanced settings. I'll provide instructions:

1. Press the Ctrl+Shift+P keyboard shortcut (Command+Shift+P for macOS users) to open the "Command Palette".
2. Select the "Preferences: Open Settings (UI)" command from the menu.
3. A "Preferences" tab will now open in the Arduino IDE main panel. In the "Search Settings" field, type editor.maxTokenizationLineLength
4. Adjust the value of the "Editor: Max Tokenization Line Length" setting according to your preference.
5. Click the X icon on the "Preferences" tab.

Please note that it was reduced for a good reason: #1343
However, it was reduced from 20000 to 500, so it might be there is a higher value that will provide you with an acceptable balance between performance and highlighting.

If it's on the same "width", the code from this line on is wrong parsed:

Please provide a minimal sketch that demonstrates the problem.

If i autoformat that sketch it won't compile anymore.

Please provide a complete minimal sketch that demonstrates the problem. The sketch you provided is not valid code, so it is expected that it won't compile.

Special characters like ö ä ü are changed after save on the cloud and reload. Example: Für goes für

This bug is being tracked here: #449

[CptHolzschnauz]

However, it was reduced from 20000 to 500, so it might be there is a higher value that will provide you with an acceptable balance between performance and highlighting.

I set it to 750 and i suggest to use this as a new standard value. I can feel NO difference in the perfomance.

Please provide a minimal sketch that demonstrates the problem.

This problem is solved with the solution described above

Please provide a minimal sketch that demonstrates the problem.

I can't. It's happen only with my megasketch (245240 Bytes). After Autoformat, compiler says he can't find a included library. Strange is also that the compiler warnings are in german while the IDE is set to english.
While testing to find out whats happening i did cmd+t and cmd+z (no coding) and then it was not compiling anymore with IDE2 (library not found). I closed IDE2, opend IDE1, compiled fine, closed IDE1 and open IDE , look what happen:
The IDE turns white, the output is tokenized, and the sketch compiles again with no problem (I never edited the code during these tests):

Something is rotten in tokenization/autoformatting/feeding the compiler. Sorry that i can't provide a proper error description, that behaviour is too flaky to describe.

[per1234]

I can't. It's happen only with my megasketch (245240 Bytes). After Autoformat, compiler says he can't find a included library.

Since it is not actionable based on the limited information we have, I will dedicate this issue exclusively to your request that the default editor.maxTokenizationLineLength value be increased.

Strange is also that the compiler warnings are in german while the IDE is set to english

Thanks for bringing this to our attention. I have submitted a dedicated report for it: #1636

In the future, please, please submit a separate report for each distinct issue. These rambling issues are very difficult to manage and make it likely we will lose track of one of the things that were reported.

[CptHolzschnauz]

After reinstallation the issues with the IDE was gone.
Ergo: Loading a sketch with a line longer than editor.maxTokenizationLineLength can corrupt the IDE.
Proposal: editor.maxTokenizationLineLength = 750

[per1234]

Hi @CptHolzschnauz.

After reinstallation the issues with the IDE was gone.

I'm glad it is working now.

Proposal: editor.maxTokenizationLineLength = 750

If you close the issue, it will not be possible for us to track your proposal. So I will reopen it for you.

[CptHolzschnauz]

FYI: The strange behaviour i observed returned also with the new installed IDE. Out of a sudden no compiling, can't find lib, the 1.8 IDE compiles fine the same sketch, reopen in IDE 2 eveything is fine. Can't say why, can't reproduce. Only hint - it's a large sketch of 212 kB.