Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use 'backslashreplace' not to crash on malformed UTF from subprocess #700

Merged
merged 1 commit into from
Dec 20, 2023

Conversation

marc-hb
Copy link
Collaborator

@marc-hb marc-hb commented Dec 15, 2023

Giant commit
zephyrproject-rtos/hal_nxp@f9f0944bc2b4fce "Update to SDK 2.14" added files with malformed UTF-8, more precisely with the DEGREE SIGN (°) encoded in 8bit latin1/CP1252.

This crashes west grep.

Before this fix:

nxp$ west grep 'TEMPERATURE_CONV_FACTOR.*Will give'

Traceback (most recent call last):
  File ".local/bin/west", line 33, in <module>
    sys.exit(load_entry_point('west', 'console_scripts', 'west')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "west/src/west/app/main.py", line 1085, in main
    app.run(argv or sys.argv[1:])
  File "west/src/west/app/main.py", line 244, in run
    self.run_command(argv, early_args)
  File "west/src/west/app/main.py", line 503, in run_command
    self.run_builtin(args, unknown)
  File "west/src/west/app/main.py", line 611, in run_builtin
    self.cmd.run(args, unknown, self.topdir,
  File "west/src/west/commands.py", line 194, in run
    self.do_run(args, unknown)
  File "west/src/west/app/project.py", line 1765, in do_run
    completed_process = self.run_subprocess(
                        ^^^^^^^^^^^^^^^^^^^^
  File "west/src/west/commands.py", line 325, in run_subprocess
    return subprocess.run(args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 2146, in _communicate
    stdout = self._translate_newlines(stdout,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1086, in _translate_newlines
    data = data.decode(encoding, errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 385: invalid start byte

After this fix, no crash and no interruption and:

...
mcux-sdk/middleware/issdk/sensors/fxpq3115_drv.h:#define FXPQ3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */
mcux-sdk/middleware/issdk/sensors/mpl3115_drv.h:#define MPL3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */
...

Giant commit
zephyrproject-rtos/hal_nxp@f9f0944bc2b4fce
"Update to SDK 2.14" added files with malformed UTF-8, more precisely
with the DEGREE SIGN (°) encoded in 8bit latin1/CP1252. Maybe others.

This crashes `west grep`.

Before this fix:

```
nxp$ west grep 'TEMPERATURE_CONV_FACTOR.*Will give'

Traceback (most recent call last):
  File ".local/bin/west", line 33, in <module>
    sys.exit(load_entry_point('west', 'console_scripts', 'west')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "west/src/west/app/main.py", line 1085, in main
    app.run(argv or sys.argv[1:])
  File "west/src/west/app/main.py", line 244, in run
    self.run_command(argv, early_args)
  File "west/src/west/app/main.py", line 503, in run_command
    self.run_builtin(args, unknown)
  File "west/src/west/app/main.py", line 611, in run_builtin
    self.cmd.run(args, unknown, self.topdir,
  File "west/src/west/commands.py", line 194, in run
    self.do_run(args, unknown)
  File "west/src/west/app/project.py", line 1765, in do_run
    completed_process = self.run_subprocess(
                        ^^^^^^^^^^^^^^^^^^^^
  File "west/src/west/commands.py", line 325, in run_subprocess
    return subprocess.run(args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 550, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1209, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 2146, in _communicate
    stdout = self._translate_newlines(stdout,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1086, in _translate_newlines
    data = data.decode(encoding, errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 385: invalid start byte
```

After this fix, no crash and no interruption and:

mcux/mcux-sdk/middleware/issdk/sensors/fxpq3115_drv.h:#define FXPQ3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */
mcux/mcux-sdk/middleware/issdk/sensors/mpl3115_drv.h:#define MPL3115_TEMPERATURE_CONV_FACTOR (256) /* Will give \xb0C */

Signed-off-by: Marc Herbert <[email protected]>
@marc-hb marc-hb marked this pull request as ready for review December 15, 2023 09:21
@marc-hb
Copy link
Collaborator Author

marc-hb commented Dec 15, 2023

west grep 'TEMPERATURE_CONV_FACTOR.*Will give'

Obviously, this is the not the search string I was using when I found this crash :-)

@mbolivar-ampere mbolivar-ampere merged commit 3260b41 into zephyrproject-rtos:main Dec 20, 2023
9 checks passed
mbolivar-ampere added a commit to mbolivar-ampere/west that referenced this pull request Dec 20, 2023
Bug fixes:

- Hotfix for west grep. For details, see:
  zephyrproject-rtos#700

Signed-off-by: Martí Bolívar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants