Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chiptest] Fix UTF-8 decoding error in stdout/err #16101

Merged
merged 2 commits into from
Mar 12, 2022
Merged

[chiptest] Fix UTF-8 decoding error in stdout/err #16101

merged 2 commits into from
Mar 12, 2022

Conversation

arkq
Copy link
Contributor

@arkq arkq commented Mar 11, 2022

Problem

Without such a change the bug #15796 was impossible to detect:

2022-03-11 09:36:07.432 INFO    Starting iteration 1                                                                                                                                                                 
2022-03-11 09:36:10.104 INFO    DL_LockUnlock        - Completed in 2.67 seconds                                                                                                                                     
2022-03-11 09:36:12.999 INFO    DL_Schedules         - Completed in 2.89 seconds                                                                                                                                     
Exception in thread APP  OUT:                                                                                                                                                                                        
Traceback (most recent call last):                                                                                                                                                                                   
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner                                                                                                                                              
    self.run()                                                                                                                                                                                                       
  File "/home/a.bokowy/connectedhomeip/scripts/tests/chiptest/runner.py", line 65, in run                                                                                                                            
    for line in iter(self.pipeReader.readline, ''):                                                                                                                                                                  
  File "/usr/lib/python3.8/codecs.py", line 322, in decode                                                                                                                                                           
    (result, consumed) = self._buffer_decode(data, self.errors, final)                                                                                                                                               
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 145: invalid start byte

Change overview

  • fix UTF-8 decoding
  • unifies PIPE type for Linux and Darwin platforms. From now, all platforms will use PTY-based PIPE for reading subprocess output in real time (without 4k block buffering).

Testing

Checked that running /scripts/tests/run_test_suite.py --target-glob "DL_*" run with reversed commit 0b58f58 properly shows ASAN output.

@github-actions
Copy link

github-actions bot commented Mar 11, 2022 •
edited
Loading

PR #16101: Size comparison from ae514b6 to e745c0c

Full report (18 builds for cyw30739, efr32, esp32, k32w, linux, mbed, nrfconnect, p6, telink)
platform target config section ae514b6 e745c0c change % change
cyw30739 light cyw930739m2evb_01 (read/write) 599102 599102 0 0.0
.app_xip_area 506700 506700 0 0.0
.bss 75108 75108 0 0.0
.data 640 640 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
lock cyw930739m2evb_01 (read/write) 556986 556986 0 0.0
.app_xip_area 466112 466112 0 0.0
.bss 73612 73612 0 0.0
.data 604 604 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
ota-requestor cyw930739m2evb_01 (read/write) 566766 566766 0 0.0
.app_xip_area 466228 466228 0 0.0
.bss 82932 82932 0 0.0
.data 564 564 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
efr32 lighting-app BRD4161A (read only) 921128 921128 0 0.0
(read/write) 128232 128232 0 0.0
.bss 126176 126176 0 0.0
.data 2052 2052 0 0.0
.text 921120 921120 0 0.0
BRD4161A+rpc (read only) 949940 949940 0 0.0
(read/write) 144184 144184 0 0.0
.bss 141952 141952 0 0.0
.data 2232 2232 0 0.0
.text 949932 949932 0 0.0
window-app BRD4161A (read only) 853732 853732 0 0.0
(read/write) 126216 126216 0 0.0
.bss 124312 124312 0 0.0
.data 1904 1904 0 0.0
.text 853724 853724 0 0.0
esp32 all-clusters-app c3devkit (read only) 958524 958524 0 0.0
(read/write) 1393858 1393858 0 0.0
.dram0.bss 63584 63584 0 0.0
.dram0.data 14220 14220 0 0.0
.flash.rodata 197144 197144 0 0.0
.flash.text 958524 958524 0 0.0
.iram0.text 62016 62016 0 0.0
m5stack (read only) 1014127 1014127 0 0.0
(read/write) 461092 461092 0 0.0
.dram0.bss 69104 69104 0 0.0
.dram0.data 34048 34048 0 0.0
.flash.rodata 226104 226104 0 0.0
.flash.text 1008743 1008743 0 0.0
.iram0.text 123107 123107 0 0.0
k32w light k32w061+release (read/write) 695388 695388 0 0.0
.bss 77112 77112 0 0.0
.data 1928 1928 0 0.0
.text 610548 610548 0 0.0
lock k32w061+release (read/write) 696360 696360 0 0.0
.bss 77080 77080 0 0.0
.data 1952 1952 0 0.0
.text 611528 611528 0 0.0
linux chip-tool-ipv6only arm64 (read only) 9729660 9729660 0 0.0
(read/write) 475729 475729 0 0.0
.bss 43953 43953 0 0.0
.data 1176 1176 0 0.0
.data.rel.ro 371520 371520 0 0.0
.dynamic 560 560 0 0.0
.got 55280 55280 0 0.0
.init 24 24 0 0.0
.init_array 192 192 0 0.0
.rodata 492148 492148 0 0.0
.text 8186868 8186868 0 0.0
thermostat-no-ble arm64 (read only) 2181588 2181588 0 0.0
(read/write) 148625 148625 0 0.0
.bss 65409 65409 0 0.0
.data 1024 1024 0 0.0
.data.rel.ro 74968 74968 0 0.0
.dynamic 560 560 0 0.0
.got 4256 4256 0 0.0
.init 24 24 0 0.0
.init_array 352 352 0 0.0
.rodata 133444 133444 0 0.0
.text 1829984 1829984 0 0.0
mbed lock-app CY8CPROTO_062_4343W+release (read only) 6224 6224 0 0.0
(read/write) 2351012 2351012 0 0.0
.bss 186212 186212 0 0.0
.data 5784 5784 0 0.0
.text 1313612 1313612 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read/write) 1152643 1152643 0 0.0
bss 146188 146188 0 0.0
rodata 144516 144516 0 0.0
text 786676 786676 0 0.0
p6 all-clusters-app default (read/write) 2495680 2495680 0 0.0
.bss 119640 119640 0 0.0
.data 2672 2672 0 0.0
.text 1453944 1453944 0 0.0
light-app default (read/write) 2399680 2399680 0 0.0
.bss 113104 113104 0 0.0
.data 2528 2528 0 0.0
.text 1357944 1357944 0 0.0
lock-app default (read/write) 2363240 2363240 0 0.0
.bss 112848 112848 0 0.0
.data 2488 2488 0 0.0
.text 1321504 1321504 0 0.0
telink lighting-app tlsr9518adk80d (read/write) 889766 889766 0 0.0
bss 86976 86976 0 0.0
noinit 37160 37160 0 0.0
text 628970 628970 0 0.0

bzbarsky-apple
bzbarsky-apple approved these changes Mar 12, 2022
View reviewed changes
scripts/tests/chiptest/runner.py Outdated Show resolved Hide resolved
@bzbarsky-apple
Copy link
Contributor

/rebase

@arkq @bzbarsky-apple
[chiptest] Fix UTF-8 decoding error in stdout/err
1617629 
This commit also unifies PIPE type for Linux and Darwin platforms. From
now, all platforms will use PTY-based PIPE for reading subprocess output
in real time (without 4k block buffering).
@github-actions
Copy link

github-actions bot commented Mar 12, 2022 •
edited
Loading

PR #16101: Size comparison from 5371ccb to c7c7229

Full report (26 builds for cyw30739, efr32, esp32, k32w, linux, nrfconnect, p6, telink)
platform target config section 5371ccbc c7c7229 change % change
cyw30739 light cyw930739m2evb_01 (read/write) 602530 602530 0 0.0
.app_xip_area 509708 509708 0 0.0
.bss 75576 75576 0 0.0
.data 596 596 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
lock cyw930739m2evb_01 (read/write) 560390 560390 0 0.0
.app_xip_area 469096 469096 0 0.0
.bss 74080 74080 0 0.0
.data 560 560 0 0.0
.rodata 0 0 0 0.0
.text 0 0 0 0.0
ota-requestor-no-progress-logging cyw930739m2evb_01 (read/write) 570002 570002 0 0.0
.app_xip_area 469060 469060 0 0.0
.bss 83384 83384 0 0.0
.data 520 520 0 0.0
.rodata 0 0 0 0.0
.text 112 112 0 0.0
efr32 lighting-app BRD4161A (read only) 918144 918144 0 0.0
(read/write) 128656 128656 0 0.0
.bss 126648 126648 0 0.0
.data 2008 2008 0 0.0
.text 918136 918136 0 0.0
BRD4161A+rpc (read only) 946932 946932 0 0.0
(read/write) 144612 144612 0 0.0
.bss 142424 142424 0 0.0
.data 2188 2188 0 0.0
.text 946924 946924 0 0.0
window-app BRD4161A (read only) 850616 850616 0 0.0
(read/write) 126632 126632 0 0.0
.bss 124768 124768 0 0.0
.data 1864 1864 0 0.0
.text 850608 850608 0 0.0
esp32 all-clusters-app c3devkit (read only) 961344 961344 0 0.0
(read/write) 1394994 1394994 0 0.0
.dram0.bss 64056 64056 0 0.0
.dram0.data 14188 14188 0 0.0
.flash.rodata 197832 197832 0 0.0
.flash.text 961344 961344 0 0.0
.iram0.text 62016 62016 0 0.0
m5stack (read only) 1016963 1016963 0 0.0
(read/write) 462204 462204 0 0.0
.dram0.bss 69576 69576 0 0.0
.dram0.data 34016 34016 0 0.0
.flash.rodata 226776 226776 0 0.0
.flash.text 1011579 1011579 0 0.0
.iram0.text 123107 123107 0 0.0
k32w light k32w061+release (read/write) 699052 699052 0 0.0
.bss 77560 77560 0 0.0
.data 1888 1888 0 0.0
.text 613804 613804 0 0.0
lock k32w061+release (read/write) 699884 699884 0 0.0
.bss 77552 77552 0 0.0
.data 1908 1908 0 0.0
.text 614624 614624 0 0.0
linux all-clusters-app debug (read only) 2465009 2465009 0 0.0
(read/write) 146384 146384 0 0.0
.bss 60640 60640 0 0.0
.data 1264 1264 0 0.0
.data.rel.ro 78728 78728 0 0.0
.dynamic 592 592 0 0.0
.got 4176 4176 0 0.0
.init 27 27 0 0.0
.init_array 936 936 0 0.0
.rodata 210693 210693 0 0.0
.text 2090994 2090994 0 0.0
bridge-app debug+rpc (read only) 1749589 1749589 0 0.0
(read/write) 92912 92912 0 0.0
.bss 47752 47752 0 0.0
.data 1952 1952 0 0.0
.data.rel.ro 38128 38128 0 0.0
.dynamic 592 592 0 0.0
.got 3928 3928 0 0.0
.init 27 27 0 0.0
.init_array 544 544 0 0.0
.rodata 143372 143372 0 0.0
.text 1490581 1490581 0 0.0
chip-tool debug (read only) 10094493 10094493 0 0.0
(read/write) 357680 357680 0 0.0
.bss 25696 25696 0 0.0
.data 1072 1072 0 0.0
.data.rel.ro 324864 324864 0 0.0
.dynamic 608 608 0 0.0
.got 4776 4776 0 0.0
.init 27 27 0 0.0
.init_array 616 616 0 0.0
.rodata 514965 514965 0 0.0
.text 8798277 8798277 0 0.0
chip-tool-ipv6only arm64 (read only) 9727788 9727788 0 0.0
(read/write) 475425 475425 0 0.0
.bss 44017 44017 0 0.0
.data 1128 1128 0 0.0
.data.rel.ro 371232 371232 0 0.0
.dynamic 560 560 0 0.0
.got 55256 55256 0 0.0
.init 24 24 0 0.0
.init_array 192 192 0 0.0
.rodata 492244 492244 0 0.0
.text 8185700 8185700 0 0.0
door-lock-app debug (read only) 1984225 1984225 0 0.0
(read/write) 119776 119776 0 0.0
.bss 50944 50944 0 0.0
.data 992 992 0 0.0
.data.rel.ro 62456 62456 0 0.0
.dynamic 592 592 0 0.0
.got 4112 4112 0 0.0
.init 27 27 0 0.0
.init_array 664 664 0 0.0
.rodata 178460 178460 0 0.0
.text 1656786 1656786 0 0.0
lighting-app debug+rpc (read only) 2115753 2115753 0 0.0
(read/write) 125160 125160 0 0.0
.bss 51776 51776 0 0.0
.data 1472 1472 0 0.0
.data.rel.ro 66408 66408 0 0.0
.dynamic 608 608 0 0.0
.got 4144 4144 0 0.0
.init 27 27 0 0.0
.init_array 736 736 0 0.0
.rodata 170972 170972 0 0.0
.text 1788786 1788786 0 0.0
ota-provider-app debug (read only) 1917489 1917489 0 0.0
(read/write) 115536 115536 0 0.0
.bss 50816 50816 0 0.0
.data 1256 1256 0 0.0
.data.rel.ro 57816 57816 0 0.0
.dynamic 608 608 0 0.0
.got 4368 4368 0 0.0
.init 27 27 0 0.0
.init_array 616 616 0 0.0
.rodata 163691 163691 0 0.0
.text 1602738 1602738 0 0.0
ota-requestor-app debug (read only) 1945745 1945745 0 0.0
(read/write) 118536 118536 0 0.0
.bss 51840 51840 0 0.0
.data 1416 1416 0 0.0
.data.rel.ro 59848 59848 0 0.0
.dynamic 592 592 0 0.0
.got 4160 4160 0 0.0
.init 27 27 0 0.0
.init_array 640 640 0 0.0
.rodata 160068 160068 0 0.0
.text 1634466 1634466 0 0.0
shell debug (read only) 2410849 2410849 0 0.0
(read/write) 150184 150184 0 0.0
.bss 70600 70600 0 0.0
.data 784 784 0 0.0
.data.rel.ro 73088 73088 0 0.0
.dynamic 592 592 0 0.0
.got 4160 4160 0 0.0
.init 27 27 0 0.0
.init_array 920 920 0 0.0
.rodata 207474 207474 0 0.0
.text 2047138 2047138 0 0.0
thermostat-no-ble arm64 (read only) 2206980 2206980 0 0.0
(read/write) 149377 149377 0 0.0
.bss 65649 65649 0 0.0
.data 1024 1024 0 0.0
.data.rel.ro 75368 75368 0 0.0
.dynamic 560 560 0 0.0
.got 4352 4352 0 0.0
.init 24 24 0 0.0
.init_array 360 360 0 0.0
.rodata 136820 136820 0 0.0
.text 1850848 1850848 0 0.0
tv-app debug (read only) 2655601 2655601 0 0.0
(read/write) 250776 250776 0 0.0
.bss 167840 167840 0 0.0
.data 3104 3104 0 0.0
.data.rel.ro 73800 73800 0 0.0
.dynamic 592 592 0 0.0
.got 4544 4544 0 0.0
.init 27 27 0 0.0
.init_array 880 880 0 0.0
.rodata 204245 204245 0 0.0
.text 2277218 2277218 0 0.0
nrfconnect all-clusters-app nrf52840dk_nrf52840 (read/write) 1156219 1156219 0 0.0
bss 146644 146644 0 0.0
rodata 145220 145220 0 0.0
text 789208 789208 0 0.0
p6 all-clusters-app default (read/write) 2492128 2492128 0 0.0
.bss 120080 120080 0 0.0
.data 2632 2632 0 0.0
.text 1450392 1450392 0 0.0
light-app default (read/write) 2395992 2395992 0 0.0
.bss 113544 113544 0 0.0
.data 2488 2488 0 0.0
.text 1354256 1354256 0 0.0
lock-app default (read/write) 2359536 2359536 0 0.0
.bss 113288 113288 0 0.0
.data 2448 2448 0 0.0
.text 1317800 1317800 0 0.0
telink lighting-app tlsr9518adk80d (read/write) 893842 893842 0 0.0
bss 87432 87432 0 0.0
noinit 37160 37160 0 0.0
text 631882 631882 0 0.0

@andy31415 andy31415 merged commit 2982c0c into project-chip:master Mar 12, 2022
@arkq arkq deleted the pty-pipe branch March 14, 2022 07:12
andrei-menzopol pushed a commit to andrei-menzopol/connectedhomeip that referenced this pull request Apr 14, 2022
@arkq @bzbarsky-apple @andrei-menzopol
[chiptest] Fix UTF-8 decoding error in stdout/err (project-chip#16101)
b6d2e99 
* [chiptest] Fix UTF-8 decoding error in stdout/err

This commit also unifies PIPE type for Linux and Darwin platforms. From
now, all platforms will use PTY-based PIPE for reading subprocess output
in real time (without 4k block buffering).

* Fix typo in comment.

Co-authored-by: Boris Zbarsky <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Damian-Nordic Damian-Nordic Damian-Nordic approved these changes

@bzbarsky-apple bzbarsky-apple bzbarsky-apple approved these changes

@andy31415 andy31415 Awaiting requested review from andy31415

@austinh0 austinh0 Awaiting requested review from austinh0

@Byungjoo-Lee Byungjoo-Lee Awaiting requested review from Byungjoo-Lee

@chrisdecenzo chrisdecenzo Awaiting requested review from chrisdecenzo

@chshu chshu Awaiting requested review from chshu

@chulspro chulspro Awaiting requested review from chulspro

@dhrishi dhrishi Awaiting requested review from dhrishi

@electrocucaracha electrocucaracha Awaiting requested review from electrocucaracha

@emargolis emargolis Awaiting requested review from emargolis

@erjiaqing erjiaqing Awaiting requested review from erjiaqing

@gjc13 gjc13 Awaiting requested review from gjc13

@hawk248 hawk248 Awaiting requested review from hawk248

@harsha-rajendran harsha-rajendran Awaiting requested review from harsha-rajendran

@jelderton jelderton Awaiting requested review from jelderton

@jepenven-silabs jepenven-silabs Awaiting requested review from jepenven-silabs

@jmartinez-silabs jmartinez-silabs Awaiting requested review from jmartinez-silabs

@kpschoedel kpschoedel Awaiting requested review from kpschoedel

@lazarkov lazarkov Awaiting requested review from lazarkov

@lzgrablic02 lzgrablic02 Awaiting requested review from lzgrablic02

@mlepage-google mlepage-google Awaiting requested review from mlepage-google

@msandstedt msandstedt Awaiting requested review from msandstedt

@robszewczyk robszewczyk Awaiting requested review from robszewczyk

@saurabhst saurabhst Awaiting requested review from saurabhst

@selissia selissia Awaiting requested review from selissia

@tecimovic tecimovic Awaiting requested review from tecimovic

@turon turon Awaiting requested review from turon

@vijs vijs Awaiting requested review from vijs

@wbschiller wbschiller Awaiting requested review from wbschiller

@xylophone21 xylophone21 Awaiting requested review from xylophone21

@yufengwangca yufengwangca Awaiting requested review from yufengwangca

Assignees
No one assigned
Labels
review - approved scripts
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@arkq @bzbarsky-apple @Damian-Nordic @andy31415