Enhance HITL test #2209

HiFiPhile · 2023-08-06T14:48:01Z

Describe the PR
Add HITL testing script for newly added boards #2194

Currently support cdc_dual_ports, cdc_msc, dfu, dfu_runtime, more tests will be added later on.

For my instance use python test/hitl/hitl_test.py hitl_hfp.json

I'm not familiar with Github, @hathach could you take a look for CI integration ?

Additional context

Actually it catched something !
My lpcxpresso54608 (even modified 54628 to 54608 in board.mk) failed with cdc_msc test, msc block doesn't show up :

[242341.797154] usb 4-1.7: new high-speed USB device number 89 using xhci_hcd
[242341.901761] usb 4-1.7: New USB device found, idVendor=cafe, idProduct=4003, bcdDevice= 1.00
[242341.901765] usb 4-1.7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[242341.901767] usb 4-1.7: Product: TinyUSB Device
[242341.901768] usb 4-1.7: Manufacturer: TinyUSB
[242341.901768] usb 4-1.7: SerialNumber: 0123456789ABCDEF
[242341.904327] cdc_acm 4-1.7:1.0: ttyACM2: USB ACM device
[242341.904686] usb-storage 4-1.7:1.2: USB Mass Storage device detected
[242341.904901] scsi host7: usb-storage 4-1.7:1.2
[242343.001409] usb 4-1.7: reset high-speed USB device number 89 using xhci_hcd
[242343.102100] cdc_acm 4-1.7:1.0: ttyACM2: USB ACM device
[242343.205356] usb 4-1.7: reset high-speed USB device number 89 using xhci_hcd
[242343.306000] cdc_acm 4-1.7:1.0: ttyACM2: USB ACM device
[242343.401299] usb 4-1.7: reset high-speed USB device number 89 using xhci_hcd
[242343.505975] cdc_acm 4-1.7:1.0: ttyACM2: USB ACM device
[242343.605305] usb 4-1.7: reset high-speed USB device number 89 using xhci_hcd
[242343.705934] cdc_acm 4-1.7:1.0: ttyACM2: USB ACM device

Log:
msc.zip

hathach · 2023-08-07T04:11:39Z

@HiFiPhile yeah, I notice MSC is not showed up on your VM. I thought it is due your VM not enable auto mount block device, I remembered we could find it /dev/sda. I guess we have to manual mount ? I will check out the PR later.

HiFiPhile · 2023-08-07T09:06:14Z

I thought it is due your VM not enable auto mount block device

It's true. I set udev alias, it will show up with something like /dev/blkUSB_57323020.02

hathach · 2023-08-07T16:19:41Z

I thought it is due your VM not enable auto mount block device

It's true. I set udev alias, it will show up with something like /dev/blkUSB_57323020.02

we should actually read the contents of README.TXT to verify that it works. Give me a bit of time. I will add manual mount to this PR when possible.

HiFiPhile · 2023-08-07T17:03:32Z

we should actually read the contents of README.TXT to verify that it works.

Currently I read the content directly from block device.

tinyusb/test/hitl/hitl_test.py

Lines 109 to 115 in 87150b7

    
               readme = \ 
        
               b"This is tinyusb's MassStorage Class demo.\r\n\r\n\ 
        
           If you find any bugs or get any questions, feel free to file an\r\n\ 
        
           issue at github.com/hathach/tinyusb" 
        
               assert data[0x600:0x600 + len(readme)] == readme, 'Block wrong data' 
        
               print('cdc_msc test done')

I was too lazy to do mounting rightly to take care permissions and when multiples devices running msc_cdc example.

HiFiPhile · 2023-08-07T18:51:31Z

Now it read README.TXT from mounted device.

hathach · 2023-08-08T03:55:22Z

Now it read README.TXT from mounted device.

thank you perfect, I will check this out soon enough. Though one thing pop up in my mind. I think at the end of the CI. We should flash board_test example. This will disable the usb on the tested board. Making testing the next board easier, at least it will more forgiving in our test script.

hathach · 2023-08-08T14:43:59Z

lpc54 has an issue with receiving 31 bytes for msc command and sending 13 for msc status. Somehow when the transfer is complete, it is only 30 bytes (1 residue) and 12 bytes. Seem like the interrupt triggered a bit early or something. I have spent something troubleshoot, but not quite sure what is wrong. We probably need to skip the lpc54 for now, lpc55 has no issue with this though, strange.

HiFiPhile · 2023-08-08T14:55:18Z

We probably need to skip the lpc54 for now

Yes we can fix it later.

hathach · 2023-08-08T17:27:34Z

@HiFiPhile everything looks great and the python test script is way better than my shell script and easier to work with. Though I think we may have to modify it a bit to work better with ci. I will try to wrap this up tomorrow.

HiFiPhile · 2023-08-08T18:08:49Z

I was looking around for lpc54 issue and found there are 17 USB issues in errata sheet, that's a lot...

Maybe it's related:

* USB.1: In USB high-speed device mode, the NBytes field does not decrement after BULK OUT transfer
Introduction:
The LPC546xx. device family includes a USB high-speed interface (USB1) that can
operate in device mode at high-speed. The NBytes value represents the number of bytes
that can be received in the buffer.
* Problem:
If the buffer length is less than the maximum packet size and if the application code does
not program the maximum packet size, the NByte value is not correct.
* Work-around:
Program the NByte to the maximum packet size of that particular endpoint type. The
application code must calculate the received number of bytes by subtracting the NByte
value from the programming value. The software work-around is implemented on the SDK
software platform for the LPC546xx device family.

hathach · 2023-08-09T03:04:24Z

yeah, thank for the info, I should have checked errata. As tested, I also saw it in the bulk in as well (12 instead 13 bytes for msc status) which is more severe as there is no walkaround.

A quick look around, and there is tons of errata, tx data corruption 3.22 is also severe

I could add walkarond for bulk in/out so that it is kind of usable with stock examples. but I don't know if we have time to go through all of errata, look like usb on lpc54 is a headache one. I would definitely avoid lpc54 for a production with usb.

…B.1 and USB.2 msc is mounted, but device couldn't work reliably and got constant reset due to other errata probably.

hathach · 2023-08-09T09:53:44Z

test/hitl/hitl_test.py

+
+def test_cdc_msc(id):
+    port  = f'/dev/ttyUSB_{id[-8:]}.00'
+    file = f'/media/blkUSB_{id[-8:]}.02/README.TXT'


@HiFiPhile why would you limit the serial number to 8 characters, wouldn't be simpler to use it as it is ?

For some historical reason my udev script cut serial number to 8 characters.

In fact udev rule is not even needed, cdc can be accessed by /dev/serial/by-id/usb-TinyUSB_TinyUSB_Device_41003B000E504E5457323020-if00

yeah, I figure that out as well. I have already made the changes, just want double check if you first. The id is also shorten in blk device, if it only impact serial, then we are probably safe to remove the cut ?

def get_serial_dev(id, product, ifnum): # get usb serial by id return f'/dev/serial/by-id/usb-TinyUSB_{product}_{id}-if{ifnum:02d}' def get_disk_dev(id, lun): # get usb disk by id return f'/dev/disk/by-id/usb-TinyUSB_Mass_Storage_{id}-0:{lun}' def get_hid_dev(id, product, event): return f'/dev/input/by-id/usb-TinyUSB_{product}_{id}-{event}'

PS: I am trying to get the script also running with my pi4 :)

The id is also shorten in blk device, if it only impact serial, then we are probably safe to remove the cut ?

Yes you are right.

since I am in the middle of this, let me try to do this as well. At worst, we could revert if I messed up :)

…on ci with IAR build

hathach · 2023-08-09T12:01:31Z

I pushed my changes which include:

add lpc54608 board, similar to 54628
work-around for lpc54 mcu, msc disk appears but the mcu is too troublesome, it still reset randomly, and failed with hid_boot_interfaces, probably due to USB.15 (TX corruption) and/or other usb errata. So I decide it is better to not include lpc54 into the test farm since it will be a headache in the future.
update hitl_test script and json to simplify the json configuration. by default, all tests will be run. json can specify which specific tests with tests or which to skip with tests_skip.
integrate with ci, ci currently compile with cmake + IAR. Which we will use to flash
rename hitl to hil, since it seems to be more popular abbreviation

PS: I still need to do more work to update my pi4 with this excellent script. Though pi4 auto mount msc device, but that could be done later once this running well on your VM.

hathach · 2023-08-09T12:13:56Z

.github/workflows/build_iar.yml

-          name: stm32l4
-          path: |
-            *.elf
+          python3 test/hil/hil_test.py hil_hfp.json


this is where the script is run, CI compile with IAR using cmake build in previous step. We could pick up the elf. This will conveniently test IAR for us as well. (normally I only test with gcc)

hathach

everything seems to work well now, except for the lpc54 which has horrible usb hs implementation. The MSC still use the 8 byte ID ( didn't manage time to update that). But it is all good for the merge. We can update thing later with follow-up PRs. @HiFiPhile let me know if you are ok with my changes, and/or want to make more changes.

hathach · 2023-08-09T12:36:48Z

before I forgot here is the lpc54 setting, should we want to re-add it later (couldn't find way to comment out in json)

        {
            "name": "lpcxpresso54608",
            "uid": "0123456789ABCDEF",
            "debugger": "jlink",
            "debugger_sn": "727600775",
            "cpu": "LPC54608J512",
            "tests_skip": [
                "cdc_msc"
            ]
        }

HiFiPhile · 2023-08-09T12:52:25Z

Perfect it's good for me, maybe also let it run for GCC build.

hathach · 2023-08-09T13:20:43Z

Perfect it's good for me, maybe also let it run for GCC build.

yeah, that is possible as well, though I think we are good enough for now. Since we do generally test with gcc when developement. IAR is much rarer, so I am glad it is covered by HIL. Thank you very much for work and VM/board farm for doing actual test.

HiFiPhile added 4 commits August 6, 2023 13:41

Fix LPC54 build.

4f96eeb

WIP.

9dd8969

Fix LPC54 build.

b9e614b

Add more MCU.

4d1f3f4

HiFiPhile force-pushed the hitl branch from 207267c to 4d1f3f4 Compare August 6, 2023 15:01

HiFiPhile added 2 commits August 6, 2023 21:55

Fix RTT build.

c940a2f

Add config file as argument.

87150b7

Mount block device in test_cdc_msc.

c24f10e

added cmake for lpc54, update lpc55

6d877c3

HiFiPhile added 3 commits August 8, 2023 17:34

Add dummy board_test.

3444c4a

Flash board_test when done, skip cdc_msc for LPC54.

56bed1e

Update contributor.rst

bffe321

Add simple test for hid_boot_interface.

992e17f

hathach added 2 commits August 9, 2023 12:47

update to dcd ip3511 to add work-around for lpc54628 usb hs errata US…

b8b01c1

…B.1 and USB.2 msc is mounted, but device couldn't work reliably and got constant reset due to other errata probably.

Merge branch 'hitl' into pr_hitl

1abfa9a

hathach reviewed Aug 9, 2023

View reviewed changes

update hil test and json (remove troublesome lpc54608), run hil test …

1268d1a

…on ci with IAR build

rename hitl to hil

8e690df

hathach reviewed Aug 9, 2023

View reviewed changes

hathach approved these changes Aug 9, 2023

View reviewed changes

hathach merged commit 04f0cd5 into hathach:master Aug 9, 2023
40 checks passed

HiFiPhile deleted the hitl branch April 26, 2024 16:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance HITL test #2209

Enhance HITL test #2209

HiFiPhile commented Aug 6, 2023 •

edited

Loading

hathach commented Aug 7, 2023

HiFiPhile commented Aug 7, 2023

hathach commented Aug 7, 2023

HiFiPhile commented Aug 7, 2023

HiFiPhile commented Aug 7, 2023

hathach commented Aug 8, 2023

hathach commented Aug 8, 2023

HiFiPhile commented Aug 8, 2023

hathach commented Aug 8, 2023

HiFiPhile commented Aug 8, 2023

hathach commented Aug 9, 2023

hathach Aug 9, 2023

HiFiPhile Aug 9, 2023

hathach Aug 9, 2023 •

edited

Loading

HiFiPhile Aug 9, 2023

hathach Aug 9, 2023

hathach commented Aug 9, 2023 •

edited

Loading

hathach Aug 9, 2023

hathach left a comment

hathach commented Aug 9, 2023

HiFiPhile commented Aug 9, 2023

hathach commented Aug 9, 2023

Enhance HITL test #2209

Enhance HITL test #2209

Conversation

HiFiPhile commented Aug 6, 2023 • edited Loading

hathach commented Aug 7, 2023

HiFiPhile commented Aug 7, 2023

hathach commented Aug 7, 2023

HiFiPhile commented Aug 7, 2023

HiFiPhile commented Aug 7, 2023

hathach commented Aug 8, 2023

hathach commented Aug 8, 2023

HiFiPhile commented Aug 8, 2023

hathach commented Aug 8, 2023

HiFiPhile commented Aug 8, 2023

hathach commented Aug 9, 2023

hathach Aug 9, 2023

Choose a reason for hiding this comment

HiFiPhile Aug 9, 2023

Choose a reason for hiding this comment

hathach Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

HiFiPhile Aug 9, 2023

Choose a reason for hiding this comment

hathach Aug 9, 2023

Choose a reason for hiding this comment

hathach commented Aug 9, 2023 • edited Loading

hathach Aug 9, 2023

Choose a reason for hiding this comment

hathach left a comment

Choose a reason for hiding this comment

hathach commented Aug 9, 2023

HiFiPhile commented Aug 9, 2023

hathach commented Aug 9, 2023

HiFiPhile commented Aug 6, 2023 •

edited

Loading

hathach Aug 9, 2023 •

edited

Loading

hathach commented Aug 9, 2023 •

edited

Loading