rp2040: ep 0 out was already available #1764
Replies: 16 comments 10 replies
-
|
On further investigation, this seems to happen with some MIDI devices and unrelated to my USB MUX. For example, I cannot get a Novation Remote Zero SL to enumerate. |
Beta Was this translation helpful? Give feedback.
-
|
move to discussion since it is not clear too me if the bug is caused by tinyusb stack or user error |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if my issue is related, the title of this discussion fits well, but I have a different setup. I'm using two Pi Picos and I cloned the SDK and the pico-examples repositories, then compiled the examples with CMake on an Ubuntu 22.04LTS VM. The pico-examples contain the examples from tinyusb, so I programmed the example with and connected to the Pico with minicom. I'm seeing this when connecting a keyboard to the Pico: I tried using the "develop" branch of the pico-sdk/-examples, but that doesn't really help. Things get weirder: Sometimes I see I don't even understand how it prints the format string without replacing any values. I could give the exact sequence of commands I used to clone and build the examples, if that helps... |
Beta Was this translation helpful? Give feedback.
-
|
I've got the same problem, using TinyUSB on a custom rp2040 board and I don't even understand what the error message "ep 0 out was already available" means. If the endpoint is available, doesn't that mean that everything is good? Or should it be in use and not available. It's also not clear to me how to reproduce the bug consistently, sometimes my program works fine for a while, then suddenly panics with this message. I thought it was solved by upgrading the PicoSDK to version 1.5.0, after which the error disappeared for a while, but now its back and I have to clue how to approach it |
Beta Was this translation helpful? Give feedback.
-
|
The same on my side. Bellow logs from the same program with incorrect and correct USB startup: |
Beta Was this translation helpful? Give feedback.
-
|
Small updated. I cloned tinyusb from repository instead version from rp2040 and this error is gone. |
Beta Was this translation helpful? Give feedback.
-
|
I'm seeing this happen on a personal project with latest master as of 2024-02-14, but only while disconnecting and reconnecting USB using Usually this process works fine. But sometimes I get a panic on core 2. Here's the backtrace: Some additional debugging information: Huh, that's... interesting. Looks like buffer_control is null. I just noticed that. Just happens to be that there's the rp2040 ROM at 0x0, so there's some arbitrary data there. On my pico, that first byte of memory looks like a stack pointer for ROM, specifically At a higher level, what should happen if buffer_control is null? Is it just a bug? Or should So somehow, at least for my case, something must be zero-ing out the buffer_control pointer. Now it's just a matter of figuring out what. Since it looks like a race, it might be difficult to trigger with the slowdown incurred by gdb conditional breaks... |
Beta Was this translation helpful? Give feedback.
-
|
Interestingly enough, I set up the rp2040's MPUs on both cores to deny access to the first 256 bytes of memory, and I'm seeing tinyUSB fault very early on in my overall program, while my linux computer is trying to assign it a device address. Specifically, tinyUSB is trying to deference a null pointer again, but this time it's Here's that stack trace: and here's |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, there looks to be a (different?) bug here with This explicitly sets the |
Beta Was this translation helpful? Give feedback.
-
|
I managed to get past the As for the Placing a gdb watch on the
I haven't confirmed this yet, but my money is on some race condition between some ongoing tud_task transmission and this IRQ firing due to the disconnection. |
Beta Was this translation helpful? Give feedback.
-
|
I found something in the function So... the locking is not implemented. That definitely explains the bug I'm seeing. |
Beta Was this translation helpful? Give feedback.
-
|
Just for future reference, I posted PR #2474 that fixes some null-pointer dereferences, and adds a crappy locking implementation. This has been enough to fix the issue for me, although I swear there has to be a better implementation for this lock. I'm not familiar with the coding style for tinyusb, and I'm barely familiar with the spinlock usage/conventions with the rp2040. |
Beta Was this translation helpful? Give feedback.
-
|
I have that issue too on Pico W used as HID device. Also not 100% reproduceable so likely timing dependent. Related forum post: There is quite few people running into that panic. Would be nice if we could get an expert to investigate somehow or at least explain to us what could be going wrong so that we can try to workaround it. |
Beta Was this translation helpful? Give feedback.
-
|
@hathach @kilograham @kilograham @vmilea @P33M @majbthrd @ndinsmore This issue is impacting quite a few people it seems. @gemarcano did look into that quite a bit above but still we don't have a solution or a workaround I'm aware of.
What sort of user error would cause that panic? Are there any guidelines on how to use or not to use TinyUSB on Pico? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I've been traveling. I wrote a crappy critical section implementation for the rp2040. I posted a PR here: #2474 . I'm very convinced there's a better way, but this was enough to get the critical sections to work. Note that the PR also contains some bug-fixes for some null-pointer dereferences. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting, I went for one critical section per end point and it did not fix anything. I'll try having one global critical section like you did. |
Beta Was this translation helpful? Give feedback.
-
|
I implemented a critical section from |
Beta Was this translation helpful? Give feedback.
-
|
I tried #2474 critical section implementation and I'm pretty sure it would dead lock somehow. Also the null pointer part of it would just break my HID device mode, it would not start-up. Though it could be I did not apply it all properly, I think I missed the part that sets configured to true. While it did not fix things for me, it did help me understand my issue and workaround it. In my very specific case the HID device was sending periodically a vendor specific HID report, from Logitech actually, that would somehow cause the mount callback to be called multiple times when first connecting and later on too. Not sure if the host PC driver or Logitech client software did respond to that HID report in ways that would cause this remount. In any case the TinyUSB stack really does not like that and tends to panic shortly after connecting. Filtering out that doggy HID report fixes my issue. I guess you could say that's a user error somehow 😁 If anyone from Raspberry Pi or TinyUSB is interested to look into that just let me know. I could provide details or an example to reproduce it. Though I doubt any one would rush into that given the amount of enthusiasm that issue has received so far 🤣 |
Beta Was this translation helpful? Give feedback.
-
|
I am running into what I think is the same issue. Unfortunately I cannot reproduce it on my desk, but it happens frequently in the field. I've come to the conclusion that this must be a race condition somewhere within TinyUSB independently of this discussion. One additional piece of information I can offer is that in my case I also get another crash in another unrelated module in addition to the "ep 0 out/in not available" (and yes, I do get the panic on both OUT and IN endpoints). This other crash is due to a corrupted state variable and after inspecting the map file I found that this variable is located right after TinyUSB's event buffer (defined here). The PANICs started occurring after USB usage increased (i.e. more traffic), and the other crash started popping up exactly at the same time. So I do think they are related and that the event buffer overruns and corrupts some memory. This happens daily in the field based on device logs, but so far we have only seen it about 5 times in the last 4 months, so it's really hard to reproduce AND have instrumentation ready... I can't fully exclude user error, but our code is either based on existing examples or only uses the custom driver callbacks. We have a very simple round robin scheduler, so no preemption apart from the USB ISR, of course. Unfortunately I don't have more information to share, but this is becoming a big issue for us. So like others, I would appreciate any help with this. Specifically, has anyone found a way to force this issue? |
Beta Was this translation helpful? Give feedback.
-
|
Depends on what the issue exactly is, but for me I did identify the issue and I described it starting here in this same discussion #1764 (reply in thread) I have a crappy PR that fixes the issue for me: #2474 . It does two main things: 1. Implements locking. 2. Fixes some null-pointer dereferences. There's some additional small subtle changes in logic to make sure that I can figure out when a device has been reset and react accordingly. If your issue is the same as mine, then yes, it's a race condition. There's some on-going tud task that's working on a transmission, and if a disconnect happens that fires an IRQ that clears USB state. Without being able to properly lock the state, the tud task continues on its merry way even though everything has been nulled from under it, eventually blowing up. |
Beta Was this translation helpful? Give feedback.
-
|
As mentioned above I can fairly easily reproduce it at my desk. Though I don't think I can easily provide code to reproduce it. I my case you even need a specific hardware on top of proprietary code. Mine is also both in or out. They obviously have some nasty bug in the Pico port. Shame they are not willing to support the investigation. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks guys for the reply. In my case I don't get a disconnect event though, but the missing locking can definitely open up the door to mess things up even in the normal case. @Slion your issue sounds a bit different than @gemarcano's since his locking didn't resolve the issue for you. Since you can reproduce it easily, would you be willing to try to increase the event queue buffer and see if that makes a difference? You can do this by dropping the following into And can I assume that you both are on the latest TinyUSB and PICO SDK |
Beta Was this translation helpful? Give feedback.
-
That sounds a lot like mine. As mentioned above it happens to me when a string of remount events is somehow triggered by some specific HID communications. Which makes me think that I could possibly write a sample that sends that specific HID report and providing that you use that on a Windows PC possibly with the Logitech driver installed then it should repro. I probably won't bother doing that though unless some owners are telling us they want to help fix it. There is some valuable investigation done on this thread already but no one is giving any sign they want to help us fix it. I'm guessing they are all working on the next gen chip. |
Beta Was this translation helpful? Give feedback.
-
Sure enough they just launched the Pico 2. Including massive changes to the SDK. Not sure exactly what was changed in TinyUSB. In any case I'll wait for the W variant of the Pico 2 before trying it out. |
Beta Was this translation helpful? Give feedback.
-
|
You get the panic when you are lucky, more often than not it just crashes the board without panic. |
Beta Was this translation helpful? Give feedback.
-
|
See #2474 (comment)
Nope. I'm still on Pico SDK 1.5.1 but back when you asked it was almost the tip of master I think.
Sorry I still did not get to try that and I think I'll just stick to @gemarcano patch for now. |
Beta Was this translation helpful? Give feedback.
-
Operating System
Linux
Board
Custom RP2040 board
Firmware
https://github.com/rppicomidi/midi2usbhost custom version
What happened ?
I've built a custom board which has USB Type-C for Firmware Uploads (and during boot) and a USB Type-A which is switched to the D+/D- Pins with a FSUSB42 (and which power is controlled via MOSFET).
The select pin of the FSUSB42 is pulled low, so the device starts up with the USB-C being active for firmware upload.
Then the firmware puts SEL high, calls board_init() / tusb_init() as usual and enables VBUS. The USB-C is then only used for supplying the board with power.
Unfortunately and unexpectedly, the Type-A won't work because TinyUSB says the EP 0 out was already claimed.
I've tried a workaround with calling a usb reset before calling tusb, but that won't help.
How to reproduce ?
Debug Log as txt file
Screenshots
No response
I have checked existing issues, dicussion and documentation
Beta Was this translation helpful? Give feedback.
All reactions