esp32c3: debugging "stuck"/hanging interrupt handlers, maybe using NMI (or watchdog timer)?

[sethp]

Hello! We (@dougli1sqrd & I) have gotten ourselves into trouble in the usual way with concurrency, and have introduced some kind of a deadlock or similar in our project. In an effort to provide ourselves tools to debug this, I've been working on a way to print out a backtrace on a GPIO trigger (i.e. when we push the boot button on the dev board). It works (well, not for nested interrupts, but that's a later project), but unsurprisingly not when the code is in a critical_section (i.e. interrupts are disabled).

I found a few references to "non-maskable interrupts" (NMIs) in both the code and the esp32c3 reference manual, which sure seemed like they might help "see" into a critical section. But, the my attempt at setting the NMI enabled bit and providing a GPIO_NMI handler yielded the same results as before: the interrupt doesn't fire until we leave the critical_section.

So, some questions:

1. Am I missing something? There's fairly little information on how NMIs work on the esp32c3, and I'm more than a little curious if there's a different hardware flow/vector table or something that would (ought to) be invoked? The relevant section of the privileged risc-v spec seems to indicate that's expected, but I can't find the register definitions on the esp32c3 side that would map to "an implementation-defined NMI vector running in M-mode regardless of the state of a hart’s interrupt enable bits."
2. Should we be looking at using (one of) the watchdog timers instead? I found the description of the ESP-IDF's IWDT that seemed relevant, but I didn't quite surmise how we'd get at the core execution state with any of the stages besides interrupt (the various reset flavors seem like they'd clobber it), which would present the same "how to NMI" question? I suppose it might be possible to do something cute with the reset & the sleep/restore flow, maybe?
3. Generally, do y'all have guidance on how to investigate hanging/slow programs? I have the vaguest notion that JTAG might be a tool here, in combination with a pre-existing debugger, but is that true? Are there other common ways to "pause" and introspect a running embedded CPU?

[bjoernQ]

When we enter a critical-section we clear MIE in MSTATUS and no interrupts will be served (pretty much what it should do)

The only references to NMIs in the ESP32-C3 TRM I see are related to GPIO and it looks to me that those registers exist because the IP block is the same / similar to those used in the Xtensa based chips - but I don't really know t.b.h.

The linked section in the RISC-V spec sounds to me like they are talking about exceptions there (Non-maskable interrupts (NMIs) are only used for hardware error conditions) and indeed exceptions will be handled no matter what.

Maybe the watchdog in combination with DEBUG_ASSIST's PC logging feature could help a bit. The watchdog (if configured to reset) will reset the device even if the code got stuck inside a critical-section. Then on the next boot it is possible to get the PC at the time of the reset. But nothing else will be available - just the PC

I think your best bet here would be to use JTAG debugging. On ESP32-C3 (starting at revision 3) you won't even need additional hardware for that. You can use OpenOCD + GDB or probe.rs for that. Halting the CPU via the debug interface will also work inside critical-sections and the additional benefits from using a debugger is probably worth the effort of setting up things

[sethp]

Thanks for your reply! I especially appreciate the debugging tooling pointers/confirmation on watchdog capabilities; it looks like there are some details on debugging in the Rust on ESP book, so that'll probably be my next stop.

When we enter a critical-section we clear MIE in MSTATUS and no interrupts will be served (pretty much what it should do)

Yeah, as I understand it the goal of critical-section is to offer Rust's safety guarantees re: concurrency. The only form of concurrency present in a single-hart system would be interrupt handling, and you can't have e.g. a data race with only one participant.

The linked section in the RISC-V spec sounds to me like they are talking about exceptions there (Non-maskable interrupts (NMIs) are only used for hardware error conditions) and indeed exceptions will be handled no matter what.

Oh, I think I read that a little differently: it seems like you're interpreting that as "hardware errors [internal to the CPU]" but I read it as "hardware errors [external to the current hart execution flow]." There's also this, at the end of the section:

Unlike resets, NMIs do not reset processor state, enabling diagnosis, reporting, and possible containment of the hardware error.

And, certainly elsewhere NMIs have been used to enable system-level software debugging and profiling in addition to hardware error handling (I just learned that classic Macintosh systems had a physical button for, essentially, a hardware breakpoint trigger: how neat is that?). It does seem we're in good company in wondering what, precisely, an NMI counts as for RISC-V.

I suspect the intent of the "hardware error" language is to make it clear that the basic spec NMIs aren't intended to be "resumable" (i.e. returned from), as entry might clobber processor state (specifically: mepc). I'm inferring that last bit from the proposal to define a resumable NMI that's, essentially, copying all the trap registers (which is how SiFive seems to implement "RNMIs", at least).

Either way:

The only references to NMIs in the ESP32-C3 TRM I see are related to GPIO and it looks to me that those registers exist because the IP block is the same / similar to those used in the Xtensa based chips - but I don't really know t.b.h.

There's a couple of terms in here I'm not super familiar with in this context (I assume TRM is "Technical Reference Manual" and "IP block" is sort of like the hardware equivalent of a software package), but this makes me think the "NMI" language in the esp32c3 reference is, essentially, a hardware copypasta error? As in, the GPIO "chunk" (+registers) were brought over from a context that does have an NMI implementation (Xtensa) and dropped into one that does not (ESP's RISC-V implementation), but they didn't quite finish cleaning up the dangling references? Is there a......... hardware bug tracker? Or something?

[sethp]

Oh, it looks like there was also a "Resumable NMI Proposal" sent out to the RISC-V mailing list that's both clearer and more complete than that issue I linked.

[bjoernQ]

Your problem is most probably already solved - however leaving this here since it's quite useful:

On ESP32-C3 (and I guess all chips with debug-assist) the ROM bootloader will always enable PC logging and print the last seen PC (e.g. Saved PC:0x42002ff2) if available

The only important thing is you should use a TIMG watchdog since the RTC watchdog will also reset the logged PC