esp-idf coredump to flash, esp32 crashes, and hung up, restarting failed (IDFGH-5433)

[maqian-cn]

Environment

• Development Kit: [none]
• Kit version (for WroverKit/PicoKit/DevKitC): [none]
• Module or chip used: [ESP32-WROOM-32]
• IDF version (run git describe --tags to find it):
  // v3.3.4
• Build System: [idf.py]
• Compiler version (run xtensa-esp32-elf-gcc --version to find it):
  // xtensa-esp32-elf-gcc (crosstool-NG crosstool-ng-1.22.0-96-g28523988) 5.2.0
• Operating System: [Windows]
• (Windows only) environment type: [ESP Command Prompt].
• Using an IDE?: [No]
• Power Supply: [external 5V]

Problem Description

esp32 crashes, coredump to flash successed, and restarting failed, died.

Expected Behavior

restarting successful.

Actual Behavior

restarting failed, died.

Steps to reproduce

1. waiting for crashes
2. a certain probability that it will appear

Code to reproduce this issue

Debug Logs

coredump-log.txt

Other items if possible

• sdkconfig file (attach the sdkconfig file from your project folder)
• elf file in the build folder (note this may contain all the code details and symbols of your project.)
• coredump (This provides stacks of tasks.)

[igrr]

@maqian-cn Could you please try latest release/v3.3 branch and see if the issue is resolved? I see there have been some fixes related to core dump since v3.3.4 release.

If it still fails, would you be able to trim down your application to a small program to reproduce the issue, and share that along with your sdkconfig file?

[maqian-cn]

@igrr thank u very much for reply, I'll try v3.3, and check it.

[maqian-cn]

according to your reply, based on tags v3.3.4, I did "git checkout release/v3.3" and succeed. But I do "git describe", showed version is "v3.3.4-209-g66d3783c8" instead of "release/v3.3". I don't know if this is correct.

And, another problem, please view the #6124.
I tried the assert failure, it is code below.

		if ((3 * 60 * pdMS_TO_TICKS(1000)) <= abs(xTaskGetTickCount() - xPrintTimeLogTick))
		{
			xPrintTimeLogTick = xTaskGetTickCount();
			memset(time, 0, sizeof(time));
			Common_PrintTime(time, sizeof(time));
			ESP_LOGI(TAG, "Time: %s", time);
			ESP_LOGI(TAG, "Minimum Free memory: %d bytes", esp_get_minimum_free_heap_size());
			ESP_LOGI(TAG, "Free memory: %d bytes", esp_get_free_heap_size());
			assert(0);
#if (E_SYSCONFIG_ONENET_DATASENDT_STATISTICS)
			Ctrl_OneNetDataSendStatistics(pCtrlObjHandle, time);
#endif
#if (E_SYSCONFIG_TASK_STATISTICS)
			Ctrl_AllTaskStatistics();
#endif
		}

And, got the coredump info is below.

===============================================================
==================== ESP32 CORE DUMP START ====================

================== CURRENT THREAD REGISTERS ===================
pc             0x400880d3       0x400880d3 <invoke_abort+27>
lbeg           0x400014fd       1073747197
lend           0x4000150d       1073747213
lcount         0xfffffffd       4294967293
sar            0x4      4
ps             0x60720  395040
threadptr      <unavailable>
br             <unavailable>
scompare1      <unavailable>
acclo          <unavailable>
acchi          <unavailable>
m0             <unavailable>
m1             <unavailable>
m2             <unavailable>
m3             <unavailable>
expstate       <unavailable>
f64r_lo        <unavailable>
f64r_hi        <unavailable>
f64s           <unavailable>
fcr            <unavailable>
fsr            <unavailable>
a0             0x400883ac       1074299820
a1             0x3ffcb360       1073525600
a2             0x80040004       -2147221500
a3             0x2f     47
a4             0xc      12
a5             0xffffffff       -1
a6             0x0      0
a7             0xfffffffe       -2
a8             0x0      0
a9             0x3ffcb340       1073525568
a10            0x0      0
a11            0x3ffcb2f1       1073525489
a12            0x3ffcb29f       1073525407
a13            0x31     49
a14            0x0      0
a15            0x3ffcb2a2       1073525410

==================== CURRENT THREAD STACK =====================
#0  0x400880d3 in invoke_abort () at D:/esp-idf/components/esp32/panic.c:156
#1  0x400883ac in abort () at D:/esp-idf/components/esp32/panic.c:171
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

======================== THREADS INFO =========================
  Id   Target Id         Frame
  19   process 18        0x4008a75c in xQueueGenericReceive (xQueue=0x3ffafae8, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at D:/esp-idf/components/freertos/queue.c:1592
  18   process 17        0x4008a75c in xQueueGenericReceive (xQueue=0x3ffaeab0, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at D:/esp-idf/components/freertos/queue.c:1592
  17   process 16        0x4008a75c in xQueueGenericReceive (xQueue=0x3ffcd4f8, pvBuffer=0x3ffce970, xTicksToWait=4294967295, xJustPeeking=0) at D:/esp-idf/components/freertos/queue.c:1592
  16   process 15        0x4008a75c in xQueueGenericReceive (xQueue=0x3ffafccc, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at D:/esp-idf/components/freertos/queue.c:1592
  15   process 14        0x4008a75c in xQueueGenericReceive (xQueue=0x3ffd9034, pvBuffer=0x3ffd7c00, xTicksToWait=4294967295, xJustPeeking=0) at D:/esp-idf/components/freertos/queue.c:1592
  14   process 13        0x4008a75c in xQueueGenericReceive (xQueue=0x3ffc787c, pvBuffer=0x3ffcd18c, xTicksToWait=4294967295, xJustPeeking=0) at D:/esp-idf/components/freertos/queue.c:1592
  13   process 12        0x4008a75c in xQueueGenericReceive (xQueue=0x3ffd5dd0, pvBuffer=0x3ffe1d9c, xTicksToWait=1, xJustPeeking=0) at D:/esp-idf/components/freertos/queue.c:1592
  12   process 11        0x4000bff0 in ?? ()
  11   process 10        0x4008b372 in vTaskDelay (/builds/idf/crosstool-NG/.build/src/gdb-7.10/gdb/inline-frame.c:171: internal-error: inline_frame_this_id: Assertion `!frame_id_eq (*this_id, outer_frame_id)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) [answered Y; input not from terminal]
/builds/idf/crosstool-NG/.build/src/gdb-7.10/gdb/inline-frame.c:171: internal-error: inline_frame_this_id: Assertion `!frame_id_eq (*this_id, outer_frame_id)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) [answered Y; input not from terminal]
Reading symbols from .\\build\\ec600s-esp32-pg.elf...done.
[New <main task>]
[New process 1]
[New process 2]
[New process 3]
[New process 4]
[New process 5]
[New process 6]
[New process 7]
[New process 8]
[New process 9]
[New process 10]
[New process 11]
[New process 12]
[New process 13]
[New process 14]
[New process 15]
[New process 16]
[New process 17]
[New process 18]
[Current thread is 1 (<main task>)]

======================= ALL MEMORY REGIONS ========================
Name   Address   Size   Attrs
.rtc.text 0x400c0000 0x0 RW
.rtc.dummy 0x3ff80000 0x0 RW
.rtc.force_fast 0x3ff80000 0x0 RW
.rtc_noinit 0x50000000 0x0 RW
.rtc.force_slow 0x50000000 0x0 RW
.iram0.vectors 0x40080000 0x400 R XA
.iram0.text 0x40080400 0x1655c R XA
.dram0.data 0x3ffb0000 0x49c4 RW A
.noinit 0x3ffb49c4 0x0 RW
.flash.rodata 0x3f400020 0x346f8 RW A
.flash.text 0x400d0018 0xac714 R XA
.coredump.tasks.data 0x3ffcb4c4 0x16c RW
.coredump.tasks.data 0x3ffcb2a0 0x210 RW
.coredump.tasks.data 0x3ffc8544 0x16c RW
.coredump.tasks.data 0x3ffc83a0 0x190 RW
.coredump.tasks.data 0x3ffc8cd0 0x16c RW
.coredump.tasks.data 0x3ffc8b20 0x19c RW
.coredump.tasks.data 0x3ffc9b64 0x16c RW
.coredump.tasks.data 0x3ffc9a00 0x150 RW
.coredump.tasks.data 0x3ffdfcbc 0x16c RW
.coredump.tasks.data 0x3ffe0eb0 0x1b0 RW
.coredump.tasks.data 0x3ffde364 0x16c RW
.coredump.tasks.data 0x3ffde110 0x240 RW
.coredump.tasks.data 0x3ffc7630 0x16c RW
.coredump.tasks.data 0x3ffcc240 0x1fc RW
.coredump.tasks.data 0x3ffd5c54 0x16c RW
.coredump.tasks.data 0x3ffd57d0 0x470 RW
.coredump.tasks.data 0x3ffd6d00 0x16c RW
.coredump.tasks.data 0x3ffd6b40 0x1ac RW
.coredump.tasks.data 0x3ffe2e94 0x16c RW
.coredump.tasks.data 0x3ffe2c80 0x200 RW
.coredump.tasks.data 0x3ffd8d10 0x16c RW
.coredump.tasks.data 0x3ffd8b50 0x1ac RW
.coredump.tasks.data 0x3ffd9a98 0x16c RW
.coredump.tasks.data 0x3ffd98d0 0x1b4 RW
.coredump.tasks.data 0x3ffdfe38 0x16c RW
.coredump.tasks.data 0x3ffe1c80 0x1f0 RW
.coredump.tasks.data 0x3ffcd260 0x16c RW
.coredump.tasks.data 0x3ffcd080 0x1cc RW
.coredump.tasks.data 0x3ffd7d84 0x16c RW
.coredump.tasks.data 0x3ffd7ad0 0x2a0 RW
.coredump.tasks.data 0x3ffafd34 0x16c RW
.coredump.tasks.data 0x3ffc6630 0x188 RW
.coredump.tasks.data 0x3ffcea18 0x16c RW
.coredump.tasks.data 0x3ffce850 0x1b4 RW
.coredump.tasks.data 0x3ffaf728 0x16c RW
.coredump.tasks.data 0x3ffaf580 0x194 RW
.coredump.tasks.data 0x3ffafb50 0x16c RW
.coredump.tasks.data 0x3ffc6020 0x188 RW

===================== ESP32 CORE DUMP END =====================
===============================================================
Done!

==================== CURRENT THREAD STACK =====================
#0 0x400880d3 in invoke_abort () at D:/esp-idf/components/esp32/panic.c:156
#1 0x400883ac in abort () at D:/esp-idf/components/esp32/panic.c:171
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

why Backtrace stopped? It is the incomplete Backtrace info. @igrr

[igrr]

Yes, this is the same issue as #6124. We don't have a solution for it yet, unfortunately.

[maqian-cn]

oh, it is really bad. Will this problem be solved in tags v3.x.x later? Because this function is still very useful when debugging. @igrr

[maqian-cn]

I'v tried tags/v3.3.5. There is also the same problem of incomplete info. @igrr

[igrr]

That's correct, the issue #6124 occurs in all releases and branches. I'm sorry but it's unlikely it will be resolved soon.

Is the issue you had originally with the chip not restarting after coredump solved now?

[maqian-cn]

@igrr
First of all， Thank you very much for your prompt reply .

OK, I'm only waiting for the incomplete Backtrace info problem to be resolved.

The problem that the chip is not restarting after coredump is likely resolved.
according to your suggestion, I got release/v3.3 branch by "git fetch" and "git checkout release/v3.3", but showed version is "v3.3.4-209-g66d3783c8" by "git describe -- tags", I don't know if this version is correct.

Based on this branch, The problem not restarting has not occurred until now. If the follow-up happens, I will report it to you again.

[igrr]

Since all issues except #6124 have been resolved, I will close this issue. We will update #6124 once the fix is released.