Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Guru Meditation Error: Core 0 panic'ed (LoadStoreAlignment) caused by ODR violation. (IDFGH-5506) #7236

Closed
b1ackviking opened this issue Jul 6, 2021 · 12 comments
Labels
Resolution: Done Issue is done internally Status: Done Issue is done internally

Comments

@b1ackviking
Copy link
Contributor

Environment

  • Development Kit: custom PCB
  • Module or chip used: ESP32-S2
  • IDF version: v4.4-dev-1849-g8e3e65a47
  • Build System: CMake
  • Compiler version: xtensa-esp32s2-elf-gcc (crosstool-NG esp-2021r1) 8.4.0
  • Operating System: Linux
  • Using an IDE?: No
  • Power Supply: external 3.3V

Problem Description

This commit (found with git bisect) defines the __cxx_eh_arena_size_get function which is defined in $HOME/.espressif/tools/xtensa-esp32s2-elf/esp-2021r1-8.4.0/xtensa-esp32s2-elf/bin/../lib/gcc/xtensa-esp32s2-elf/8.4.0/../../../../xtensa-esp32s2-elf/lib/no-rtti/libstdc++.a(eh_alloc.o).

This is a violation of the One Definition Rule, and it causes a crash due to unaligned memory access in the aes_ll_write_key function when I use external SPI RAM (ESP-PSRAM64H).

Reverting the commit fixes the bug.

Expected Behavior

Everything works.

Actual Behavior

The crash happens inside the aes_ll_write_key function during 1) WiFi initialization 2) establishing a secure network connection (with TLS).

Steps to reproduce

  1. Take ESP-IDF v4.4-dev-1849-g8e3e65a47
  2. Take devkit or PCB with ESP32-S2 and ESP-PSRAM64H.
  3. Activate PSRAM in the menuconfig and force malloc to use it (see sdkconfig.defaults below)
  4. Run the example located in examples/protocols/mqtt/ssl_mutual_auth (see also https://test.mosquitto.org/ssl/index.php)

Code to reproduce this issue

The example located in examples/protocols/mqtt/ssl_mutual_auth.

Debug Logs

NOTE: the line I (24) boot: ESP-IDF v4.4-dev-1849-g8e3e65a47-dirty 2nd stage bootloader says dirty because of the modified client.key and client.crt in the example. There are no other changes.

SPIWP:0xee
mode:DIO, clock div:1
load:0x3ffe6100,len:0x173c
load:0x4004c000,len:0xa04
load:0x40050000,len:0x2b54
entry 0x4004c1f4
I (24) boot: ESP-IDF v4.4-dev-1849-g8e3e65a47-dirty 2nd stage bootloader
I (24) boot: compile time 14:31:14
I (24) boot: chip revision: 0
I (37) boot.esp32s2: SPI Speed      : 80MHz
I (37) boot.esp32s2: SPI Mode       : DIO
I (38) boot.esp32s2: SPI Flash Size : 2MB
I (42) boot: Enabling RNG early entropy source...
I (48) boot: Partition Table:
I (51) boot: ## Label            Usage          Type ST Offset   Length
I (59) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (66) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (74) boot:  2 factory          factory app      00 00 00010000 00100000
I (81) boot: End of partition table
I (85) esp_image: segment 0: paddr=00010020 vaddr=3f000020 size=204d0h (132304) map
I (120) esp_image: segment 1: paddr=000304f8 vaddr=3ffc77e0 size=03120h ( 12576) load
I (123) esp_image: segment 2: paddr=00033620 vaddr=40024000 size=0c9f8h ( 51704) load
I (139) esp_image: segment 3: paddr=00040020 vaddr=40080020 size=866b0h (550576) map
I (249) esp_image: segment 4: paddr=000c66d8 vaddr=400309f8 size=06ddch ( 28124) load
I (256) esp_image: segment 5: paddr=000cd4bc vaddr=50000000 size=00010h (    16) load
I (266) boot: Loaded app from partition at offset 0x10000
I (266) boot: Disabling RNG early entropy source...
I (277) cache: Instruction cache 	: size 8KB, 4Ways, cache line size 32Byte
I (278) cache: Data cache 		: size 8KB, 4Ways, cache line size 32Byte
I (281) spiram: Found 64MBit SPI RAM device
I (286) spiram: SPI RAM mode: sram 40m
I (290) spiram: PSRAM initialized, cache is in normal (1-core) mode.
I (297) cpu_start: Pro cpu up.
I (1943) spiram: SPI SRAM memory test OK
I (1951) cpu_start: Pro cpu start user code
I (1951) cpu_start: cpu freq: 240000000
I (1951) cpu_start: Application information:
I (1954) cpu_start: Project name:     mqtt_ssl_mutual_auth
I (1960) cpu_start: App version:      v4.4-dev-1849-g8e3e65a47-dirty
I (1967) cpu_start: Compile time:     Jul  6 2021 14:31:13
I (1973) cpu_start: ELF file SHA256:  b90b8c00906ee4e8...
I (1979) cpu_start: ESP-IDF:          v4.4-dev-1849-g8e3e65a47-dirty
I (1986) heap_init: Initializing. RAM available for dynamic allocation:
I (1994) heap_init: At 3FF9E000 len 00002000 (8 KiB): RTCRAM
I (2000) heap_init: At 3FFCE580 len 0002DA80 (182 KiB): DRAM
I (2006) heap_init: At 3FFFC000 len 00003A10 (14 KiB): DRAM
I (2013) spiram: Adding pool of 8192K of external SPI memory to heap allocator
I (2021) spi_flash: detected chip: winbond
I (2025) spi_flash: flash io: dio
W (2029) spi_flash: Detected size(16384k) larger than the size in the binary image header(2048k). Using the size in the binary image header.
I (2046) cpu_start: Starting scheduler on PRO CPU.
I (2048) MQTTS_EXAMPLE: [APP] Startup..
I (2048) MQTTS_EXAMPLE: [APP] Free memory: 8576219 bytes
I (2058) MQTTS_EXAMPLE: [APP] IDF version: v4.4-dev-1849-g8e3e65a47-dirty
I (2078) wifi:wifi driver task: 3ffd4038, prio:23, stack:6656, core=0
I (2078) system_api: Base MAC address is not set
I (2078) system_api: read default base MAC address from EFUSE
I (2088) wifi:wifi firmware version: ff5f4ea
I (2088) wifi:wifi certification version: v7.0
I (2088) wifi:config NVS flash: enabled
I (2088) wifi:config nano formating: disabled
I (2098) wifi:Init data frame dynamic rx buffer num: 32
I (2098) wifi:Init management frame dynamic rx buffer num: 32
I (2108) wifi:Init management short buffer num: 32
I (2108) wifi:Init static tx buffer num: 16
I (2118) wifi:Init tx cache buffer num: 32
I (2118) wifi:Init static rx buffer size: 1600
I (2128) wifi:Init static rx buffer num: 10
I (2128) wifi:Init dynamic rx buffer num: 32
I (2128) wifi_init: rx ba win: 6
I (2138) wifi_init: tcpip mbox: 32
I (2138) wifi_init: udp mbox: 6
I (2148) wifi_init: tcp mbox: 6
I (2148) wifi_init: tcp tx win: 5744
I (2148) wifi_init: tcp rx win: 5744
I (2158) wifi_init: tcp mss: 1440
I (2158) wifi_init: WiFi/LWIP prefer SPIRAM
I (2168) wifi_init: WiFi IRAM OP enabled
I (2168) wifi_init: WiFi RX IRAM OP enabled
I (2178) example_connect: Connecting to IS-CAPsMAN2...
I (2178) phy_init: phy_version 1800,e7ef680,Apr 13 2021,11:45:08
I (2258) wifi:mode : sta (7c:df:a1:03:5e:88)
I (2258) wifi:enable tsf
I (2258) example_connect: Waiting for IP(s)
I (4308) wifi:new:<1,1>, old:<1,0>, ap:<255,255>, sta:<1,1>, prof:1
I (4868) wifi:state: init -> auth (b0)
I (4868) wifi:state: auth -> assoc (0)
I (4888) wifi:state: assoc -> run (10)
Guru Meditation Error: Core  0 panic'ed (LoadStoreAlignment). Exception was unhandled.

Core  0 register dump:
PC      : 0x400e4e81  PS      : 0x00060530  A0      : 0x800c26ad  A1      : 0x3ffd3dd0
0x400e4e81: aes_ll_write_key at /home/b1ack/projects/esp-idf/examples/protocols/mqtt/ssl_mutual_auth/build/../../../../../components/hal/esp32s2/include/hal/aes_ll.h:52
 (inlined by) aes_hal_setkey at /home/b1ack/projects/esp-idf/components/hal/aes_hal.c:27

A2      : 0x3f791226  A3      : 0x00000004  A4      : 0x00000000  A5      : 0x3f791236
A6      : 0x0000007b  A7      : 0x0000003d  A8      : 0x6003a000  A9      : 0x00000000
A10     : 0x3f791226  A11     : 0xfffffff0  A12     : 0x3ff9e9f4  A13     : 0x00000000
A14     : 0x3ffdec08  A15     : 0x00000004  SAR     : 0x00000008  EXCCAUSE: 0x00000009
EXCVADDR: 0x3f791226  LBEG    : 0x3ff9e9f4  LEND    : 0x00000000  LCOUNT  : 0x40026d19
0x40026d19: _xt_user_exc at /home/b1ack/projects/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:627



Backtrace:0x400e4e7e:0x3ffd3dd00x400c26aa:0x3ffd3e00 0x40092b3e:0x3ffd3e20 0x40092bb6:0x3ffd3e40 0x40092fac:0x3ffd3e60 0x400e6a40:0x3ffd3eb0 0x400e7051:0x3ffd3ef0 0x400d6eed:0x3ffd3f20 0x40032666:0x3ffd3f50 0x40032809:0x3ffd3fa0 0x40035d15:0x3ffd3fc0 0x40034173:0x3ffd3fe0 0x4002eab5:0x3ffd4010
0x400e4e7e: aes_ll_write_key at /home/b1ack/projects/esp-idf/examples/protocols/mqtt/ssl_mutual_auth/build/../../../../../components/hal/esp32s2/include/hal/aes_ll.h:52
 (inlined by) aes_hal_setkey at /home/b1ack/projects/esp-idf/components/hal/aes_hal.c:27

0x400c26aa: esp_aes_crypt_ecb at /home/b1ack/projects/esp-idf/components/mbedtls/port/aes/dma/esp_aes.c:659

0x40092b3e: aes_crypt at /home/b1ack/projects/esp-idf/components/wpa_supplicant/src/crypto/crypto_mbedtls.c:304

0x40092bb6: aes_decrypt at /home/b1ack/projects/esp-idf/components/wpa_supplicant/src/crypto/crypto_mbedtls.c:336

0x40092fac: aes_unwrap at /home/b1ack/projects/esp-idf/components/wpa_supplicant/src/crypto/aes-unwrap.c:61 (discriminator 3)

0x400e6a40: wpa_supplicant_decrypt_key_data at /home/b1ack/projects/esp-idf/components/wpa_supplicant/src/rsn_supp/wpa.c:1706

0x400e7051: wpa_sm_rx_eapol at /home/b1ack/projects/esp-idf/components/wpa_supplicant/src/rsn_supp/wpa.c:1941

0x400d6eed: sta_rx_eapol at ??:?

0x40032666: sta_input at ??:?

0x40032809: sta_rx_cb at ??:?

0x40035d15: ppRxPkt at ??:?

0x40034173: ppTask at ??:?

0x4002eab5: vPortTaskWrapper at /home/b1ack/projects/esp-idf/components/freertos/port/xtensa/port.c:168

Other items if possible

The sdkconfig.defaults file.

CONFIG_ESP32S2_SPIRAM_SUPPORT=y
CONFIG_SPIRAM_TYPE_AUTO=y
CONFIG_SPIRAM_SIZE=-1
CONFIG_SPIRAM=y
CONFIG_SPIRAM_BOOT_INIT=y
CONFIG_SPIRAM_USE_MALLOC=y
CONFIG_SPIRAM_MEMTEST=y
CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=0
CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=0
@espressif-bot espressif-bot added the Status: Opened Issue is new label Jul 6, 2021
@github-actions github-actions bot changed the title Guru Meditation Error: Core 0 panic'ed (LoadStoreAlignment) caused by ODR violation. Guru Meditation Error: Core 0 panic'ed (LoadStoreAlignment) caused by ODR violation. (IDFGH-5506) Jul 6, 2021
@espressif-bot espressif-bot added Status: In Progress Work is in progress and removed Status: Opened Issue is new labels Jul 7, 2021
@ESP-Marius
Copy link
Collaborator

Hi @b1ackviking thanks for reporting this.

libstdc++ __cxx_eh_arena_size_get is defined as weak function so this is probably not related to any ODR violation. Seems like buffer ends up unaligned.

We'll take a look at it.

@espressif-bot espressif-bot added Resolution: NA Issue resolution is unavailable Status: Done Issue is done internally Resolution: Done Issue is done internally and removed Status: In Progress Work is in progress Resolution: NA Issue resolution is unavailable labels Aug 27, 2021
@Alvin1Zhang
Copy link
Collaborator

Thanks for reporting, fix is available 3907634, feel free to reopen.

@b1ackviking
Copy link
Contributor Author

Hi @Alvin1Zhang and @ESP-Marius,
Thanks for fixing the issue with unaligned memory access in the aes_ll. However, there is a similar issue in another module:

Guru Meditation Error: Core  0 panic'ed (LoadStoreAlignment). Exception was unhandled.

Core  0 register dump:
PC      : 0x40167393  PS      : 0x00060f30  A0      : 0x8011bf26  A1      : 0x3ffea890
0x40167393: xor_data at /home/b1ack/projects/esp-idf/components/mbedtls/port/aes/esp_aes_gcm.c:113

A2      : 0x3f7b8f14  A3      : 0x3f7b4bf9  A4      : 0x00000000  A5      : 0x00000010
A6      : 0x00000010  A7      : 0x00000001  A8      : 0x8011a25b  A9      : 0x3ffea870
A10     : 0x3f4c004c  A11     : 0x00000000  A12     : 0x3ffe3220  A13     : 0x00000000
A14     : 0x00000001  A15     : 0x00000000  SAR     : 0x00000004  EXCCAUSE: 0x00000009
EXCVADDR: 0x3f7b4bf9  LBEG    : 0x3ffe3220  LEND    : 0x00000000  LCOUNT  : 0x40026f39
0x40026f39: _xt_user_exc at /home/b1ack/projects/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:633



Backtrace:0x40167390:0x3ffea8900x4011bf23:0x3ffea8b0 0x4011c24e:0x3ffea8e0 0x4011c2f2:0x3ffea940 0x4011c36b:0x3ffea960 0x40110509:0x3ffea9e0 0x4010958d:0x3ffeaa20 0x4010978d:0x3ffeaab0 0x40109868:0x3ffeaad0 0x40109a7a:0x3ffeaaf0 0x4010c873:0x3ffeab10 0x40108e01:0x3ffeab30 0x40108e3d:0x3ffeab50 0x4010583e:0x3ffeab70 0x40104be9:0x3ffeab90 0x4010501c:0x3ffeabb0 0x40105337:0x3ffeac30 0x40106bdb:0x3ffeac60 0x40166331:0x3ffeac90 0x400cf658:0x3ffeacb0 0x40031b65:0x3ffeace0
0x40167390: xor_data at /home/b1ack/projects/esp-idf/components/mbedtls/port/aes/esp_aes_gcm.c:110

0x4011bf23: esp_gcm_ghash at /home/b1ack/projects/esp-idf/components/mbedtls/port/aes/esp_aes_gcm.c:299

0x4011c24e: esp_aes_gcm_update at /home/b1ack/projects/esp-idf/components/mbedtls/port/aes/esp_aes_gcm.c:464

0x4011c2f2: esp_aes_gcm_crypt_and_tag_partial_hw at /home/b1ack/projects/esp-idf/components/mbedtls/port/aes/esp_aes_gcm.c:536

0x4011c36b: esp_aes_gcm_crypt_and_tag at /home/b1ack/projects/esp-idf/components/mbedtls/port/aes/esp_aes_gcm.c:569

0x40110509: mbedtls_cipher_auth_encrypt at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/cipher.c:1073

0x4010958d: ssl_encrypt_buf at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:1624

0x4010978d: mbedtls_ssl_write_record at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:3495

0x40109868: mbedtls_ssl_write_handshake_msg at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:3418

0x40109a7a: mbedtls_ssl_write_finished at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:6615

0x4010c873: mbedtls_ssl_handshake_client_step at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/ssl_cli.c:3875

0x40108e01: mbedtls_ssl_handshake_step at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:8189

0x40108e3d: mbedtls_ssl_handshake at /home/b1ack/projects/esp-idf/components/mbedtls/mbedtls/library/ssl_tls.c:8213

0x4010583e: esp_mbedtls_handshake at /home/b1ack/projects/esp-idf/components/esp-tls/esp_tls_mbedtls.c:138

0x40104be9: esp_tls_handshake at /home/b1ack/projects/esp-idf/components/esp-tls/esp_tls.c:76

0x4010501c: esp_tls_low_level_conn at /home/b1ack/projects/esp-idf/components/esp-tls/esp_tls.c:423 (discriminator 15)

0x40105337: esp_tls_conn_new_sync at /home/b1ack/projects/esp-idf/components/esp-tls/esp_tls.c:485

0x40106bdb: ssl_connect at /home/b1ack/projects/esp-idf/components/tcp_transport/transport_ssl.c:119

0x40166331: esp_transport_connect at /home/b1ack/projects/esp-idf/components/tcp_transport/transport.c:181

0x400cf658: esp_mqtt_task at /home/b1ack/projects/esp-idf/components/mqtt/esp-mqtt/mqtt_client.c:1381

0x40031b65: vPortTaskWrapper at /home/b1ack/projects/esp-idf/components/freertos/port/xtensa/port.c:159

The problem is in the xor_data function:

/* Function to xor two data blocks */
static void xor_data(uint8_t *d, const uint8_t *s)
{
uint32_t *dst = (uint32_t *) d;
uint32_t *src = (uint32_t *) s;
*dst++ ^= *src++;
*dst++ ^= *src++;
*dst++ ^= *src++;
*dst++ ^= *src++;
}

Should be:

/* Function to xor two data blocks */
static void xor_data(uint8_t *dst, const uint8_t *src)
{
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
    *dst++ ^= *src++;
}

Will you be able to fix that ASAP yourself, or would it be better if I open a PR for this fix?

@b1ackviking
Copy link
Contributor Author

Also, @Alvin1Zhang, I can not reopen the issue because of the GitHub rules (I am not the person who closed it). Will you reopen it, or should I create another one and add a reference?

@Alvin1Zhang Alvin1Zhang reopened this Aug 30, 2021
@ESP-Marius
Copy link
Collaborator

@b1ackviking Ah, thanks. We'll have to take a more thorough look then and some more tests to root out all of these issues.

Sorry for your inconvenience.

@AxelLin
Copy link
Contributor

AxelLin commented Sep 6, 2021

Thanks for reporting, fix is available 3907634, feel free to reopen.

Hi @ESP-Marius

I'm wondering if v4.3 branch requires this fix or not.
(The fix cannot cleanly apply to v4.3, so if v4.3 needs the fix I need to wait for the backport.)

@ESP-Marius
Copy link
Collaborator

@AxelLin Yes, v4.3 will have the same issue, we are backporting it. This only seems to cause issues when using large PSRAMs though, so if you are not using this then you shouldn't see any issues.

@AxelLin
Copy link
Contributor

AxelLin commented Sep 6, 2021

@AxelLin Yes, v4.3 will have the same issue, we are backporting it. This only seems to cause issues when using large PSRAMs though, so if you are not using this then you shouldn't see any issues.

ok, thanks for the clarify. (I don't use PSRAM).

@espressif-bot espressif-bot added Status: Opened Issue is new and removed Resolution: Done Issue is done internally Status: Done Issue is done internally labels Sep 9, 2021
@espressif-bot espressif-bot added Resolution: NA Issue resolution is unavailable Status: Done Issue is done internally Resolution: Done Issue is done internally and removed Status: Opened Issue is new Resolution: NA Issue resolution is unavailable labels Nov 4, 2021
@AxelLin
Copy link
Contributor

AxelLin commented Nov 11, 2021

@AxelLin Yes, v4.3 will have the same issue, we are backporting it. This only seems to cause issues when using large PSRAMs though, so if you are not using this then you shouldn't see any issues.

@ESP-Marius
Just curious why it took such long time to backport the fix to v4.3? (maybe it got lost?)

@mahavirj
Copy link
Member

@AxelLin Remaining fix (per #7236 (comment)) was recently added with 1cb07a1. We missed out on adding reference to this issue. Backport MRs combining both these fixes are already in final approval stage, they should appear on github in next few days.

@Alvin1Zhang
Copy link
Collaborator

Thanks for reporting, sorry for slow turnaround, fix on release/4.3 is available at 79b5994 and fix on release/4.2 is available at a7a495f, feel free to reopen.

@ugurakas
Copy link

ugurakas commented Feb 2, 2023

Did anyone find a solution for this problem, ı'm getting the same error at version 5.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Resolution: Done Issue is done internally Status: Done Issue is done internally
Projects
None yet
Development

No branches or pull requests

7 participants