Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PSRAM init fault when PSRAM does not exist and ignore not-found is set (IDFGH-4204) #6063

Closed
ryankurte opened this issue Nov 3, 2020 · 9 comments

Comments

@ryankurte
Copy link
Contributor

ryankurte commented Nov 3, 2020

When CONFIG_SPIRAM_IGNORE_NOTFOUND=y is set initialising a device with SPIRAM enabled causes a fault (rather than continuing to boot without enabling the additional PSRAM)

Environment

  • Development Kit: [ESP32-Wrover-Kit|ESP32-DevKitC|ESP32-PICO-Kit|ESP32-LyraT|ESP32-LyraTD-MSC|none]
  • Kit version (for WroverKit/PicoKit/DevKitC): n/a
  • Module or chip used: [ESP32-WROOM-32D]
  • IDF version (run git describe --tags to find it): v4.2-dev-2182-gdddcc2ede
  • Build System: [Make|CMake|idf.py]
  • Compiler version (run xtensa-esp32-elf-gcc --version to find it): xtensa-esp32-elf-gcc (crosstool-NG esp-2020r2) 8.2.0
  • Operating System: [Windows|Linux|macOS]
  • Using an IDE?: [No|Yes (please give details)]
  • Power Supply: [USB|external 5V|external 3.3V|Battery]

Problem Description

Configuring SPIRAM with auto-detect and ignore-notfound should (i believe) attempt to check the PSRAM exists and continue booting if not, instead this panics at esp_spiram_get_chip_size.

SPIRAM KConfig fields:

+CONFIG_ESP32_SPIRAM_SUPPORT=y
+
+#
+# SPI RAM config
+#
+CONFIG_SPIRAM_TYPE_AUTO=y
+# CONFIG_SPIRAM_TYPE_ESPPSRAM16 is not set
+# CONFIG_SPIRAM_TYPE_ESPPSRAM32 is not set
+# CONFIG_SPIRAM_TYPE_ESPPSRAM64 is not set
+CONFIG_SPIRAM_SIZE=-1
+CONFIG_SPIRAM_SPEED_40M=y
+CONFIG_SPIRAM=y
+CONFIG_SPIRAM_BOOT_INIT=y
+CONFIG_SPIRAM_IGNORE_NOTFOUND=y
+# CONFIG_SPIRAM_USE_MEMMAP is not set
+# CONFIG_SPIRAM_USE_CAPS_ALLOC is not set
+CONFIG_SPIRAM_USE_MALLOC=y
+CONFIG_SPIRAM_MEMTEST=y
+CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=16384
+CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
+CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=32768
+# CONFIG_SPIRAM_ALLOW_BSS_SEG_EXTERNAL_MEMORY is not set
+CONFIG_SPIRAM_CACHE_WORKAROUND=y

Expected Behavior

The device starts, attempts to detect SPIRAM and fails, then continues booting with available onboard memory.

Actual Behavior

The device detects RAM was not available, continues, and appears to fault while trying to fetch the SPIRAM size. See the logs below.

Steps to reproduce

  • Enable SPIRAM on a non-supported platform (or platform without SPIRAM)

Debug Logs

I (886) psram: This chip is ESP32-D0WD
E (892) psram: PSRAM ID read error: 0xffffffff
E (896) spiram: SPI RAM enabled but initialization failed. Bailing out.
I (903) cpu_start: Failed to init external RAM; continuing without it.
E (910) spiram: SPI RAM not initialized
Re-enable cpu cache.

abort() was called at PC 0x400d4faa on core 0
0x400d4faa: esp_spiram_get_chip_size at /opt/esp/idf/components/esp32/spiram.c:132 (discriminator 1)


Backtrace:0x40095b22:0x3ffe3b30 0x40096201:0x3ffe3b50 0x4009a939:0x3ffe3b70 0x400d4faa:0x3ffe3be0 0x400d5043:0x3ffe3c00 0x40082856:0x3ffe3c20 0x40082176:0x3ffe3c40 0x400793a2:0x3ffe3c80 |<-CORRUPTED
0x40095b22: panic_abort at /opt/esp/idf/components/esp_system/panic.c:330

0x40096201: esp_system_abort at /opt/esp/idf/components/esp_system/system_api.c:106

0x4009a939: abort at /opt/esp/idf/components/newlib/abort.c:46

0x400d4faa: esp_spiram_get_chip_size at /opt/esp/idf/components/esp32/spiram.c:132 (discriminator 1)

0x400d5043: esp_spiram_get_size at /opt/esp/idf/components/esp32/spiram.c:220

0x40082856: esp_spiram_init_cache at /opt/esp/idf/components/esp32/spiram.c:117

0x40082176: call_start_cpu0 at /opt/esp/idf/components/esp32/cpu_start.c:189

Notes

  • It would be viable to work around this if it was plausible to init / mount the PSRAM at runtime via an API? Whether this was possible to me was not clear from the API documentation.
  • Enabling PSRAM on a platform with this supported always overflows IRAM and requires wifi optimisations disabled (per Upgrading v3.3 to v4.0 with PSRAM, app no longer fits in IRAM (IDFGH-3664) #5598), it may be nice to document this.
@github-actions github-actions bot changed the title PSRAM init fault when PSRAM does not exist and ignore not-found is set PSRAM init fault when PSRAM does not exist and ignore not-found is set (IDFGH-4204) Nov 3, 2020
@Spritetm
Copy link
Member

Spritetm commented Nov 3, 2020

Fix is incoming, thanks for reporting.

For now: I think you should be able to work around this indeed by disabling SPIRAM_BOOT_INIT, then calling esp_spiram_init() and if that succeeds, calling esp_spiram_init_cache, esp_spiram_test, esp_spiram_add_to_heapalloc and heap_caps_malloc_extmem_enable. (Although that won't give 100% the same effect, as some things that would have been allocated in psram will be in internal memory then.)

@ryankurte
Copy link
Contributor Author

Thanks for the response! I tried this but it appears that whenever I enable CONFIG_ESP32_SPIRAM_SUPPORT, with SPIRAM_BOOT_INIT=n and without doing anything else, malloc and free stop working in interesting ways (typically in esp wifi/ble or lwip modules). This appears to be independent of MEMMAP/MALLOC/CAPS_ALLOC configuration.

I haven't extracted it from our application code but I believe this failure should be reproduced by enabling bluetooth support in menuconfig then instead of configuring it calling esp_bt_mem_release(ESP_BT_MODE_BTDM); from main.

@Spritetm
Copy link
Member

Spritetm commented Nov 4, 2020

Strange, I'll see if I can look into this.

The fix for the original issue, by the way, is an one-liner. In components/esp_system/port/cpu_start.c, around line 330, there is a line that says esp_spiram_init_cache();. Change this to if (g_spiram_ok) esp_spiram_init_cache(); and the crash should be gone. If you're okay with running a modified version of esp-idf until the change has trickled through to master, you can edit it locally.

@ryankurte
Copy link
Contributor Author

thanks for all the quick responses! we're currently running against a patched-to-heck version of the dockerhub release-v4.2 tag, no problem adding more, though i guess the fixes to this may not make it back to this series for a while? (impediments to running against release-v4.3 are that that does not appear to be published to dockerhub, and some change to master recently ate a whole lot of RAM we didn't have to spare so we're no longer up for latest).

applying the suggested patch to v4.2-dev-2182-gdddcc2ede with SPIRAM autodetect we now get through init but still see a bunch of malloc failures, regardless of the SPIRAM_USE_ mode. A sample:

Jan 01 00:00:02 CRIT coap_new_string: malloc
E (1517) wifi:fail to alloc timer, type=8
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x40126f63  PS      : 0x00060830  A0      : 0x8012719a  A1      : 0x3ffbc4d0  
0x40126f63: coap_get_resource_from_uri_path at /opt/esp/idf/components/coap/libcoap/src/resource.c:518

A2      : 0x3fff6774  A3      : 0x00000000  A4      : 0x3fffff78  A5      : 0x3fffffec  
A6      : 0x00007367  A7      : 0x00000005  A8      : 0x80127e58  A9      : 0x3ffbc4b0  
A10     : 0x00000000  A11     : 0x00000000  A12     : 0x00060620  A13     : 0x00060623  
A14     : 0x000000fe  A15     : 0x00000001  SAR     : 0x0000000a  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000004  LBEG    : 0x40093415  LEND    : 0x40093425  LCOUNT  : 0xfffffffe  
0x40093415: strlen at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/machine/xtensa/strlen.S:84

0x40093425: strlen at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/machine/xtensa/strlen.S:96


Backtrace:0x40126f60:0x3ffbc4d0 0x40127197:0x3ffbc4f0 0x401206fd:0x3ffbc510 0x400d90c6:0x3ffbc530 0x400d463a:0x3ffbc970 0x400960c1:0x3ffbc9a0
0x40126f60: coap_get_resource_from_uri_path at /opt/esp/idf/components/coap/libcoap/src/resource.c:515

0x40127197: coap_add_resource at /opt/esp/idf/components/coap/libcoap/src/resource.c:459

0x401206fd: COAP_MGR_bind_resource at /work/build/../modules/drivers/coap_mgr/coap_mgr.c:209

0x400d90c6: app_main at /work/build/../main/app_main.c:545 (discriminator 9)

0x400d463a: main_task at /opt/esp/idf/components/esp32/cpu_start.c:592

0x400960c1: vPortTaskWrapper at /opt/esp/idf/components/freertos/xtensa/port.c:143

The v4.2 patch:

diff --git a/components/esp32/cpu_start.c b/components/esp32/cpu_start.c
index 791a3b0c4..f9cb6563c 100644
--- a/components/esp32/cpu_start.c
+++ b/components/esp32/cpu_start.c
@@ -186,7 +186,10 @@ void IRAM_ATTR call_start_cpu0(void)
         abort();
 #endif
     }
-    esp_spiram_init_cache();
+    
+    if (s_spiram_okay) {
+        esp_spiram_init_cache();
+    }
 #endif
 
     ESP_EARLY_LOGI(TAG, "Pro cpu up.");

The v4.3 version (in case it's useful to anyone else):

diff --git a/components/esp_system/port/cpu_start.c b/components/esp_system/port/cpu_start.c
index 92080e000..289411f1d 100644
--- a/components/esp_system/port/cpu_start.c
+++ b/components/esp_system/port/cpu_start.c
@@ -276,7 +276,10 @@ void IRAM_ATTR call_start_cpu0(void)
         abort();
 #endif
     }
-    esp_spiram_init_cache();
+
+    if (g_spiram_ok) {
+	esp_spiram_init_cache();
+    }
 #endif
 
 #if !CONFIG_ESP_SYSTEM_SINGLE_CORE_MODE
``

@ryankurte
Copy link
Contributor Author

Okay so enabling the PSRAM on a board that has it also appears to cause failures in mbedtls, lwip, and with the bluetooth stack... this is definitely not going as well as I had hoped :-/

@ryankurte
Copy link
Contributor Author

ryankurte commented Nov 20, 2020

right so the culprit is also CONFIG_SPIRAM_IGNORE_NOTFOUND. when enabled on a device that has SPIRAM this appears to cause anything that is able to allocate to SPIRAM (with the appropriate feature enabled) to attempt this and fail on startup. Resolved with separate builds for each of the boards but, certainly seems there are some further logic errors there.

@persan666
Copy link

persan666 commented Jan 28, 2021

Will this patch 4b44431 also be merge into 4.2? @Spritetm

@SinglWolf
Copy link

Will this patch 4b44431 also be merge into 4.2? @Spritetm

I support the question. It is not correct when in one project, because of this misunderstanding, you need to compile two different firmware.

@karawin
Copy link

karawin commented Jan 12, 2022

Same for esp-idf 3.3.5
Patched in esp-idf\components\esp32\spiram.c
-esp_spiram_size_t esp_spiram_get_chip_size()
{
if (!spiram_inited) {
#if CONFIG_SPIRAM_IGNORE_NOTFOUND
ESP_EARLY_LOGE(TAG, "SPI RAM not initialized");
return ESP_SPIRAM_SIZE_INVALID;
#endif
abort();
}_

Seems to be introduced by
Commits on Jun 25, 2021 psram: support for esp32-pico-v3-02
bc60eb6#diff-32ff69eb3440f75e0152e521896352744eed5efcfaed61b8798b8039c9f77c5d

See also
https://issueexplorer.com/issue/espressif/esp-idf/7988

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants