setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913) (OCD-670) #264

zfields · 2022-12-08T12:48:56Z

Answers checklist.

I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.0

Operating System used.

macOS

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32-S3-WROOM-1 (N8R2)

Power Supply used.

USB

What is the expected behavior?

I expect to be able to allocate PSRAM in chunks bigger than 24540 bytes.

What is the actual behavior?

It starts to overwrite memory when 24541 and larger bytes are allocated.

Steps to reproduce.

main.c

#include <stdio.h>
#include <string.h>

void app_main(void)
{
    // CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL == 16384, so any block larger than this                                      
    // will be forced into PSRAM.  The following shows an overwrite.                                                    
    // 24540 works; 24541 begins to show overwrite; 1000000 is massive overwrite                                        
    uint8_t *obj = calloc(24541,sizeof(uint8_t));
    (void)obj;

    // Test breakpoints, to be set when you are stopped at app_main.                                                    
    // - if you only set "b 15" then "c", when it breaks do an "x/128xb obj" you'll see 0's as you should               
    // - but if you set "b 15" AND "b 16" AND "b 17" then "c", when it breaks you'll see bad values                     
    printf("breakpoint 1\n");
    printf("breakpoint 2\n");
    printf("breakpoint 3\n");
}

Use an ESP32-S3 (N8R2) and plug both USB's into your PC
In esp-idf, make sure you "./install.sh esp32s3" to install tooling for the S3
In this project do "idf.py set-target esp32s3"
in "idf.py menuconfig"
1. Compiler Options / Optimization Level / Debug without optimization (-O0)
2. Serial flasher config -> Flash size -> 8 MB
3. Component Config -> ESP PSRAM -> ENABLE Support for external, SPI-connected RAM
idf.py build & idf.py flash to flash...
Start two terminal sessions, and in both make sure you do ". ./export.sh" to set env vars
In terminal 1, run:
openocd -f board/esp32s3-builtin.cfg -c "esp32s3.cpu1 configure -rtos FreeRTOS"
In terminal 2, run:
xtensa-esp32s3-elf-gdb -tui -ex 'set pagination off' -ex 'target extended-remote :3333' -ex 'mon reset halt' -ex 'maintenance flush register-cache' -ex 'thb app_main' -ex 'monitor [target current] configure -event gdb-detach { shutdown }' -ex 'c' ./build/main.elf

Debug Logs.

No response

More Information.

Allocating 24540 works; 24541 begins to show overwrite
if you only set "b 15" then "c", when it breaks do an "x/128xb obj" you'll see 0's as you should
if you set "b 15" AND "b 16" AND "b 17" then "c", when it breaks you'll see bad values

The text was updated successfully, but these errors were encountered:

igrr · 2022-12-08T13:29:23Z

@zfields Do you observe this issue when you aren't connecting OpenOCD? I think OpenOCD on ESP32-S3 doesn't support PSRAM yet, so PSRAM data may be corrupted when OpenOCD is setting/removing a breakpoint.

zfields · 2022-12-08T14:23:25Z

If that's true, that would certainly explain it. How would I connect without OpenOCD to check?

igrr · 2022-12-08T14:33:18Z

I think you can try to still run OpenOCD but disable the software breakpoints in flash, then force GDB to use only hardware breakpoints. As a result, OpenOCD will not write to flash and this should prevent PSRAM corruption.

In step 7:

openocd -c 'set ESP_FLASH_SIZE 0' -f board/esp32s3-builtin.cfg

When running GDB, add

-ex 'set remote hardware-breakpoint-limit 2'

and use thb (temporary hardware breakpoint) instead of b to use only hardware breakpoints and keep the number of breakpoints <=2.

zfields · 2022-12-08T19:24:20Z

Thanks @igrr for following up with me so quickly!

When dumping memory even with these fixes and even with 1 breakpoint, the first byte looks bad when you first dump it. However, it seems to "clear up" if you dump it twice. Any ideas?

Also, is there any time frame for when using OpenOCD on the ESP32-S3 will support PSRAM?

igrr · 2022-12-13T12:46:49Z

@zfields Have transferred this issue to OpenOCD project so that it is tracked properly.

Regarding the successful read on 2nd attempt, there has been a similar issue we had fixed recently; however it looks like you are already using the latest release which includes the fix. Could you please confirm the version of OpenOCD you are running?

Regarding PSRAM support for ESP32-S3, it will likely be done in Q1 2023.

zfields · 2022-12-13T16:00:51Z

Thanks @igrr!

Could you please confirm the version of OpenOCD you are running?

I'm using OpenOCD v0.11.0-esp32-20221026.

Regarding PSRAM support for ESP32-S3, it will likely be done in Q1 2023.

Thanks, that's great news!

Have transferred this issue to OpenOCD project so that it is tracked properly.

No, I have not. Is there a way to "transfer" it using the GitHub UI, or do you mean reopen it there?

igrr · 2022-12-13T16:09:20Z

Sorry for not phrasing this clearly. I meant that I have already transferred this issue to the OpenOCD project. You don't need to do anything.

erhankur · 2023-01-04T17:27:12Z

Hi @zfields Can I ask you to test with the attached version?

openocd-esp32-macos-0.11.0-esp32-20221026-86-g283b87d64.tar.gz

zfields · 2023-01-05T00:57:37Z

I'm working on something at this very moment, but I will test it ASAP and get back to you.

Thanks!

zfields · 2023-01-05T02:40:59Z

I tried it with the new openocd.

As you can see, the three breakpoints say that they were set just fine. However, when I pressed "c" to continue, then it spit out the error messages shown the terminal.

erhankur · 2023-03-02T11:30:16Z

@zfields We provided the solution in the latest release. Can you please confirm your issue has been fixed?

zfields · 2023-03-02T15:18:01Z

I'm sorry, but it does NOT appear to be fixed.

My Test

rm -rf of both ~/.espressif and ~/esp/esp-idf

reinstalled the 5.0.1-tagged release

double-checking the versions:

$ idf.py --version
ESP-IDF v5.0.1
$ which openocd
/Users/rozzie/.espressif/tools/openocd-esp32/v0.11.0-esp32-20221026/openocd-esp32/bin/openocd
$ openocd --version
Open On-Chip Debugger v0.11.0-esp32-20221026 (2022-10-26-14:48)
$ which xtensa-esp32-elf-gdb
/Users/rozzie/.espressif/tools/xtensa-esp-elf-gdb/11.2_20220823/xtensa-esp-elf-gdb/bin/xtensa-esp32-elf-gdb
$ xtensa-esp32-elf-gdb --version
GNU gdb (esp-gdb) 11.2_20220823

rm -rf of the build folder as a brute force idf.py clean
rebuilt and re-flashed the app
set the breakpoints
exactly same problem happens

Repro Instructions

sample_project.zip

unzip this sample_project
open two terminal windows
do . ./export.sh in each
in each terminal, cd to the new sample_project directory
in one terminal, do an idf.py build and idf.py flash
in the same terminal, type ./serve.sh
in the other terminal, type ./debug.sh

when it hits the first breakpoint in gdb, look at the source code and follow the instructions

gerekon · 2023-03-02T15:24:57Z

@zfields IDF is not updated to use the latest release. In your example you still use v0.11.0-esp32-20221026.
To test it you need to download and run it manually.

zfields · 2023-03-02T15:29:53Z

I'm happy to test, but this is the message I responded to:

@zfields We provided the solution in the latest release. Can you please confirm your issue has been fixed?

Can you please provide detailed instructions on what exactly you would like for me to test?

gerekon · 2023-03-02T15:39:01Z

I'm happy to test, but this is the message I responded to:

I suppose that was about the latest OpenOCD release.

Can you please provide detailed instructions on what exactly you would like for me to test?

It would be great of you checked your scenario described in the first message of this topic. But with the latest OpenOCD release. So your modified command should look like

$PATH_TO_LATEST_RELEASE/bin/openocd -s $PATH_TO_LATEST_RELEASE/share/openocd/scripts -f board/esp32s3-builtin.cfg -c "esp32s3.cpu1 configure -rtos FreeRTOS"

zfields · 2023-03-02T15:46:11Z

Okay, so everything else will be the same, and you want me to additionally update/replace openocd-esp32 with the latest release.

I'll will test this 👆, and get back to you.

zfields · 2023-03-02T21:37:17Z

I downloaded the latest openocd-esp32, but I'm still having the issue.

$ which openocd-esp32/bin/openocd
openocd-esp32/bin/openocd
$ openocd-esp32/bin/openocd --version
Open On-Chip Debugger v0.12.0-esp32-20230221 (2023-02-21-11:31)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html

downloaded from: https://github.com/espressif/openocd-esp32/releases/tag/v0.12.0-esp32-20230221

The problem happens when I say "c" to continue after setting the breakpoint. On the gdb side it gives the warning that I can't insert the breakpoint, and on the openocd side it says "Warn" "address not writable".

GDB Output:

OpenOCD Output:
openocd-esp32/bin/openocd -f board/esp32s3-builtin.cfg

sample_project $ serve.sh
++ openocd-esp32/bin/openocd -f board/esp32s3-builtin.cfg
Open On-Chip Debugger v0.12.0-esp32-20230221 (2023-02-21-11:31)
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
Info : only one transport option; autoselecting 'jtag'
Info : esp_usb_jtag: VID set to 0x303a and PID to 0x1001
Info : esp_usb_jtag: capabilities descriptor set to 0x2000
Warn : Transport "jtag" was already selected
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections
Info : esp_usb_jtag: serial (68:B6:B3:2C:D6:10)
Info : esp_usb_jtag: Device found. Base speed 40000KHz, div range 1 to 255
Info : clock speed 40000 kHz
Info : JTAG tap: esp32s3.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32s3.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : starting gdb server for esp32s3.cpu0 on 3333
Info : Listening on port 3333 for gdb connections
Info : [esp32s3.cpu0] Target halted, PC=0x4200CA15, debug_reason=00000001
Info : [esp32s3.cpu0] Reset cause (3) - (Software core reset)
Info : Set GDB target to 'esp32s3.cpu0'
Info : [esp32s3.cpu1] Target halted, PC=0x4037A900, debug_reason=00000000
Info : [esp32s3.cpu1] Reset cause (3) - (Software core reset)
Info : accepting 'gdb' connection on tcp/3333
Warn : No symbols for FreeRTOS!
Info : [esp32s3.cpu0] Target halted, PC=0x40374400, debug_reason=00000001
Error: Failed to get flash maps (151587081)!
Warn : Failed to get flash mappings (-4)!
Info : [esp32s3.cpu0] Target halted, PC=0x403B244A, debug_reason=00000001
Info : [esp32s3.cpu0] Target halted, PC=0x403B244A, debug_reason=00000001
Info : Auto-detected flash bank 'esp32s3.cpu0.flash' size 148034 KB
Info : Using flash bank 'esp32s3.cpu0.flash' size 148034 KB
Info : [esp32s3.cpu0] Target halted, PC=0x40374400, debug_reason=00000001
Error: Failed to get flash maps (151587081)!
Warn : Failed to get flash mappings (-4)!
Info : [esp32s3.cpu0] Target halted, PC=0x403B244A, debug_reason=00000001
Info : Using flash bank 'esp32s3.cpu0.irom' size 0 KB
Info : [esp32s3.cpu0] Target halted, PC=0x40374400, debug_reason=00000001
Error: Failed to get flash maps (151587081)!
Warn : Failed to get flash mappings (-4)!
Info : [esp32s3.cpu0] Target halted, PC=0x403B244A, debug_reason=00000001
Info : Using flash bank 'esp32s3.cpu0.drom' size 0 KB
Info : New GDB Connection: 1, Target esp32s3.cpu0, state: halted
Warn : negative reply, retrying
Info : Detected FreeRTOS version: (10.4.3)
Info : JTAG tap: esp32s3.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32s3.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : [esp32s3.cpu0] requesting target halt and executing a soft reset
Info : [esp32s3.cpu0] Debug controller was reset.
Info : [esp32s3.cpu0] Core was reset.
Info : [esp32s3.cpu0] Target halted, PC=0x500000EF, debug_reason=00000000
Info : [esp32s3.cpu0] Reset cause (3) - (Software core reset)
Info : [esp32s3.cpu1] requesting target halt and executing a soft reset
Info : [esp32s3.cpu0] Core was reset.
Info : [esp32s3.cpu0] Target halted, PC=0x40000400, debug_reason=00000000
Info : [esp32s3.cpu1] Debug controller was reset.
Info : [esp32s3.cpu1] Core was reset.
Info : [esp32s3.cpu1] Target halted, PC=0x40000400, debug_reason=00000000
Info : [esp32s3.cpu1] Reset cause (3) - (Software core reset)
Info : [esp32s3.cpu0] Reset cause (3) - (Software core reset)
Info : [esp32s3.cpu1] Debug controller was reset.
Info : [esp32s3.cpu1] Core was reset.
Info : [esp32s3.cpu1] Target halted, PC=0x40041A79, debug_reason=00000000
Info : [esp32s3.cpu1] Reset cause (3) - (Software core reset)
Info : [esp32s3.cpu0] Target halted, PC=0x4200CA15, debug_reason=00000001
Info : Set GDB target to 'esp32s3.cpu0'
Info : [esp32s3.cpu1] Target halted, PC=0x4037A900, debug_reason=00000000
Info : Detected FreeRTOS version: (10.4.3)
Warn : address 0x4200ca36 not writable
Error: [esp32s3.cpu0] Error writing instruction to memory: -4
Error: [esp32s3.cpu0] Failed to write breakpoint instruction (-4)!
Error: [esp32s3.cpu0] Failed to add SW breakpoint!
Error: can't add breakpoint: unknown reason

erhankur · 2023-03-02T23:06:44Z

@zfields please make sure the OPENOCD_SCRIPTS env variable points to the new location. (path to the v0.12.0-esp32-20230221/openocd-esp32/share/openocd/scripts)

zfields · 2023-03-03T02:51:53Z

Eureka! That worked! 👍

Do you know which official IDF release is planned to contain the latest release of openocd-esp32, and approximately when that release is scheduled to occur?

Thanks,
Zak

erhankur · 2023-03-03T03:07:00Z

Afaik, it will be in the IDF v5.1 release around June. Thanks for providing reproducible code and support to get this done.

espressif-bot added the Status: Opened label Dec 8, 2022

github-actions bot changed the title ~~Bad allocations to PSRAM when size exceeds 24540~~ Bad allocations to PSRAM when size exceeds 24540 (IDFGH-8913) Dec 8, 2022

zfields changed the title ~~Bad allocations to PSRAM when size exceeds 24540 (IDFGH-8913)~~ Bad allocations to PSRAM during initialization. Setting gdb software or hardware breakpoints causes memory overwrite (IDFGH-8913) Dec 8, 2022

zfields changed the title ~~Bad allocations to PSRAM during initialization. Setting gdb software or hardware breakpoints causes memory overwrite (IDFGH-8913)~~ setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913) Dec 8, 2022

igrr transferred this issue from espressif/esp-idf Dec 13, 2022

github-actions bot changed the title ~~setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913)~~ setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913) (OCD-670) Dec 13, 2022

espressif-bot assigned erhankur Jan 4, 2023

espressif-bot assigned gerekon and unassigned erhankur Jan 25, 2023

espressif-bot added Status: In Review and removed Status: Opened labels Feb 6, 2023

espressif-bot assigned gerekon and unassigned gerekon Feb 6, 2023

espressif-bot added Resolution: Done Status: Done and removed Status: In Review labels Feb 20, 2023

erhankur closed this as completed Mar 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913) (OCD-670) #264

setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913) (OCD-670) #264

zfields commented Dec 8, 2022

igrr commented Dec 8, 2022

zfields commented Dec 8, 2022

igrr commented Dec 8, 2022 •

edited

Loading

zfields commented Dec 8, 2022

igrr commented Dec 13, 2022

zfields commented Dec 13, 2022 •

edited

Loading

igrr commented Dec 13, 2022

erhankur commented Jan 4, 2023

zfields commented Jan 5, 2023

zfields commented Jan 5, 2023 •

edited

Loading

erhankur commented Mar 2, 2023

zfields commented Mar 2, 2023

gerekon commented Mar 2, 2023

zfields commented Mar 2, 2023 •

edited

Loading

gerekon commented Mar 2, 2023

zfields commented Mar 2, 2023

zfields commented Mar 2, 2023

erhankur commented Mar 2, 2023 •

edited

Loading

zfields commented Mar 3, 2023 •

edited

Loading

erhankur commented Mar 3, 2023

setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913) (OCD-670) #264

setting gdb breakpoints at init causes PSRAM memory overwrite (IDFGH-8913) (OCD-670) #264

Comments

zfields commented Dec 8, 2022

Answers checklist.

IDF version.

Operating System used.

How did you build your project?

If you are using Windows, please specify command line type.

Development Kit.

Power Supply used.

What is the expected behavior?

What is the actual behavior?

Steps to reproduce.

Debug Logs.

More Information.

igrr commented Dec 8, 2022

zfields commented Dec 8, 2022

igrr commented Dec 8, 2022 • edited Loading

zfields commented Dec 8, 2022

igrr commented Dec 13, 2022

zfields commented Dec 13, 2022 • edited Loading

igrr commented Dec 13, 2022

erhankur commented Jan 4, 2023

zfields commented Jan 5, 2023

zfields commented Jan 5, 2023 • edited Loading

erhankur commented Mar 2, 2023

zfields commented Mar 2, 2023

My Test

Repro Instructions

gerekon commented Mar 2, 2023

zfields commented Mar 2, 2023 • edited Loading

gerekon commented Mar 2, 2023

zfields commented Mar 2, 2023

zfields commented Mar 2, 2023

erhankur commented Mar 2, 2023 • edited Loading

zfields commented Mar 3, 2023 • edited Loading

erhankur commented Mar 3, 2023

igrr commented Dec 8, 2022 •

edited

Loading

zfields commented Dec 13, 2022 •

edited

Loading

zfields commented Jan 5, 2023 •

edited

Loading

zfields commented Mar 2, 2023 •

edited

Loading

erhankur commented Mar 2, 2023 •

edited

Loading

zfields commented Mar 3, 2023 •

edited

Loading