Filesystem images created with the default_16MB.csv or large_spiffs_16MB.csv partition tables fail to upload
#7467
Assignees
Labels
Chip: ESP32-S3
Issue is related to support of ESP32-S3 Chip
IDE: PlaformIO
Issue relates to PlatformIO IDE
Comments
CoryCharlton
changed the title
|
Confirm same "Invalid head of packet (0x6F)" error with Unexpected Maker ProS3 16M board [ESP32-S3 (revision v0.1)] and that the custom partition table here fixes the issue in PlatformIO 2.5.5. |
|
@CoryCharlton - Could you please test the UM FeatherS3 with the Arduino IDE? Configuration for the UM board in the IDE would be:
|
|
The Arduino Core 2.0.5 uses esptool.py v4.2.1 |
SuGlider
added
Chip: ESP32-S3
CoryCharlton
changed the title
default_16MB.csv or large_spiffs_16MB.csv partition tables fail to upload
|
@SuGlider I updated the title to be more clear that the issue appears to be related specifically to I browse the repo but I'm not clear which partition table csv at https://github.com/espressif/arduino-esp32/tree/master/tools/partitions corresponds to the "16M Flash (2MB APP/12.5MB FATFS)" you show in the UI. If you can point me to the correct file I can compare the differences to see if anything jumps out. That being said I'm not in a position right now to test the Arduino IDE but I did test earlier using default and large spiffs and encountered the same error I did when using PlatformIO or the esptool directly. |
|
Then again the data in the I modified So this may not be an issue with the partition tables or Arduino Core and the esptool issue might still be the cause (I'll let you two teams fight it out :P) Whatever side the issue is on this could be worked around in this repo by reducing the size of the spiffs partitions in |
1 task
|
Have you consider that it could be a Flash chip failure? Or even a bad cable issue... |
|
Another possible test, would be to lower the flashing baud rate to, maybe, 115200... |
|
|
Here's what I have tried, all of which resulted in the same error. All of these combinations work fine if the filesystem image is slightly smaller than the entire 16MB.
I do not have any ESP32-S3 dev boards with 16MB flash that are not UM FeatherS3 but @pauldupuyjr is using an UM ProS3. As far as flash chip failure goes it seems highly unlikely that multiple chips would all fail within the last 1KB of address space but work fine otherwise. Flashing application code or smaller filesystems works fine.
I agree which is why I started on the esptool side. The fact that I could modify the partition and get it to work then made me assume that it was related to the partition definition and open this issue. Admittedly I never actually validated the definitions but now that I have they appear correct; which is pointing me back at the esptool side (or even the ESP32-S3 🤷). @SuGlider two and a half questions:
|
|
Does it work if you make your spiffs bin 1K smaller? I don't know exactly how spiffs handles it, but ffat has 1K overhead that it is using for wear levelling, so the fs image must be 1k smaller than the partition size. Perhaps the platform.io tool is not calculating correctly? |
Yes, I just tried the ESP32-S3-Box board. https://www.espressif.com/en/news/ESP32-S3-BOX_video I used Arduino IDE 1.8.15 and Arduino Core 2.0.5 to flash a simple Blink.ino, using as selected board the It worked fine with https://github.com/espressif/arduino-esp32/blob/master/tools/partitions/default_16MB.csv It also worked fine with https://github.com/espressif/arduino-esp32/blob/master/tools/partitions/large_spiffs_16MB.csv The 32MB board I referenced earlier is an ESP32-S3-DevKitC-1 v1.1 board with an ESP32-S3-WROOM-2-N32R8V module with Octal SPI.
This is the file: ========= |
|
In case it can help, the complete successful building/flashing log, using Win11, is: |
|
I noticed a few differences with your upload log....
|
|
Arduino IDE --> When I use Arduino IDE I get: |
|
I think that there is something wrong with PIO building/flashing process. |
|
@SuGlider They are uploading filesystem images, not firmware. The equivalent in Arduino IDE would be using https://github.com/me-no-dev/arduino-esp32fs-plugin. |
|
Mhh, it does work in general since i have a S3 16MB flash /8MB PSRAM (qio/opi) which does work when flashed from PIO. |
Ok. I see now. Anyway, this issue isn't related to Arduino. |
|
I still think this is a PIO issue instead... |
|
You guys are missing the issue. Application code uploads fine. This is an issue related to uploading a filesystem that contains files (a simple TXT for is enough to reproduce). Uploading a filesystem is not directly possible in Arduino IDE and you would need a plugin: https://randomnerdtutorials.com/install-esp32-filesystem-uploader-arduino-ide/#installing-windows (this may have changed in Arduino IDE 2+). All PIO or the Arduino IDE plug-in does is call out to the esptools. So long story short this is likely an issue with the esptool at this point but I'm still waiting on them to confirm/reproduce. |
|
OK, thanks. Indeed, I didn't get that at first glance. It doesn't work for the S3... it only works for the ESP32. @me-no-dev FYI |
|
@SuGlider I think that might be the different as well. You're writing a single file while I'm flashing the entire partition. Can you try uploading this spiffs.zip [had to zip the .bin so it would upload]. PIO used the following command to build the image from the I suppose |
|
@CoryCharlton That command shows the data size as 0x6F0000 (7274496), whereas your partition is 0x680000. Can you try manually running mkspiffs with the correct size, and see if that uploads properly? This certainly looks like a problem with the pio script that is building the image file. |
|
@lbernstone that is using the My custom partition shown above works fine and decreased the spiffs partition by 1KB to 0x680000 |
|
Just to be sure... Verified that the created spiffs partition with files is correctly generated? |
|
@Jason2866 I'm not sure how I would verify the bin file with the partition but I attached it in this comment. That being said the fact I can generate and upload a working partition bin by only decreasing the size slightly I'm going to assume it is a valid file (assumptions always bite me but I still like making them :P) This comment from @dobairoland also points out that the error is being triggered by the ESP32-S3 throwing an exception. Unfortunately I do not have a non-S3 board with a 16MB flash to test with but I'm willing to make the following assumptions (see, I love them) at this point:
Based on those assumptions I'm going to submit a PR shortly that will decrease the 16MB spiffs partitions by 1KB. To me this seems like an acceptable work around based on the fact that it's only 1KB being wasted. The other alternative would be to create S3 specific partition tables but this seems like more work (ie: the boards list would need to be changed for them all). That also assumes the error is specific to the S3 which is 100% assumption on my part at this time. |
|
@CoryCharlton - I tested writing the Flash up to the last byte at 0xFFFFFF (16MB) I used esptool v4.2.1 and the ESP32-S3-Box that has QIO Flash of 16MB After flashing the firmware, I uploaded the spiffs.zip file to it as shown below.Then I reset the board and the sketch was working fine as well. So, again, I can't reproduce this issue.I set the board in |
|
Another detail... I think that esptool.py doesn't care about the partition scheme. It seems to be an issue with PlatformIO scripts, instead. |
CoryCharlton
added a commit
to CoryCharlton/arduino-esp32
that referenced
this issue
Nov 14, 2022
|
@SuGlider correct, However the IDE (in this case PIO) is aware of the partition scheme and uses those values to determine what parameters to pass. PIO is just passing the values as they are specified in the partition CSV file and everything indicates it's doing the correct thing at this point (based on the data in the CSV). Here's an example: default_16MB.csv - From this repo SPIFFS offset: 0xc90000 - 13172736 Make filesystem: Command: "mkspiffs_espressif32_arduino" -c data -s 3604480 -p 256 -b 4096 .pio\build\debug\spiffs.bin Upload filesystem: Command: "C:\Users\ccharlton.platformio\penv\Scripts\python.exe" "C:\Users\ccharlton.platformio\packages\tool-esptoolpy\esptool.py" --chip esp32s3 --port "COM14" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 16MB 13172736 .pio\build\debug\spiffs.bin default_16MB.csv - From my fork SPIFFS offset: 0xc90000 - 13172736 Make filesystem: Command: **"mkspiffs_espressif32_arduino" -c data -s 3603456 -p 256 -b 4096 .pio\build\debug\spiffs.bin Upload filesystem: Command: "C:\Users\ccharlton.platformio\penv\Scripts\python.exe" "C:\Users\ccharlton.platformio\packages\tool-esptoolpy\esptool.py" --chip esp32s3 --port "COM14" --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 16MB 13172736 .pio\build\debug\spiffs.bin So you can see PIO is just passing the values supplied from the partition CSV but without the 1KB buffer the partition fails to upload (due to a bug in the ESP32-S3?). Long story short my PR will workaround this issue and allow users to create and upload a SPIFFS filesystem to an ESP32-S3 with 16MB flash. As it stands now users are unable to do this due to a mysterious/vague error message that may not be able to get any clearer. |
|
@CoryCharlton The PR is not needed. This partitions.csv works for me. Platformio uploads to a S3 without any issue. No need to reduce size. |
|
@CoryCharlton - I just tested your case. 4 bytes is just a word (32 bits) in difference of the size that works or not.... working case with 3.604.476 bytes:NOT working case with 3.604.480 bytes: |
|
Files Used in this testing (remove, if necessary, the .txt extension of the file): |
|
@SuGlider Are you generating the files with mkspiffs? Are they created the correct size? It ought to be aligned to the page/block dimensions. I'm sure when the tool was built it was never tested against an image larger than 1MB... |
|
No, I just created it manually... but it allowed me to verify the issue anyway, |
|
@CoryCharlton - Please test it with these suggested spiffs partition sizes: tools/partitions/default_16MB.csv tools/partitions/large_spiffs_16MB.csv |
CoryCharlton
added a commit
to CoryCharlton/arduino-esp32
that referenced
this issue
Nov 14, 2022
|
@SuGlider confirmed that decreasing by 4 bytes resolves the issue. PR updated. |
|
The PR is against the docs https://docs.espressif.com/projects/esp-idf/en/release-v4.4/esp32/api-guides/partition-tables.html |
|
So it seems the mkspiffs tool or whatever creates the partition content is buggy. |
|
If you make a file of 16 kB of all zeros and try to flash it to the end of the flash region ( |
|
@dobairoland that test failed with the same error. Here's the commands I used: And the data: And the output: trace_empty_16kb_compress.txt |
|
@CoryCharlton Thank you for testing and providing reports. @SuGlider I see you've got an ESP32-S3-Box. Could you please try #7467 (comment). I was not able to reproduce with 8MB flash (espressif/esptool#795 (comment)) and don't have access to a board with 16MB flash. |
|
@dobairoland I tried that one, but was not able to compile my esptool locally |
|
For sharing, I got the same error "Invalid head of packet (0x6F): Possible serial noise or corruption" with Arduino IDE when uploading data files on 16Mb Flash (using arduino-esp32fs-plugin)
As highlighted by @SuGlider, decreasing flash size by 4 bytes corrects the error, see KO partition table and OK partition table below. Thanks to @SuGlider for this workaround 😄 👍 This error does not occur when flash size is set to 8Mb or less. KO OK |
|
This has been resolved in the esptool. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Board
Unexpected Maker FeatherS3 (but likely any board with a 16MB flash)
Device Description
Unexpected Maker FeatherS3
Hardware Configuration
Nothing connected
Version
v2.0.5
IDE Name
PlatformIO
Operating System
Windows 10
Flash frequency
80Mhz
PSRAM enabled
yes
Upload speed
460800
Description
If you create a filesystem image using
default_16MB.csvorlarge_spiffs_16MB.csvit fails to upload (see: espressif/esptool#795).A custom partition table such as the following works fine:
Sketch
Debug Message
Other Steps to Reproduce
No response
I have checked existing issues, online documentation and the Troubleshooting Guide
The text was updated successfully, but these errors were encountered: