Serial flashing big spiffs fails (IDFGH-8905)

[higaski]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.0

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

Writing big spiffs images (>16MB?) using idf.py flash doesn't seem to work yet.

Here's the partition table I'm currently using

# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  24K,
phy_init, data, phy,     0xf000,  4K,
factory,  app,  factory, 0x10000, 2M,
storage,  data, spiffs,  ,        20M,

I've attached a log of some attempted writes below. The percent-mark it keeps failing on seems to depend on the amount of actual data inside the image. Having a 20MB spiffs image with only 2.9MB of actual data fails at 100% whereas an image with the same size but 13MB of data in it fails way earlier...?

What is the actual behavior?

20MB spiffs image, 2.9MB of actual data in it

Writing at 0x00529149... (97 %)
Writing at 0x0060f607... (98 %)
Writing at 0x00eb101b... (100 %)
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/__main__.py", line 11, in <module>
    esptool._main()
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/__init__.py", line 1026, in _main
    main()
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/__init__.py", line 826, in main
    operation_func(esp, args)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/cmds.py", line 556, in write_flash
    esp.flash_defl_block(block, seq, timeout=timeout)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 109, in inner
    return func(*args, **kwargs)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 991, in flash_defl_block
    self.check_command(
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 407, in check_command
    val, data = self.command(op, data, chk, timeout=timeout)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 376, in command
    p = self.read()
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 308, in read
    return next(self._slip_reader)
StopIteration
CMake Error at run_serial_tool.cmake:55 (message):
  
  /home/vinci/.espressif/python_env/idf5.0_py3.10_env/bin/python;;/home/vinci/esp/esp-idf/components/esptool_py/esptool/esptool.py;--chip;esp32s3
  failed

20MB spiffs image, 13MB of data in it

Writing at 0x006ae251... (28 %)
Writing at 0x006b264e... (28 %)
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/__main__.py", line 11, in <module>
    esptool._main()
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/__init__.py", line 1026, in _main
    main()
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/__init__.py", line 826, in main
    operation_func(esp, args)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/cmds.py", line 556, in write_flash
    esp.flash_defl_block(block, seq, timeout=timeout)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 109, in inner
    return func(*args, **kwargs)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 991, in flash_defl_block
    self.check_command(
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 407, in check_command
    val, data = self.command(op, data, chk, timeout=timeout)
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 376, in command
    p = self.read()
  File "/home/vinci/.espressif/python_env/idf5.0_py3.10_env/lib/python3.10/site-packages/esptool/loader.py", line 308, in read
    return next(self._slip_reader)
StopIteration
CMake Error at run_serial_tool.cmake:55 (message):
  
  /home/vinci/.espressif/python_env/idf5.0_py3.10_env/bin/python;;/home/vinci/esp/esp-idf/components/esptool_py/esptool/esptool.py;--chip;esp32s3
  failed

Steps to reproduce.

1. Create a project with a spiffs image size of >= 20MB
2. Run idf.py flash

Build or installation Logs.

No response

More Information.

No response

[dobairoland]

Hi @higaski. We are sorry for this issue. I think this is the same as espressif/esptool#795 and it is not related just to SPIFFS. All flashing accessing >= 16MB area are affected.

[higaski]

Hi @higaski. We are sorry for this issue. I think this is the same as espressif/esptool#795 and it is not related just to SPIFFS. All flashing accessing >= 16MB area are affected.

Yeah I figured that the python script doing the writing doesn't really care if it's code, data or 🍌 s. 😄

You're right, looks like a duplicate. And I even managed to open the issue in the wrong repo... sry for that.

[dobairoland]

This is just a note that the next esptool release (v4.5) should address this. And it also improves the error message in case of StopIteration.

[robbietree8]

@dobairoland
hi, what about quad flash? will the limit of 16MB be addressed in near future?

[dobairoland]

@robbietree8 It depends on the HW. I don't have anything like that right now. If I'm not mistaken, Espressif sells ESP32-S3s only with Quad flash and 32 MB Octal flash right now.

[robbietree8]

@dobairoland we have custom board made with 32MB quad flash.

[dobairoland]

esptool v4.5 has been released and it is included with ESP-IDF v5.0+ after you re-run the install script.