V3.2: Python struct.error when doing chip_id (ESPTOOL-338)

[dhalbert]

Ubuntu 20.04 (also seen on Windows 10)
Python 3.8.10 (Windows 10 was using 3.9.x)

ESP32-S2
Adafruit Metro ESP32-S2 with ESP32-S2-WROVER module

$ esptool.py --port /dev/ttyACM0 chip_id
esptool.py v3.2
Serial port /dev/ttyACM0
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting...
Traceback (most recent call last):
  File "/home/halbert/.local/bin/esptool.py", line 377, in detect_chip
    res = struct.unpack("<IBBBBBBBBI", res[:16])  # 4b flags, 1b flash_crypt_cnt, 7*1b key_purposes, 4b chip_id
struct.error: unpack requires a buffer of 16 bytes
[... further cascaded Python errors ...]

Does not fail immediately after a flash_erase, but after loading a firmware .bin , fails again.
Reverting to esptool.py V3.1 fixes the problem.

First reported here: https://forums.adafruit.com/viewtopic.php?f=60&t=184596

[radimkarnis]

Hello @dhalbert,
thank you for reporting this!

I will try to reproduce the issue and investigate. In the meantime, it is possible to skip chip autodetection by specifying the --chip option. This should be a quick workaround for the issue:

esptool.py --chip esp32s2 --port /dev/ttyACM0 chip_id

[radimkarnis]

Would you mind sharing the full log and steps to reproduce the issue? I was not able to do so, tried both Python 3.9 and 3.10 (esptool does not officially support Python 3.10 yet), worked well with ESP32S2.

This shouldn't happen, the struct.error is expected with ESP32S2 and should be caught. Looking at the log, it looks like it is caught and another exception happens (probably mentioned in the [... further cascaded Python errors ...] part of the log).

[dhalbert]

Sure. Here's another way to reproduce, using erase_flash:

Ubuntu 20.04
Python 3.8.10
esptool v3.2
pyserial 3.5
Adafruit Metro ESP32-S2 with ESP32-S2-WROVER module

Get into boot mode (reset with boot button held down).

$ esptool.py --chip esp32s2 -p /dev/ttyACM0 erase_flash  # WITH --chip esp32s2
esptool.py v3.2
Serial port /dev/ttyACM0
Connecting...
Chip is ESP32-S2
Features: WiFi, No Embedded Flash, No Embedded PSRAM, ADC and temperature sensor calibration in BLK2 of efuse V1
Crystal is 40MHz
MAC: 7c:df:a1:09:ae:de
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 16.5s
WARNING: ESP32-S2 chip was placed into download mode using GPIO0.
esptool.py can not exit the download mode over USB. To run the app, reset the chip manually.
To suppress this note, set --after option to 'no_reset'.

Then reset into boot mode again, which is necessary to reproduce; It does not fail without the hard reset.

$ esptool.py -p /dev/ttyACM0 erase_flash     # WITHOUT --chip esp32s2
esptool.py v3.2
Serial port /dev/ttyACM0
Connecting...
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting...
Traceback (most recent call last):
  File "/home/halbert/.local/bin/esptool.py", line 377, in detect_chip
    res = struct.unpack("<IBBBBBBBBI", res[:16])  # 4b flags, 1b flash_crypt_cnt, 7*1b key_purposes, 4b chip_id
struct.error: unpack requires a buffer of 16 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/halbert/.local/bin/esptool.py", line 5143, in <module>
    _main()
  File "/home/halbert/.local/bin/esptool.py", line 5136, in _main
    main()
  File "/home/halbert/.local/bin/esptool.py", line 4541, in main
    esp = esp or get_default_connected_device(ser_list, port=args.port, connect_attempts=args.connect_attempts,
  File "/home/halbert/.local/bin/esptool.py", line 121, in get_default_connected_device
    _esp = ESPLoader.detect_chip(each_port, initial_baud, before, trace,
  File "/home/halbert/.local/bin/esptool.py", line 388, in detect_chip
    detect_port.connect(connect_mode, connect_attempts, detecting=True, warnings=False)  # Need to connect again
  File "/home/halbert/.local/bin/esptool.py", line 663, in connect
    last_error = self._connect_attempt(mode=mode, usb_jtag_serial=usb_jtag_serial, extra_delay=extra_delay)
  File "/home/halbert/.local/bin/esptool.py", line 613, in _connect_attempt
    self.bootloader_reset(usb_jtag_serial, extra_delay)
  File "/home/halbert/.local/bin/esptool.py", line 598, in bootloader_reset
    self._setDTR(False)  # IO0=HIGH, done
  File "/home/halbert/.local/bin/esptool.py", line 532, in _setDTR
    self._port.setDTR(state)
  File "/home/halbert/.local/lib/python3.8/site-packages/serial/serialutil.py", line 603, in setDTR
    self.dtr = value
  File "/home/halbert/.local/lib/python3.8/site-packages/serial/serialutil.py", line 473, in dtr
    self._update_dtr_state()
  File "/home/halbert/.local/lib/python3.8/site-packages/serial/serialposix.py", line 715, in _update_dtr_state
    fcntl.ioctl(self.fd, TIOCMBIC, TIOCM_DTR_str)
BrokenPipeError: [Errno 32] Broken pipe

[radimkarnis]

Thank you for providing the extended log. The real issue is now clear.

The USB CDC serial console of the ESP32-S2 has some problems when the sequence of steps you've described happens. The issue is most likely on the side of the host drivers (connection fails even when the chip is manually hard-reset). This results in the 'BrokenPipeError: [Errno 32] Broken pipe'.

This can be patched in esptool, as ESP32-S2 doesn't really need a second connection attempt. A fix is already in the works!