2 Bugs in new fatfs_create_spiflash_image() feature (IDFGH-6383)

[m24h]

Environment

• Development Kit: none
• Kit version (for WroverKit/PicoKit/DevKitC): none
• Module or chip used: ESP32-WROOM-32
• IDF version: master at 2021-12-9
• Build System: idf.py
• Compiler version : (crosstool-NG esp-2021r2) 8.4.0
• Operating System: Windows
• (Windows only) environment type:ESP Command Prompt
• Using an IDE?: No
• Power Supply: USB

Problem Description

When I use the new fatfs_create_spiflash_image() feature in fatfs componenet, like following usage in CMakeLists.txt:
--------------------begining--------------------------------
fatfs_create_spiflash_image(resource resource FLASH_IN_PROJECT)
---------------------end-------------------------------

Then something got wrong like following logs at the building time:
-----------------------------begining------------------------------------------------
Traceback (most recent call last):
File "D:/ESPIDF/components/fatfs/wl_fatfsgen.py", line 196, in
wl_fatfs.wl_generate(input_dir)
File "D:/ESPIDF/components/fatfs/wl_fatfsgen.py", line 175, in wl_generate
self.plain_fatfs.generate(input_directory=input_directory)
File "D:\ESPIDF\components\fatfs\fatfsgen.py", line 190, in generate
self._generate_partition_from_folder(folder_name, folder_path=path_to_folder, is_dir=True)
File "D:\ESPIDF\components\fatfs\fatfsgen.py", line 183, in _generate_partition_from_folder
self._generate_partition_from_folder(os.path.join(smaller_path, path), folder_path=folder_path)
File "D:\ESPIDF\components\fatfs\fatfsgen.py", line 183, in _generate_partition_from_folder
self._generate_partition_from_folder(os.path.join(smaller_path, path), folder_path=folder_path)
File "D:\ESPIDF\components\fatfs\fatfsgen.py", line 171, in _generate_partition_from_folder
content = file.read()
UnicodeDecodeError: 'gbk' codec can't decode byte 0x83 in position 1418: illegal multibyte sequence
----------------------------------end--------------------------------
it seens that fatfs partition generator wants to encode my raw file and failed.
I modified the file wl_fatfsgen.py, at line 168, added "wb" instruction as "with open(output_path, 'wb') as output:", and modified the file fs_object.py, at line 50, removed "encode()" method as "content_as_list = content_part", then it seens to work right.

But why does fatfs generator want to encode a file being put to a partition, is it enough to just treat the file as an unkown binary raw file.

Another bug is that the created partion burned into flash, will pollute the partition just next to it, cause mounting failure like following logs:
--------------------------begining-------------------------
W (260) vfs_fat_spiflash: f_mount failed (13)
--------------------------end-------------------------

I reserved a gap between those two partition, the problem seens to be temporary solved.

[dobairoland]

Thanks for your report. We will investigate this soon.

Could you please share the output of python --version run in your terminal?

[sio13]

Hi @m24h, thanks a lot for sharing the information about the issue. "encode()" seems to be redundant and we'll consider removing it. Thank you for this as well. Regarding the second problem, would it be possible to share your partition table?

[m24h]

Hi, Sorry, my partition table has been modified since that, and I can't find out the exact old version with that problem, but it might be likely this way: ----------------------------------------- # Name, Type, SubType, Offset, Size, Flags # Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap nvs, data, nvs, 0x9000, 0x6000, phy_init, data, phy, 0xf000, 0x1000, factory, app, factory, 0x10000, 0x200000, resource, data, fat, , 0x80000, storage, data, fat, , 0x100000, ---------------------------------------- "resource" is the partition generated by fatfs_create_spiflash_image(), and "storage" is another FAT/VFS partition with wear_levelling feature. And now I'm not very sure about that issue, I didn't test it more than several times, just leaved a gap and found that the problem disappeared. My project changed too much, even I tried to modify some codes, tried to take the problem back, and I found that it seems all right, no failure happened. From: Martin Gaňo Date: 2021-12-16 18:07 To: espressif/esp-idf CC: m24h; Mention Subject: Re: [espressif/esp-idf] 2 Bugs in new fatfs_create_spiflash_image() feature (IDFGH-6383) (Issue #8042) Hi @m24h, thanks a lot for sharing the information about the issue. "encode()" seems to be redundant and we'll consider removing it. Thank you for this as well. Regarding the second problem, would it be possible to share your partition table? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[m24h]

It's "Python 3.8.7" From: Roland Dobai Date: 2021-12-09 16:19 To: espressif/esp-idf CC: m24h; Author Subject: Re: [espressif/esp-idf] 2 Bugs in new fatfs_create_spiflash_image() feature (IDFGH-6383) (Issue #8042) Thanks for your report. We will investigate this soon. Could you please share the output of python --version run in your terminal? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[sio13]

Hi @m24h, thanks a lot for providing the information. I will test it on my own and resolve the encoding issue.