RFD - SD Card support

[devsaurus]

I'm currently looking into the bits and pieces that are needed to add support for SD Cards to the NodemMCU firmware. Even though that the topic is quite clear, I'd like to understand the requirements of potential use cases a bit better before starting with code. So please feel free to feed back to my specific questions or update the topic with additional items.

Low level driver

This will be based on SPI mode as I don't feel like venturing into SD protocol topics on undocumented hardware. It's a drawback performance-wise but common use in the Arduino world. A good start for a driver code is SdFat which implements all SD related functions in one file, supporting

• Standard cards with up to 2 MByte
• SDHC cards with 4 MByte and above
• Old MMCs eventually

Reading and writing raw blocks of the card could be exposed as Lua API, but I don't see a need for that at the moment. Is there any interest in accessing the card without a file system?

File system

Most prominent will be support for FAT16/FAT32 since it allows for simple data exchange with PCs. There appear to be different shades of FAT support available, trading in memory consumption for features. Thus the most interesting items for selecting an fs implementation are lurking here. How important are the following aspects for your use case?

• Long file names vs. 8.3 scheme
• Support for directories in read mode
• Creating directories
• Correct timestamps for file creation and modification
• gets() / puts() like API on top of chunk based operations
• File attributes (read-only, system, archive, hidden)
• Multiple cards / volumes
• Others

FatFs would cover all the items mentioned above but I'm open for other proposals.

What about other file systems like ext2/3/4?

[devyte]

For what it's worth, here's what would be my usage:

• Card capacity: SDHC at minimum, SDXC would be nice to go beyond 32GB.
• Low level access: no
• Long filenames vs. 8.3: long file name support is a must, or interop with windows is too limited
• support for dirs in read mode: yes
• creating dirs: a must
• timestamps: a must, I believe there are even some OSs that will choke if file timestamps are bad
• gets()/puts(): nice to have. A standard open/close/read/write would be enough
• File attributes: read-only at least should be handled correctly
• Multiple cards/volumes: multiple cards: no. Multiple partitions on the same card: yes
• Other fs: no, fat32 would be enough

[pjsg]

Large card sizes are a must.
creating directories: yes
fat32 is good enough.

It would be nice to have an API that was compatible with the SPIFFS API (or at least a way to build a compatible API).

[devsaurus]

I've been working on a prototype during the last days and have basic SD access in place now. While incrementally adding FAT file system functions I get a better view on the Lua API thus it's time for an update and call for feedback.

SPI layer

The user is supposed to initialize SPI upfront. This allows some degree of freedom regarding the system configuration:

• maximum overall SPI frequency
• potentially other SPI slaves

SD card access

The SD card is identified by its SS/CS pin and can theoretically coexist with other slaves on the same bus. The driver will temporarily reduce SPI clock to 400 kHz during card initialization and switches back to the user configuration.

File system layer

This is based on ChaN's FatFS R0.12 code.

I see the possibility to have a moderate hierarchical object design while addressing a file or folder is done in a flat way with paths across volumes and directories. This feels more like the traditional approach and avoids artificial mappings between an object tree and FatFS' path definition.

• fatfs.mount(part) - mounts a partition (primary 0-3) and returns a volume object
• fatfs.opendir(path) - opens a folder and returns a directory object
• fatfs.open(path, mode) - opens/creates a file and returns a file object
• fatfs.stat(path) - returns an info object (file or folder)
• fatfs.mkdir(path) - create a folder
• fatfs.remove(path) - delete a file or folder
• fatfs.rename(old_name, new_name) - rename a file or folder
• fatfs.chmod() - change attributes (RO, SYS, ARCH, HIDDEN)
• fatfs.utime() - change timestamp
• fatfs.chdir() - change current working directory
• fatfs.chdrive() - change current working driver (partition)
• fatfs.getcwd()

Volume object

• vol:umount() - unmount volume
• vol:getfree() - get total size and free size on the volume
• vol:getlabel() - get volume label

Directory object

• dir:read() - read next item in directory, returned as an info object
• dir:close() - close dir

File object

• file:read() - read chunk of data from file
• file:readline() - read next line from file
• file:write() - write chunk of data to file
• file:writeline() - write a string to the open file and append '\n' at the end
• file:seek() - move read/write pointer, expand size
• file:eof() - query end-of-file
• file:size() - get file size
• file:tell() - get current read/write pointer
• file:error() - check error condition
• file:flush() - flush buffered data
• file:close() - close access to file

Info object

• info:size() - get size
• info:name() - get item name
• info:date() - get modification date
• info:time() - get modification time
• info:is_dir() - query file or directory
• info:is_rdonly() - query read-only attribute
• info:is_hidden() - query hidden attribute
• info:is_sys() - query sys attribute
• info:is_arch() - query archive attribute
• info:close() - close object

BTW I'm developing this on a Wemos D1 Mini with Micro SD shield - perfect match 😉.

[devyte]

Would it make sense to try to mimic the Lua api a bit closer? What I mean is:
Lua 5.1 manual

• add file:flush(), if it makes sense
• file:lseek() -> file:seek()
• maybe add file:setvbuf(), if it makes sense
• os.remove()/.rename()

If not, it's ok, I don't think this is a big deal. It's just that I think it's best to stick to standards where possible :)

[devsaurus]

Thanks for the feedback @devyte.

add file:flush(), if it makes sense

Ok

file:lseek() -> file:seek()

Ok

maybe add file:setvbuf(), if it makes sense

I don't see a way atm to influence FatFS' buffering strategy.

os.remove()/.rename()

unlink() -> rename() is a good point. It's also in line with the spiffs module.

I've updated my previous post accordingly.

[devsaurus]

FAT stores modification time in year/month/day etc. format. Support for correct timestamps requires date&time info to be broken down into these components.
My current idea is to provide a callback hook that gets fired when the filesystem needs current time. User is then free to choose how to generate this info. Preferably from rtctime.get(), but this one delivers seconds since epoch - not y/m/d/h/m/s.
I've opened #1370 to enable easy conversion from rtctime to FAT compatible format.

[devsaurus]

Pushed a first version to my fatfs branch. Some more clean-up and testing required.
Please feel free to feed back with any comments (no docs yet).

[marcelstoer]

User is then free to choose how to generate this info. Preferably from rtctime.get()...

You say "(potentially) dependent modules" and I say #386 😉

[devsaurus]

The filesystem functions can hit a couple of blocking points and will report these as result codes to the Lua wrapper. This information is currently propagated to Lua land and the script is responsible for any error detection. Eg. fatfs.open("file", "r") returns nil instead of the file object in case the file doesn't exist. It could, on the other hand, also throw an error().

Is there a preferred way how to report these errors?

[marcelstoer]

I read somewhere about Lua

The usual practice is to throw errors when the program cannot recover (bad file handle for instance) and signal errors when it can recover (file not found for instance)

Returning nil would be considered signaling errors in this case, right?

To me it also depends on whether there's some kind of "does-file-exist" function available. If clients have the chance to check for a file's existence prior to opening it it may be ok to throw an error if they don't and the file is missing indeed. If clients have no chance to verify first it'd be a bit "unfair" to throw an error.

[eku]

@marcelstoer I agree with you, but this applies only to single-threaded environments. The result of the test may be invalid immediately after the call.

[marcelstoer]

this applies only to single-threaded environments

Yes, and even then there's no 100% guarantee although for all practical purposes it can be assumed.

[devsaurus]

Thanks for the feedback.
I tend to stick to the paradigm "throw errors when the program cannot recover and signal errors when it can recover" for the time being.

[devsaurus]

The more this module evolves, the more I get the impression that the whole file system topic should be more generic. The separation between FAT SD cards and SPIFFS flash is something which feels more and more like a suboptimal approach.
Unifying both is quite a task in itself, thus I propose the following roadmap:

1. Rename this module to ufs - "Unified File System" - and continue with a PR which adds FatFs as the first implementation for ufs.
2. PR to Integrate SPIFFS into ufs. There are a couple of features that are not supported with the current file API; eg. opening multiple files. This will deprecate file in the long term.
3. PR to push the unified system from Lua down to the C layer. This opens access to SD card storage for other modules and will finally enable to execute scripts from SD card.

The ufs API will increment with moderate, back-ward compatible adjustments along the path, I assume.
Let me know if you see any issues with this approach.

[eku]

@devsaurus vfs is a common name for the file system abstraction layer.

[marcelstoer]

Thanks Arnim for the well thought-out plan 👍. However, just as Erik I also think that "Unified File System" is not the correct term. You're a bstracting, g eneralizing or v irtualizing the file system.

[devsaurus]

Thanks @eku and @marcelstoer. vfs seems to be a suitable name.

[devsaurus]

This will deprecate file in the long term.

I've been contemplating this aspect a while now - it just feels wrong from whatever point of view. A better approach IMO is to incrementally enhance the file module itself:

1. Encapsulate spiffs and fatfs functions by a vfs layer on C level.
2. Use vfs layer from within the file module.
3. Add object-oriented methods to file while keeping the existing flat API.
4. Use vfs layer in all other C sources (mainly the Lua part itself) -> dofile() etc from SD card
5. Optional: deprecate the flat file API after some time.

The term flat API refers to the module functions that restrict Lua land to 1 open file at a time:

• file.open()
• file.read*()
• file.write*()
• file.close()
• file.flush()
• file.seek()

Steps 1.-4. keep full backwards compatibility (with single-file restriction for both spiffs and fatfs). While 3. adds API methods that enable multiple open files and directory handling.