FWFS Mountpoint Support #2315
Merged
FWFS Mountpoint Support #2315
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Remove object stores, simplify code Support mounting filesystem instead of object store Don't align filesystem images (simplification)
mikee47
changed the title
mikee47
force-pushed
the
feature/fwfs-mountpoints
branch
from
188f125 to
540539b
Compare
mikee47
force-pushed
the
feature/fwfs-mountpoints
branch
2 times, most recently
from
092e52c to
9f548ca
Compare
mikee47
force-pushed
the
feature/fwfs-mountpoints
branch
2 times, most recently
from
9c2d1c8 to
09bd1ba
Compare
slaff
pushed a commit
that referenced
this pull request
Apr 29, 2021
This PR simplifies FWFS further by storing object identifiers as image offsets. Advantages: - No cache required - Faster because we can jump directly to the referenced record - Debugging and introspection is much easier The references are stored as packed integers (1 - 4 bytes). Note: Existing images need to be rebuilt. This change should have been included in #2315 as the use of numeric object IDs was part of the abstraction for object stores.
slaff
pushed a commit
that referenced
this pull request
Sep 27, 2021
**Remove object stores** LittleFS is an excellent choice for flexible metadata handling, making further development of FWFS writeable object stores a rather pointless excercise. This PR reverts FWFS back to pre-object store model which is faster and simpler. This is largely an internal change and does not affect applications, but does simplify operation considerably as the filing system is now just mounted directly on a partition. **Mountpoint support** Images can be built with nominated directories as mount-points. Instead of object stores, filesystems (LittleFS, SPIFFS, etc) can be mounted there instead using the new `IFileSystem::setVolume` method. See `Basic_IFS` sample for a demonstration of how this works. **Remove object alignment** This is a **Breaking change** and will require existing applications to rebuild their FWFS images. FWFS started life as a binary image linked into the firmware (IMPORT_FSTR) so benefitted from having object headers word-aligned. This was fast but inflexible, and with a partition-based system alignment is not necessary so the code can be simplified by removing this. It also saves a little space which could be significant for large numbers of small files, or files with large numbers of small attributes. **Fixes** Volume ID missing from returned FWFS filesystem information. LittleFS `stat` information sometimes a bit weird because structure initialisation not performed. Additional parameter checking to avoid exceptions calling filesystem functions if not mounted. Additional checks on paths, use general `isRootPath()` function. Fail FWFS file open requests with write access **Other changes** Add `FWFS_CACHE_SPACING` build variable to balance RAM usage vs. lookup speed. Allocate `opendir` descriptors on heap instead of consuming a regular file handle. This allows recursive directory enumeration without running out of handles. Add performance module to test application to help evaluate filesystems. Only generate MD5 Hash attributes for files. Previously directories were included, which is not necessary. Improve FWFS performance by caching root directory.
slaff
pushed a commit
that referenced
this pull request
Sep 27, 2021
This PR simplifies FWFS further by storing object identifiers as image offsets. Advantages: - No cache required - Faster because we can jump directly to the referenced record - Debugging and introspection is much easier The references are stored as packed integers (1 - 4 bytes). Note: Existing images need to be rebuilt. This change should have been included in #2315 as the use of numeric object IDs was part of the abstraction for object stores.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Remove object stores
LittleFS is an excellent choice for flexible metadata handling, making further development of FWFS writeable object stores a rather pointless excercise. This PR reverts FWFS back to pre-object store model which is faster and simpler. This is largely an internal change and does not affect applications, but does simplify operation considerably as the filing system is now just mounted directly on a partition.
Mountpoint support
Images can be built with nominated directories as mount-points. Instead of object stores, filesystems (LittleFS, SPIFFS, etc) can be mounted there instead using the new
IFileSystem::setVolumemethod. SeeBasic_IFSsample for a demonstration of how this works.Remove object alignment
This is a Breaking change and will require existing applications to rebuild their FWFS images.
FWFS started life as a binary image linked into the firmware (IMPORT_FSTR) so benefitted from having object headers word-aligned. This was fast but inflexible, and with a partition-based system alignment is not necessary so the code can be simplified by removing this. It also saves a little space which could be significant for large numbers of small files, or files with large numbers of small attributes.
Fixes
Volume ID missing from returned FWFS filesystem information.
LittleFS
statinformation sometimes a bit weird because structure initialisation not performed.Additional parameter checking to avoid exceptions calling filesystem functions if not mounted.
Additional checks on paths, use general
isRootPath()function.Fail FWFS file open requests with write access
Other changes
Add
FWFS_CACHE_SPACINGbuild variable to balance RAM usage vs. lookup speed.Allocate
opendirdescriptors on heap instead of consuming a regular file handle. This allows recursive directory enumeration without running out of handles.Add performance module to test application to help evaluate filesystems.
Only generate MD5 Hash attributes for files. Previously directories were included, which is not necessary.
Improve FWFS performance by caching root directory.