Introduce nydus-v2.0/rafsv6 #234
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Simpilify refine Cargo.toml for readability. Signed-off-by: Liu Jiang <[email protected]>
Refine `trait BlobBackend` by: - introducing `trait BlobReader` and `trait BlobWriter` - `BlobBackend` now is a manager to manage reader and writer - `BlobBackend` will be extend to support GC later Signed-off-by: Liu Jiang <[email protected]>
Refine LocalFs storage backend by: - migrate to the new Backend interfaces - redesign the access trace and prefetching logic - add unit test cases Signed-off-by: Liu Jiang <[email protected]>
Refine request.rs by: - add some documentation - rename build_client() as build_connection() - add Request::shutdown() to teardown background working threads - add some unit test cases Signed-off-by: Liu Jiang <[email protected]>
Refine oss storage backend by: - migrate to the new backend interfaces - add more unit tests Signed-off-by: Liu Jiang <[email protected]>
Refine registry storage backend by: - migrate to the new backend interfaces - add more unit tests Signed-off-by: Liu Jiang <[email protected]>
The request actually stands for connection, so rename it. Signed-off-by: Liu Jiang <[email protected]>
enhance connection::respond() to simplify code. Signed-off-by: Liu Jiang <[email protected]>
Rename Backend::release() to Backend::shutdown(), and explicitly shutdown all background working threads. Signed-off-by: Liu Jiang <[email protected]>
enhance copyv() and MemSliceCrusor in util by: - stricter validation of input parameter - simplify code - add more unit test cases - fix a bug in BlobReader::readv Signed-off-by: Liu Jiang <[email protected]>
Signed-off-by: Liu Jiang <[email protected]>
Refine RafsCache related code and documentation by: - adding more documentation - introduce function compress::compute_compressed_gzip_size() - validate data size returned by backend - rename BlobCache::release() as BlobCache::destroy() Signed-off-by: Liu Jiang <[email protected]>
Explicitly group Rafs v5 specific logic by: - move Rafs v5 specific data structures into submodule v5 - also rename device::RafsXXX to device::BlobXXX. Signed-off-by: Liu Jiang <[email protected]>
Refine ChunkMap implementations by: - get rid of dependency on v5 data struct by using BlobChunkInfo instead of BlobV5ChunkInfo - export DigestedChunkMap and IndexedChunkMap directly - rename ChunkMap::has_ready() as ChunkMap::is_ready() - rename ChunkMap::has_ready_nowait() as ChunkMap::is_ready_nowait() - rename ChunkMap::finish() as ChunkMap::notify_ready() - Make Status::wait_for_inflight() return current value Signed-off-by: Liu Jiang <[email protected]>
There's possible race condition in IndexedChunkMap::new(), so refine it to close those race windows. Introduce IndexedChunkMap v1, which has a flag for "all chunk ready". Also add more unit test cases. Signed-off-by: Liu Jiang <[email protected]>
Introduce NoopChunkMap, which will simplify report all chunk as ready. The NoopChunkMap may be used support disk based blob storage. Signed-off-by: Liu Jiang <[email protected]>
Improve documentation for chunkmap. Signed-off-by: Liu Jiang <[email protected]>
Move RafsCache into cache::v5::BlobV5Cache, so all rafs V5 related code into dedicated submodule. Signed-off-by: Liu Jiang <[email protected]>
Introduce trait BlobCacheMgr/BlobCache and migrate dummycache.rs to the new interfaces. blobcache.rs will handled with dedicated changes. Signed-off-by: Liu Jiang <[email protected]>
Introduce `BlobIoChunk` enumeration and refactor `BlobV5Io/BlobV5IoDesc` as `BlobIoDesc/BlobIoVec`. Now BlobIoChunk/BlobIoDesc/BlobIoVec becomes rafs version independent, which is essential to enhance blobcache.rs to support rafs v6. With this refactor, the abstraction becomes much more elegant. Signed-off-by: Liu Jiang <[email protected]>
Enhance the DummyCacheMgr by: - explicitly control prefetching behavior - more docs - optimize DummyCacheMgr::read() for rafs v5 Signed-off-by: Liu Jiang <[email protected]>
Enhance BlobInfo to provide more information about compression and digest algorithm etc, then we could make these configuration as per blob instead of per cache manager. Signed-off-by: Liu Jiang <[email protected]>
Make all fields of BlobInfo as private and provide access methods for them. Also rename BlobChunkInfo::block_id() as BlobChunkInfo::chunk_id(), as there's no any annotation of 'block' within the storage subsystem. Signed-off-by: Liu Jiang <[email protected]>
Temporarily disable blobcache, will be added back later. Signed-off-by: Liu Jiang <[email protected]>
Refactor interfaces of the cache subsystem to move main operation from BlobCacheMgr to BlobCache. Signed-off-by: Liu Jiang <[email protected]>
- rename MergedBackendRequest to ChunkIoMerged - rename ChunkIoInfo to ChunkIoTag - with help of BlobIoChunk, make ChunkIoMerged independent of v5 - enhance state validation in MergedBackendRequest::new/merge() Signed-off-by: Liu Jiang <[email protected]>
Add read(), read_chunks(), read_raw_chunk() and process_raw_chunk() to BlobCache, so a BlobCache object could be used to access blob data. Rename BlobCache::is_chunk_cached() to is_chunk_ready(). Also optimize copyv() to avoid create a Vec<&[u8]> from Vec<Vec<u8>>. Signed-off-by: Liu Jiang <[email protected]>
BlobV5Cache actually represents a blob cache object instead of a blob cache manager object, so change pub trait BlobV5Cache: BlobCacheMgr" to "pub trait BlobV5Cache: BlobCache". With this refactor, we could remove much v5 specific code. Also refactor the factory module to explicity manage blob cache manager and provide interface to support gc. Signed-off-by: Liu Jiang <[email protected]>
Now cache::v5::BlobV5Cache is not used anymore, so remove it. Signed-off-by: Liu Jiang <[email protected]>
One big difference is that v6 blob may use different compressor and digester with image's default ones. Signed-off-by: Liu Jiang <[email protected]> Signed-off-by: Liu Bo <[email protected]>
Special inode like device, char, pipe are now supported. Signed-off-by: Liu Bo <[email protected]>
Given OCI image spec supports sock file, lets stick to it. Signed-off-by: Liu Bo <[email protected]>
Although we still use v5 inode/chunk, we can use v6 wrapper for better naming. Signed-off-by: Liu Bo <[email protected]>
Set rafs v6's blocks of superblock to a non-zero value. Signed-off-by: Liu Bo <[email protected]>
This adds a few verbose errors for validate(). Signed-off-by: Liu Bo <[email protected]>
Now that we've put a sanity check in RafsV6Blob's validate(), this makes blob's uncompressed size align to 4096 which is EROFS_BLOCK_SIZE in use. Signed-off-by: Liu Bo <[email protected]>
UT needs to be updated after Node honors "version". Signed-off-by: Liu Bo <[email protected]>
In order to exit more gracefully. Signed-off-by: Liu Bo <[email protected]>
size_with_xattr() is only for v6, lets put a dummy value for v5. Signed-off-by: Liu Bo <[email protected]>
Even we exclude this option from nydusd with feature fusedev applied, it can still be used from API mount. Like libfuse, we just want to provide a simple example that built-in fuse-rs works well and test can be composed to verify code in nydusd/rafs when linking to virtiofs front-end. Signed-off-by: Changwei Ge <[email protected]>
Async prefetch is always started when nydusd starts. Files that have to be prefetched in a list will be sent to filecache async workers manager. After all file hint are sent via channel, we should call `stop_prefetch` to release resources synchronously, such as worker threads. Signed-off-by: Changwei Ge <[email protected]> Signed-off-by: Gao Xiang <[email protected]>
Temporary workadround. Otherwise, the rest prefetch requests will be dropped directly. Signed-off-by: Gao Xiang <[email protected]>
Some functions will be used in the near future, so `allow_dead` code to silent compiling warnings temporarily. Signed-off-by: Changwei Ge <[email protected]>
Builder panic on multi-layers build: ``` thread 'main' panicked at 'range end index 4293918793 out of range for slice of length 1048576', src/bin/nydus-image/core/node.rs:307:39` ``` The `dir_set_v6_offset` method will rewrite the inode size of regular file, that causes the above issue. Signed-off-by: Yan Song <[email protected]>
Signed-off-by: Liu Bo <[email protected]>
If blob's ci compressor is None, the uncompressed buffer should be read out and returned. Signed-off-by: Liu Bo <[email protected]>
Thus to make it more suggestive Signed-off-by: Changwei Ge <[email protected]>
It is not always rafs. Signed-off-by: Peng Tao <[email protected]>
For rafs, we would like to disable open. For passthroughfs, we would like to enable open and killpriv_v2. Signed-off-by: Peng Tao <[email protected]>
To ask kernel to cache file and directory contents instead of dropping them across open/close. Signed-off-by: Peng Tao <[email protected]>
So that we can enable necessary options for passthroughfs when running rafs and passthroughfs under the same vfs. Signed-off-by: Peng Tao <[email protected]>
Current inode loading logic wrongly used inode number to load inode from inodes table. Signed-off-by: Changwei Ge <[email protected]>
Clean clippy warning by `cargo fmt`. Signed-off-by: Liu Jiang <[email protected]>
liubogithub
approved these changes
Dec 3, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patchset is a little big(huge), which does: