Builder: support tar build #351
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imeoer
force-pushed
the
nydus-tar-build
branch
6 times, most recently
from
90799d1
to
64394b8
Compare
imeoer
force-pushed
the
nydus-tar-build
branch
2 times, most recently
from
0b40a9f
to
226b58c
Compare
imeoer
force-pushed
the
nydus-tar-build
branch
2 times, most recently
from
64b0a70
to
4286bcf
Compare
imeoer
force-pushed
the
nydus-tar-build
branch
4 times, most recently
from
9c05f17
to
a923990
Compare
hsiangkao
approved these changes
Apr 6, 2022
bergwolf
reviewed
Apr 6, 2022
| Some(_) => { | ||
| // Insert removal operations at the head, so they will be handled first when | ||
| // applying to lower layer. | ||
| nodes.insert(0, node); |
There was a problem hiding this comment.
nits: Insert whiteouts at the head, xxxx
| flags = Some(current_flags); | ||
| } | ||
|
|
||
| let parent_blobs = rs.superblock.get_blob_infos(); |
There was a problem hiding this comment.
Why not simply bail if len(parent_blobs) > 1? Then the parent_blob_added logic can be dropped.
There was a problem hiding this comment.
It is assumed that the nydus-image create at each layer and nydus-image merge commands use the same chunk dict bootstrap, so there is at most one blob from the parent bootstrap and all other blobs from the chunk dict bootstrap. Added a comment.
There was a problem hiding this comment.
Added more detailed comment.
|
|
||
| // Blob list recorded in manifest annotation should be equal with | ||
| // the blob list recorded in blob table of bootstrap | ||
| if !reflect.DeepEqual(bootstrap.Blobs, blobListInLayer) { |
There was a problem hiding this comment.
We should still validate that all the blobs in bootstrap are in the image manifest, to make sure we don't reference any non-existing blobs in bootstrap.
imeoer
force-pushed
the
nydus-tar-build
branch
2 times, most recently
from
d3629cb
to
12fd452
Compare
Add `--whiteout-spec none` option for builer command, which will build all `.wh.*` and `.wh..wh..opq` files into bootstrap. This is used for tar build, packing blob and bootstrap into a tar stream, so that correspond to the OCI tar layer one by one. Signed-off-by: Yan Song <[email protected]>
Match blob table by fs version in `boostrap.dump()`. Signed-off-by: Yan Song <[email protected]>
The `merge` sub-command merges multiple nydus boostraps (from every layer of image) to a final boostrap. Signed-off-by: Yan Song <[email protected]>
Nydus tar build converts a OCI formatted tar stream to a nydus formatted tar stream. The nydus blob tar stream contains blob and bootstrap files with the following file tree structure: /image ├── image.blob ├── image.boot So for the chunk of files in the nydus boostreap, a blob compressed offset of 1024 (size_of(tar_header) * 2) is required. Add `--blob-offset` option to builder to support this case. Signed-off-by: Yan Song <[email protected]>
Add `--chunk-dict` option support for merge sub-command, We need get blob list from chunk-dict bootstrap to fill blob table in final bootstrap. Signed-off-by: Yan Song <[email protected]>
Add `--prefetch-policy` option support for merge sub-command, this option has the same function as in the create sub-command, it is used to enable prefetch to optimize inode/chunk layout. Signed-off-by: Yan Song <[email protected]>
Signed-off-by: Yan Song <[email protected]>
Set node's layer index to distinguish same inode number (from bootstrap) between different layers. Signed-off-by: Yan Song <[email protected]>
1. reuse options with create sub-command; 2. add context error message for user friendly; Signed-off-by: Yan Song <[email protected]>
For the tar build scene, the regular layer represents blob+bootstrap, for an empty data layer (for example `chmod +x`), the blob table in bootstrap doesn't include the blob entry of this layer, so we need remove the check of comparing the number of blobs and the blob order between bootstrap and layers on the manifest, just ensure nydus blobs in the blob table of bootstrap should all appear in the layers of manifest. Signed-off-by: Yan Song <[email protected]>
Keep final bootstrap following superblock flags of source bootstraps. Signed-off-by: Yan Song <[email protected]>
The behavior change of RafsSuper::walk_inodes api break diff build, adjust it to ignore non-regular file handle. Signed-off-by: Yan Song <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce a new build mode named tar build in nydusify and builder (nydus-image).
In order to correspond to the OCI image layer, we need a new build mode, that is
to convert the OCI-formatted tar stream directly into a nydus-formatted tar stream.
The nydus tar stream contains blob and bootstrap files, and the workflow is divided
into two steps, Convert and Merge.
Convert: converts each layer of the OCI tar stream to a nydus tar stream.
Merge: reads the bootstrap files from each layer of the nydus tar stream and merges
them into a final bootstrap.
Support merge sub-command in builder, it will be used in converter package, and this
package will be imported by acceleration-service and buildkit to support the conversion
or build of nydus image.