tart push: use fixed size chunks to allow for better deduplication #821
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fkorotkov
reviewed
May 14, 2024
| while let (compressedData, uncompressedSize, uncompressedDigest) = try compressNextLayerOfLimitBytesOrMore(mappedDisk: mappedDisk, offset: offset) { | ||
| offset += uncompressedSize | ||
| // each equal ``Self.layerLimitBytes`` bytes or less due to LZ4 compression | ||
| for data in mappedDisk.chunks(ofCount: layerLimitBytes) { |
There was a problem hiding this comment.
Should we put layerLimitBytes in the manifest in case it's gonna change at some point?
There was a problem hiding this comment.
I think it's already there by the virtue of passing the UInt64(data.count) as uncompressedSize argument to OCIManifestLayer(...):
tart/Sources/tart/OCI/Manifest.swift
Lines 94 to 96 in 13d5ddb
fkorotkov
approved these changes
May 14, 2024
This was referenced May 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this PR, we'll yield exactly 2 layers per GB of VM disk image when doing
tart push, which at the moment equates to:ghcr.io/cirruslabs/macos-sonoma-base:latestinstead of the current 38 layersghcr.io/cirruslabs/macos-runner:sonomainstead of the current 162 layersThis slightly increases the bookkeeping costs, but allows for better deduplication in
tart pull(in the upcoming PR).Our calculations have shown that current layer deduplication ratios are:
ghcr.io/cirruslabs/macos-sonoma-base:latestimagesghcr.io/cirruslabs/macos-runner:sonomaimagesWith the fixed-size chunks of 500 MB, the deduplication ratios will be as follows:
ghcr.io/cirruslabs/macos-sonoma-base:latestimagesghcr.io/cirruslabs/macos-runner:sonomaimages