What are your plans about zstd:chunked

[Romain-Geissler-1A]

Hi,

I would like to know if you already have a roadmap wrt zstd:chunked. I have actually lots of hope to reduce internal network and disk usage with this format for use inside Amadeus, although I didn't used it yet. I guess you definitely have some plans about it, and would like to know if you can share it publicly already.

Couple of questions:

• Will you eventually switch the default compression format to zstd:chunked ? I am pretty sure you are waiting that Docker 22.06 is released, I am too !
• Will you eventually switch the enable_partial_images = true option by default in storage.conf ?
• Are you aware of any issues with famous registries vendors (for example Dockerhub, or JFrog's Artifactory) with this format ? I know you mentionned Quay.io had issues with zstd when you introduced it ?
• If you want to keep all this experimental for now, are you interested to have companies like Amadeus test this internally (we don't use podman a lot yet to be honest, we slowly start to use it for develpoment workflows, and of course, we could try to switch it on cri-o side for our openshift nodes) ?
• Are you aware of "big" blockers except the fact that Docker doesn't support zstd yet ? When I announced widely to the whole Amadeus developers that we would migrate to zstd:chunked as soon as possible (ie after Docker 22.06 is released and deployed on our Jenkins nodes), one came to me with a tool I didn't know about: jib that apparently shall be considered a standard to build containers in the java world. I asked @rquinio1A to open this issue Support pulling zstd-compressed base layers GoogleContainerTools/jib#3714 to track this problem (that we have to fix before we internally use ztsd:chunked).

Cheers,
Romain

[mheon]

@giuseppe @rhatdan PTAL

[rhatdan]

• Will you eventually switch the default compression format to zstd:chunked ? I am pretty sure you are waiting that Docker 22.06 is released, I am too !
  Yes we would love to change to default to zstd:chunked. Right now the plan, as I understand it, is to start pushing image in two different formats (Which will be able to be configured in containers.conf). Then Podman and friends will be smart enough to use the zstd by default. Older versions of Podman and Docker will still use the gzip format.

• Will you eventually switch the enable_partial_images = true option by default in storage.conf ?
  Yes that is the plan.

• Are you aware of any issues with famous registries vendors (for example Dockerhub, or JFrog's Artifactory) with this format ? I know you mentionned Quay.io had issues with zstd when you introduced it ?
  @giuseppe has tested at Docker.io and now at Quay.io and it works. I am not sure if he has access to test against Artifactory, could you?

• If you want to keep all this experimental for now, are you interested to have companies like Amadeus test this internally (we don't use podman a lot yet to be honest, we slowly start to use it for develpoment workflows, and of course, we could try to switch it on cri-o side for our openshift nodes) ?
  We would love to have you test this internally. We plan on getting this into CRI-O as soon as we have a resolution to the installed base of Docker handled.

• Are you aware of "big" blockers except the fact that Docker doesn't support zstd yet ?
  Nope only blocker I know of is Docker and originally quay.io not supporting the images, but this has been rectified. There could be other registry issues, though. As we have seen in the past with support for OCI images in general.

@giuseppe @flouthoc any other comments.

[Romain-Geissler-1A]

Ok thanks for the update on this.

We use JFrog's Artifactory internally. Since I am still blocked by the Docker release, the only thing I did was to test that I can push zstd:chunked image (in practice it was a manifest list, using the --format=oci), and I can only say that both podman push and then podman pull/run with this pushed image worked. Once Docker will be released and loaded in our Jenkins nodes, I will start generating zstd:chunked images by default so this will test it more.

Support for partial pull is still not in cri-o ? I thought this implementation would be in the common go packages between podman and cri-o, and that this support would come "for free" thanks to that. So even in OpenShift 4.11 I won't be able to enable enable_partial_images in the crio.conf, ok, it's noted. Does cri-o support already zstd (without the partial pull part) ?

About the idea to push using two manifests using two distinct format is the easiest way to be 100% compatible, yet a the cost of doubled upload time and double storage usage on registry side, so it shall be considered with care for the ones who will do it. Our Artifactory is already full very often. So on our side I will most likely choose the breaking option of pushing only using zstd, and forcing my users to use either podman, or Docker >= 22.06.

Beyond zstd:chunked, is the long-awaited "OCI v2" an area will try to explore for in the coming years ? I don't know all the plans about OCI v2, but like many what I wish is that eventually we find a way to share files also when an image is pushed/stored in a registry (as we have LOTS of images with LOTS of duplicated big files in them). I wonder if somehow we could upload each and every file tar + zstd files to the registry under the blob directory, but still act as if it was a single layer tar + zstd file obtained by the concatenation of all these files, and just "teach" the registry in which order the final tarball shall be obtained. This way the registry can serve a "fake" tarball file containing the whole layer to "old" clients, and new clients would download each individual blobs, one per file. On implementation side, if the registry side doesn't want to "re-create" this fake file at each request, maybe on XFS-like filesystem a whole tarball file made of ranged reflinks to the individual file could be created at push time to avoid the double storage ? My knowledge about how reflinks work is very limited, so maybe this is not possible.

[Malix-Labs]

Hi @rhatdan!

Has there been any progress or changes since your reply?

[giuseppe]

we have already enabled partial pulls by default in the client, so if a zstd:chunked image is found, Podman will perform a partial pull even if you've not configured it to do so.

Next steps is to change buildah/podman to build zstd:chunked by default, we are aiming at Fedora 41: https://fedoraproject.org/wiki/Changes/zstd:chunked

[Malix-Labs]

Thanks for that info and to your fast answer @giuseppe!