VolumeSource: OCI Artifact and/or Image

[sallyom]

Enhancement Description

• One-line enhancement description (can be used as a release note): VolumeSource: OCI Artifact or OCI Image

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/4639-oci-volume-source

• Discussion Link:

• Primary contact (assignee): [email protected], [email protected]

• Responsible SIGs: sig-node, sig-storage

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.31
  • Beta release target (x.y):
  • Stable release target (x.y):

• Alpha

  • KEP (k/enhancements) update PR(s):
    • KEP-4639: adding OCI VolumeSource #4642
    • KEP-4639: update to most recent implementation state #4897
  • Code (k/k) update PR(s):
    • [KEP-4639] Add OCI VolumeSource CRI API kubernetes#125659
      • [KEP-4639] Update CRI API and workflow #4751
    • [KEP-4639] Add ImageVolumeSource API kubernetes#125660
    • [KEP-4639] Add ImageVolumeSource implementation kubernetes#125663
    • Nice to have:
      • [KEP-4639] Mention that fsGroupChangePolicy has no effect kubernetes#126281
      • Fix runtime panic in imagevolume CanSupport method kubernetes#126323
      • [KEP-4639] Add ImageVolumeSource node e2e tests kubernetes#126220
  • Out of tree update PR(s):
    • Add OCI Volume Source support to crictl [create|run] --with-pull kubernetes-sigs/cri-tools#1464
    • Add OCI/Image Volume Source support cri-o/cri-o#8317
    • Make ImageVolume garbage collection work cri-o/cri-o#8408
    • CRI-O: add pull-kubernetes-node-crio-cgrpv2-imagevolume-e2e test job test-infra#33071
    • Fix imagevolume test focus test-infra#33088
  • Docs (k/website) update PR(s):
    • Document OCI volume sources / KEP-4639 website#46946
    • Blog post about OCI volume sources / KEP-4639 website#46960

• Beta

  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[sallyom]

I've had informal discussions about this - there's enough interest IMO to open a KEP & I will present this issue at upcoming sig-node, sig-storage mtgs with a KEP draft

/sig node
/sig storage

[xing-yang]

Can you use the Volume Populator? It allows you to create a PVC from an extenal data source. https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1495-volume-populators

[saschagrunert]

Happy to support here from a SIG node perspective.

cc @kubernetes/sig-node-proposals

[mikebrow]

+1 happy to help from the SIG-Node/CRI and OCI image/distribution spec perspectives..

[SergeyKanzhelev]

/label lead-opted-in
/milestone v1.31

as discussed at SIG Node meeting this week, we will try and see if this can make it to 1.31

[SergeyKanzhelev]

/stage alpha

[rhuss]

For reference, in KServe a workaround for directly accessing files within an OCI image is currently implemented and available via a sidecar approach ("modelcar") by leveraging root FS system access via the /proc filesystem when shareProcessNamespace: true is set on the Pod. You can find details in the KServe documentation and in the Design Document. It actually implements the desired behavior with current means, but of course is more or less just a workaround for the OCI volume type (as discussed here and raised already a long time ago in kubernetes/kubernetes#831).

So KServe would be more than happy to leverage such a volume type, and we are happy to support any efforts in this direction.

[sreeram-venkitesh]

/stage alpha

[sreeram-venkitesh]

Hello @sallyom 👋, v1.31 Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

This enhancement is targeting for stage alpha for v1.31 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.31. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline (6th June) so that the PRR team has enough time to review your KEP before the enhancements freeze.

For this KEP, most of the above items are taken care of in #4642. We'd need to do the following:

• Update the status from provisional to implementable in the kep.yaml file here
• Create a prod-readiness yaml file as shown here.
• Update the graduation criteria in the KEP readme file.
• Make sure that the PRR questionnaire is filled.

The status of this enhancement is marked as At risk for enhancements freeze. Once the above tasks are done, I can mark it as tracked.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Let me know if you have any questions! Thank you!

[sreeram-venkitesh]

@sallyom Pinging once again as a slight reminder that we're approaching the enhancements freeze deadline on 14th June, this Friday!

[dipesh-rawat]

Hi @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach the enhancements freeze in few hours, at 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

The current status of this enhancement is marked as at risk for enhancement freeze. There are a few requirements mentioned in the comment #4639 (comment) that are addressed as part of PR #4642 which still needs to be merged.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[dipesh-rawat]

Hello @sallyom @SergeyKanzhelev 👋, 1.31 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.31, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

[sreeram-venkitesh]

/milestone clear

[bitoku]

/assign

[jenshu]

Hello @saschagrunert @sallyom 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 11th October 2024 / 19:00 PDT Thursday 10th October 2024.

This enhancement is targeting stage alpha for v1.32 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.32.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 3rd October 2024 so that the PRR team has enough time to review your KEP.

For this KEP, we would just need to update the following:

• Fill out the Test Plan section of the readme
• Fill out the Graduation Criteria section of the readme
• In your kep.yaml the latest-milestone needs to be updated to v1.32

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well.

If you anticipate missing enhancements freeze, you can file an exception request in advance. Thank you!

[sohankunkerkar]

@saschagrunert @sallyom, I'm part of the SIG-NODE KEPs wrangler team for this release. It looks like the plan for this release is to push this feature to Beta. Could one of you open a PR to move it forward?

[haircommander]

@saschagrunert is out this week, but I'm going to open a PR in his stead to get it on PRR's purview, then he'll probably correct all the incorrect stuff I've written next week 🙂

[haircommander]

update: I've opened #4897

[haircommander]

/stage beta

[haircommander]

/stage alpha
/remove-milestone v1.32
/remove-label lead-opted-in

I believe we won't make stage progress in 1.32. We can still push for subpath support but I don't think we need to track that here.

[jenshu]

1.32 Enhancements team here. Per the above comments, I've updated the 1.32 status to deferred

[tjons]

/milestone clear

[tarilabs]

Hi 👋
from the "model-spec-discussion" CNCF slack channel,
from the last meeting 2024-12-05,
we wanted to raise to this great KEP-4639 efforts,
one suggestion to be evaluated that maybe could also help see motivations for OCI Artifact to be indeed in the longer-term a goal for this KEP-4639 and the Container Runtimes.

One of the great advantage of the OCI Artifact from the OCI spec, is the possibility to define media_type (and/or artifact_type) with a semantic meaning.

This would potentially allow a user to mount only a set of determined layers instead of the whole OCI Image (as it is the case in practice today with alpha), ie something ~similar to:

  volumes:
    - name: volume
      image:
        reference: quay.io/crio/artifact:v1
        layer_mediaType_filter_IN: ["application/x-mlmodel"] # mount only layer(s) from Artifact which is the ML model, not Data layers, etc.

or like:

  volumes:
    - name: volume
      image:
        reference: quay.io/crio/artifact:v1
        layer_mediaType_filter_IN: ["application/x", "application/y"] # mount only layer(s) marked for X and Y, and for example ignoring layers of type Z if the Artifact provides them

please notice these examples (and naming this layer_mediaType_filter_IN) are in no means representative of the actual Yaml/Spec being suggested, but only to substantiate in a more concrete term an idea which was found meaningful from many within the group.

As one can imagine, if the single OCI Artifact is comprehensive of many assets duly separated in their own semantic layers, but only some layers are functional to runtime scope, this would be a great advantage for the definition of the Deployment workload. wdyt?

I hope this comment is helpful to the wide CNCF community and of interest in the scope of the continued work for this KEP-4639 🚀
cc @bmicklea , @bergwolf , @amisevsk , @chlins , @tarilabs (this), @gorkem , @sabre1041

[sudo-bmitch]

we wanted to raise to this great KEP-4639 efforts,
one suggestion to be evaluated that maybe could also help see motivations for OCI Artifact to be indeed in the longer-term a goal for this KEP-4639 and the Container Runtimes.

One of the great advantage of the OCI Artifact from the OCI spec, is the possibility to define media_type (and/or artifact_type) with a semantic meaning.

When mounting other media types, what are the filesystem settings for the mounted layer? Is something extracted from a tar, requiring all Artifacts to be packaged as a tar? Or is the Artifact layer mounted as a file? How is the filename set, owner, permissions, and other filesystem parameters? From my understanding, the KEP authors were not ready to answer those questions and intentionally marked OCI Artifacts as out of scope for now, leaving individual runtimes free to experiment with their own implementations.

[amisevsk]

From my perspective, tarballs are a natural starting point for packaging OCI artifact layers, and these layers are generally going to be compatible with OCI image layer types (e.g. in testing the existing alpha-level feature, I'm able to mount artifacts by just changing the mediaType on layers and generating a simple image config).

Considering that the current state of the proposal is limited to OCI images, it's not a large leap to extend support for any OCI artifact whose media types are compatible with OCI image layers. The mediatype filtering field would then be an assertion from the user that these mediaTypes can be processed like image layers.

Alternatively, we could consider making this explicit in some way -- runtimes understand certain media types (tar, tar+gzip, etc.); does it make sense to enable defining artifact media types as synonyms for "known" mediatypes?

[gaius-qi]

Alternatively, we could consider making this explicit in some way -- runtimes understand certain media types (tar, tar+gzip, etc.); does it make sense to enable defining artifact media types as synonyms for "known" mediatypes?

I think it's a good idea for container runtimes to understand certain media types (tar, tar+gzip, etc.). As User Stories describes in KEP, we can mount large language model weights, binary, etc. If only image media types are used, it will be confusing to understand what is stored. However, if there are no restrictions on artifact media types, various custom media types need to be handled, which will bring a lot of work.

[dmesser]

I concur. I wonder, though, if there is a discussion already in the runtime space, how to adopt OCI artifacts or if it needs to be kicked off?

[sabre1041]

I concur. I wonder, though, if there is a discussion already in the runtime space, how to adopt OCI artifacts or if it needs to be kicked off?

I have not seen any recent conversations, but it is a topic that would be good to raise and get additional mindshare

[saschagrunert]

As you folks already outlined: runtimes are free to do anything with the feature. I don't think that Kubernetes itself should maintain the list of allowed and supported media types, that would be a better fit for the OCI. But said, runtimes like CRI-O are free to support any format which makes sense.