Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added new task export-image-tar #1570

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

Jared-Sprague
Copy link

This pull request is to add a new simple task for exporting images as oci-archive tar file and pushing as an oci artifact. This is needed in some cases where images must be shared without the use of an image registry.

script: |
podman --version

podman pull $(params.input-image)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should this work when the image is multi-arch? With this command, it will only pull the one image matching the system's architecture (i.e. x86). In the future, if we are going to change over to arm node pools by default, this would effectively be a breaking change as it would pull a different image (if that is even possible).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arewm Oh that is a really good point. I could add the an arch parameter to pass through to podman pull --arch ARCH

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@arewm Ok I added an input-arch param to specify a specific arch to pull if it's different from the host arch.

@arewm
Copy link
Member

arewm commented Nov 4, 2024

/ok-to-test

Copy link
Contributor

@chmeliik chmeliik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this PR allow you to do that you can't already achieve with skopeo copy docker://$IMAGE oci-archive:archive.tar?

@Jared-Sprague
Copy link
Author

What does this PR allow you to do that you can't already achieve with skopeo copy docker://$IMAGE oci-archive:archive.tar?

@chmeliik This makes it possible to do this as part of a pipeline task for a release. There are many tasks you can do manually that konflux automates for you. Like I can clone a repo with git clone but there still is a clone-repository task. Our release requirements are such that we need to give customers the ability to download an image tar file directly. Since there is no task currently in Konflux for saving and uploading an image as a tar file. That is what this is for. And just because you can do it manually with skopeo doesn't help with CI/CD.

@chmeliik
Copy link
Contributor

chmeliik commented Nov 4, 2024

@chmeliik This makes it possible to do this as part of a pipeline task for a release. There are many tasks you can do manually that konflux automates for you. Like I can clone a repo with git clone but there still is a clone-repository task. Our release requirements are such that we need to give customers the ability to download an image tar file directly. Since there is no task currently in Konflux for saving and uploading an image as a tar file. That is what this is for. And just because you can do it manually with skopeo doesn't help with CI/CD.

Is the oras-uploaded artifact pullable directly? I thought you pretty much needed oras pull to download something uploaded by oras push (which is why it seems no different from telling your customer to use skopeo)

@arewm
Copy link
Member

arewm commented Nov 4, 2024

@chmeliik , the intention of this task is to be able to create a component image which can be released outside of a container registry. When released, the task wouldn't perform a skopeo copy-type action, but it would need to download the blob (i.e. with oras) and then push it to some content delivery location (i.e. s3 or other depending on implementation). The artifact produced can be handled similar as if the build-vm-image task is used.

@chmeliik
Copy link
Contributor

chmeliik commented Nov 4, 2024

@chmeliik , the intention of this task is to be able to create a component image which can be released outside of a container registry. When released, the task wouldn't perform a skopeo copy-type action, but it would need to download the blob (i.e. with oras) and then push it to some content delivery location (i.e. s3 or other depending on implementation). The artifact produced can be handled similar as if the build-vm-image task is used.

I think I'm still missing the point 😄

If it's about doing some special handling on release, shouldn't the special handling be done by the release pipeline? Why does the build pipeline need to know what the release pipeline is going to want to do?

@arewm
Copy link
Member

arewm commented Nov 4, 2024

At some level, these operations can be done at release time, but that comes down to an implementation detail. I feel like operations that modify artifacts should be done before release time to enable them to be more fully tested. Build-time activities are those like producing container images, binary artifacts, tarfiles, etc. Release-time activities are those like retrieving, signing, pushing, copying content. This separation can also ensure that we have appropriate provenance available for the produced artifacts.

In this case, the release pipeline wouldn't know that it is processing a container image necessarily. It would just be passed a path to retrieve a tarball and a destination for it.

I think that this task could be changed to do something similar to what I am exploring here: https://github.com/konflux-ci/build-definitions/pull/1534/files#diff-ac0937f0ae23d79101fafbbe1d7b9325c299712e27bcb8a781e0ec34ef8a4643R441-R447

These tarballs are tightly coupled with the container image(s) itself. Instead of producing separate artifacts, it would make sense to create artifacts and attach them to the original container using the referrer's API. Based on feedback from the contract team, however, it has been requested to not reuse the Trusted Artifact flow for this and instead to create a separate utility to simplify the artifact pushing. This approach is, however, less supported than it is to generate a tarball on its own. It would require the feature requested in https://issues.redhat.com/browse/KONFLUX-5222.

@chmeliik
Copy link
Contributor

chmeliik commented Nov 4, 2024

At some level, these operations can be done at release time, but that comes down to an implementation detail. I feel like operations that modify artifacts should be done before release time to enable them to be more fully tested. Build-time activities are those like producing container images, binary artifacts, tarfiles, etc. Release-time activities are those like retrieving, signing, pushing, copying content. This separation can also ensure that we have appropriate provenance available for the produced artifacts.

Agreed. And by this definition, this task is a release task. It doesn't produce anything new, it copies the built image into the same registry but in a weird wrapper

Edit: well, it's not quite a release task. The actual release task would skip the step of transforming OCI into OCI-in-OCI and would just push the tarball to CDN

@arewm
Copy link
Member

arewm commented Nov 4, 2024

Right. It could be handled on either side of build or release. I added this topic to the Konflux community architecture call for tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants