Sign docker images using github attestations #8685

mjpieters · 2024-10-29T20:41:46Z

GitHub attestation signing uses the GitHub action ID token to retrieve an ephemeral code signing certificate from Fulcio, and store the signature in the Rekor transparency log.

Once an image has been successfully signed, you should be able to verify the signature with:

gh attestation verify --owner astral-uv oci://ghcr.io/astral-sh/uv:latest

Closes #8670

mjpieters · 2024-10-29T20:44:34Z

A word of warning: I am not 100% certain I fully understand everything that the build-docker.yml pipeline does. I'd appreciate some feedback on how the digest of the final docker-republish image manifest is shared with the signing step, for example.

zanieb · 2024-10-29T23:20:51Z

cc @samypr100 (as always, no obligation)

.github/workflows/build-docker.yml

samypr100 · 2024-10-30T12:51:05Z

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E

https://github.com/samypr100/uv/actions/runs/11593512252

samypr100 · 2024-10-30T13:09:24Z

@mjpieters

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E

https://github.com/samypr100/uv/actions/runs/11593512252

@mjpieters signed images are here https://github.com/samypr100/uv/pkgs/container/uv in case you'd like to verify the signatures are working as expected

mjpieters · 2024-10-30T13:55:54Z

@mjpieters

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E
https://github.com/samypr100/uv/actions/runs/11593512252

@mjpieters signed images are here https://github.com/samypr100/uv/pkgs/container/uv in case you'd like to verify the signatures are working as expected

Excellent! I was working my way through understanding cargo dist in my own fork to try this out but hadn't gotten as far as actually seeing the workflow complete.

The first thing I notice is that the the .sig files are now obscuring the actual docker images on that page, which is a bummer.

The signatures are valid however!

% cosign verify ghcr.io/samypr100/uv:latest --certificate-identity-regexp='.*' --certificate-oidc-issuer-regexp='.*' --output-file /tmp/samypr100-uv-verify.json

Verification for ghcr.io/samypr100/uv:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

% jq -r '.[].optional.Subject' /tmp/samypr100-uv-verify.json
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test

I'll have to see what can be done about those signature files in the registry.

mjpieters · 2024-10-30T15:42:25Z

So cosign pushes the signature as a new 'container' to the registry, tagged with sha256-$DIGEST.sig, with the signature contained in the manifest (see https://hackmd.io/@maelvls/how-cosign-uses-registries-to-store-signatures for an overview). It's these tags that are shown at the top of the page by ghcr.io, because it only ever shows the latest tags that have been pushed. The GH packages API and UI (which include the ghcr container registry) is very very sparse and doesn't offer any control over this.

I've looked at a number of other projects that use cosign to sign their containers (of which there are a very large number), and all the ones I looked at so far show these tags at the top. E.g. the Dependabot update bundler (a Github project!) has signature tags at the top. You see this on other registries too, e.g. the boxyhq jackson container on the docker hub.

I see but a few options here:

Accept that the signature tags are going to be pushed last and end up on top.
Experiment with the Github packages API to delete and then restore specific tags, to see if that puts them back at the top again.
Tag the containers after signing. Build and push the images, store the digests, sign the digests, then handle the version and 'latest' tags afterwards.
Tell cosign to push signatures to a separate registry. This would complicate verification of the images, I don't quite know how that'd work.
Decline this PR and forgo the security of having signed containers (which would be a huge pity, security is important, especially in supply-chain tools like uv).

zanieb · 2024-10-30T15:53:12Z

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

mjpieters · 2024-10-30T16:30:17Z

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

You push not only a tag, but a manifest as well. You'd have to separate the two steps; push the manifest without tags and then tag the digest of the manifest afterwards. It could be that you can retag the same digest; I'll try this out in my own fork registry.

samypr100 · 2024-10-30T17:25:19Z

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

You push not only a tag, but a manifest as well. You'd have to separate the two steps; push the manifest without tags and then tag the digest of the manifest afterwards. It could be that you can retag the same digest; I'll try this out in my own fork registry.

I think the main issue here is how GH ranks packages and gives no control over the ordering to orgs/users of what's displayed, so we have to rely on these ordering hacks related to publishing.

mjpieters · 2024-10-30T17:30:43Z

I think the main issue here is how GH ranks packages and gives no control over the ordering to orgs/users of what's displayed, so we have to rely on these ordering hacks related to publishing.

and from my experiments it is clear that it is order that the manifest were pushed to the registry that is used to list them. I tried deleting then restoring, and retagging (moving tags to another manifest then back again to the right manifest), and nothing shifts the signature from the top.

So the only thing that would work is to build and not push, sign (and let cosign push the signature manifest), and only then push the actual image manifest.

mjpieters · 2025-01-30T17:08:31Z

.github/workflows/build-docker.yml

+        # PRs from forks don't have access to secrets, disable this step in that case.
+        if: ${{ github.event.pull_request.head.repo.full_name == 'astral-sh/uv' }}


I added this in here so this workflow would at least build the docker image in this PR. The jobs otherwise would just fail with a Password required error.

Note, I think there's 3 other places in this file missing this conditional (if we're in favor of adding it)

I didn't think there was any point as those 3 all are part of jobs that only run on release.

zanieb

Thank you! I might wait for @samypr100 to give it a look too.

.github/workflows/build-docker.yml

samypr100 · 2025-01-31T04:09:20Z

.github/workflows/build-docker.yml

+        # PRs from forks don't have access to secrets, disable this step in that case.
+        if: ${{ github.event.pull_request.head.repo.full_name == 'astral-sh/uv' }}


Note, I think there's 3 other places in this file missing this conditional (if we're in favor of adding it)

samypr100 · 2025-01-31T04:11:02Z

Good news is that this approach does keep the tags working as-is.

I tried it on a test run here https://github.com/samypr100/uv/actions/runs/13066181068

Which did keep the package versions tags intact https://github.com/samypr100/uv/pkgs/container/uv

Verifications also work!

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:latest                          [23:14:17]
Loaded digest sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc for oci://ghcr.io/samypr100/uv:latest
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:0.5.26                          [23:14:23]
Loaded digest sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc for oci://ghcr.io/samypr100/uv:0.5.26
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:0.5                             [23:14:27]
Loaded digest sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc for oci://ghcr.io/samypr100/uv:0.5
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:ebd769519134c730eb1af61ddd06a8f5629f5082d0c2d8f02a6d577fe4b2a9dc was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

$ gh attestation verify --owner samypr100 oci://ghcr.io/samypr100/uv:python3.13-bookworm             [23:14:41]
Loaded digest sha256:5537896c0c8bd2351350d32dea35c590997c418e954fb5b31d6fcbfd96126562 for oci://ghcr.io/samypr100/uv:python3.13-bookworm
Loaded 1 attestation from GitHub API
✓ Verification succeeded!

sha256:5537896c0c8bd2351350d32dea35c590997c418e954fb5b31d6fcbfd96126562 was attested by:
REPO          PREDICATE_TYPE                  WORKFLOW
samypr100/uv  https://slsa.dev/provenance/v1  .github/workflows/build-docker.yml@refs/heads/cosign-docker-images

...

cosign uses the GitHub action ID token to retrieve an ephemeral code signing certificate from Fulcio, and store the signature in the Rekor transparency log.

zanieb · 2025-01-31T15:01:01Z

Would you mind following with a brief guide on how to verify images in https://docs.astral.sh/uv/guides/integration/docker/ ?

mjpieters · 2025-01-31T15:54:45Z

Would you mind following with a brief guide on how to verify images in https://docs.astral.sh/uv/guides/integration/docker/ ?

I'll see what I can do!

mjpieters temporarily deployed to release October 29, 2024 20:41 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from fd48d69 to 5abc4eb Compare October 29, 2024 20:43

mjpieters temporarily deployed to release October 29, 2024 20:43 — with GitHub Actions Inactive

zanieb reviewed Oct 29, 2024

View reviewed changes

.github/workflows/build-docker.yml Outdated Show resolved Hide resolved

zanieb reviewed Oct 29, 2024

View reviewed changes

.github/workflows/build-docker.yml Outdated Show resolved Hide resolved

samypr100 reviewed Oct 30, 2024

View reviewed changes

.github/workflows/build-docker.yml Show resolved Hide resolved

mjpieters force-pushed the cosign-docker-images branch from 5abc4eb to d1a3809 Compare October 30, 2024 10:19

mjpieters temporarily deployed to release October 30, 2024 10:19 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from d1a3809 to 2872b9a Compare October 30, 2024 12:29

mjpieters temporarily deployed to release October 30, 2024 12:29 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from 2872b9a to 9b162d3 Compare October 30, 2024 12:30

mjpieters temporarily deployed to release October 30, 2024 12:31 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from 9b162d3 to 4a278c1 Compare October 30, 2024 13:09

mjpieters temporarily deployed to release October 30, 2024 13:10 — with GitHub Actions Inactive

mjpieters had a problem deploying to release January 30, 2025 16:45 — with GitHub Actions Failure

mjpieters force-pushed the cosign-docker-images branch from 6dceed9 to 995e461 Compare January 30, 2025 16:48

mjpieters temporarily deployed to release January 30, 2025 16:49 — with GitHub Actions Inactive

mjpieters force-pushed the cosign-docker-images branch from 995e461 to 1d03cda Compare January 30, 2025 17:05

mjpieters temporarily deployed to release January 30, 2025 17:06 — with GitHub Actions Inactive

mjpieters commented Jan 30, 2025

View reviewed changes

mjpieters force-pushed the cosign-docker-images branch from 1d03cda to 59afb8e Compare January 30, 2025 17:09

mjpieters temporarily deployed to release January 30, 2025 17:10 — with GitHub Actions Inactive

mjpieters requested review from zanieb and samypr100 January 30, 2025 17:11

zanieb approved these changes Jan 30, 2025

View reviewed changes

mjpieters mentioned this pull request Jan 30, 2025

Use docker cache mounts for apt, pip and cargo #11106

Open

samypr100 reviewed Jan 31, 2025

View reviewed changes

Sign docker images using cosign

04398a4

cosign uses the GitHub action ID token to retrieve an ephemeral code signing certificate from Fulcio, and store the signature in the Rekor transparency log.

mjpieters force-pushed the cosign-docker-images branch from 59afb8e to 04398a4 Compare January 31, 2025 10:07

mjpieters temporarily deployed to release January 31, 2025 10:07 — with GitHub Actions Inactive

mjpieters requested a review from samypr100 January 31, 2025 12:47

zanieb approved these changes Jan 31, 2025

View reviewed changes

zanieb added releases Related to building and distributing release artifacts of uv security labels Jan 31, 2025

zanieb merged commit 47f80a6 into astral-sh:main Jan 31, 2025
96 checks passed

mjpieters changed the title ~~Sign docker images using cosign~~ Sign docker images using github attestations Jan 31, 2025

mjpieters deleted the cosign-docker-images branch January 31, 2025 15:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sign docker images using github attestations #8685

Sign docker images using github attestations #8685

mjpieters commented Oct 29, 2024 •

edited

Loading

mjpieters commented Oct 29, 2024

zanieb commented Oct 29, 2024

samypr100 commented Oct 30, 2024 •

edited

Loading

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters commented Oct 30, 2024 •

edited

Loading

zanieb commented Oct 30, 2024 •

edited

Loading

mjpieters commented Oct 30, 2024

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters Jan 30, 2025 •

edited

Loading

samypr100 Jan 31, 2025

mjpieters Jan 31, 2025

zanieb left a comment

samypr100 Jan 31, 2025

samypr100 commented Jan 31, 2025 •

edited

Loading

zanieb commented Jan 31, 2025

mjpieters commented Jan 31, 2025

		# PRs from forks don't have access to secrets, disable this step in that case.
		if: ${{ github.event.pull_request.head.repo.full_name == 'astral-sh/uv' }}

Sign docker images using github attestations #8685

Sign docker images using github attestations #8685

Conversation

mjpieters commented Oct 29, 2024 • edited Loading

mjpieters commented Oct 29, 2024

zanieb commented Oct 29, 2024

samypr100 commented Oct 30, 2024 • edited Loading

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters commented Oct 30, 2024 • edited Loading

zanieb commented Oct 30, 2024 • edited Loading

mjpieters commented Oct 30, 2024

samypr100 commented Oct 30, 2024

mjpieters commented Oct 30, 2024

mjpieters Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

samypr100 Jan 31, 2025

Choose a reason for hiding this comment

mjpieters Jan 31, 2025

Choose a reason for hiding this comment

zanieb left a comment

Choose a reason for hiding this comment

samypr100 Jan 31, 2025

Choose a reason for hiding this comment

samypr100 commented Jan 31, 2025 • edited Loading

zanieb commented Jan 31, 2025

mjpieters commented Jan 31, 2025

mjpieters commented Oct 29, 2024 •

edited

Loading

samypr100 commented Oct 30, 2024 •

edited

Loading

mjpieters commented Oct 30, 2024 •

edited

Loading

zanieb commented Oct 30, 2024 •

edited

Loading

mjpieters Jan 30, 2025 •

edited

Loading

samypr100 commented Jan 31, 2025 •

edited

Loading