TaskRun failing with duplicate unique image found

[Fabian-K]

Expected Behavior

The following PipelineRun should succeed and print the gomplate version:

apiVersion: "tekton.dev/v1beta1"
kind: "PipelineRun"
metadata:
  generateName: "beekeeper-seed-"
spec:
  pipelineSpec:
    tasks:
    - name: "issue-repro"
      taskSpec:
        steps:
        - args:
          - "--version"
          image: "hairyhenderson/gomplate:v3.11.4"
          name: "step"

Actual Behavior

The PipelineRun is failing, the status of the created TaskRun is

status:
  completionTime: "2023-03-01T10:37:36Z"
  conditions:
  - lastTransitionTime: "2023-03-01T10:37:36Z"
    message: 'failed to create task run pod "beekeeper-seed-655hl-issue-repro": translating
      TaskSpec to Pod: duplicate unique image found for platform: unknown/unknown:
      found sha256:5f2142b3bfc923b4f34805f61e73b47b8948c22fe21578f2331ba0d6f006e5bd
      and sha256:156980dbc3ee7a13b555440de275c4f3083c786b04a95f67fec5207a55b191eb.
      Maybe invalid TaskSpec'
    reason: CouldntGetTask
    status: "False"
    type: Succeeded

Steps to Reproduce the Problem

1. deploy tekton-pipelines
2. create the pipeline run
3. check the status of the task run

Additional Info

• Kubernetes version: v1.23.13
• Tekton Pipeline version: v0.45.0

This does not seem to be an issue with 0.45.0, it also exists in 0.42.0. I noticed that hairyhenderson/gomplate:v3.11.4 is failing, however hairyhenderson/gomplate:v3.11.3 is working fine. The difference between the images seems to be that v3.11.4 has "minimal SLSA Provenance attestation" enabled / was published with that. (hairyhenderson/gomplate#1652 for details).

When inspecting the manifest, for the working image (docker buildx imagetools inspect hairyhenderson/gomplate:v3.11.3) it does not include and attestation-manifest.

Name:      docker.io/hairyhenderson/gomplate:v3.11.3
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:ee2e558d353a1d22d526349131dc4515600aba8790ee89d0be109df356881165

Manifests:
  Name:      docker.io/hairyhenderson/gomplate:v3.11.3@sha256:e4c86e4ded335e41bfff3905b375b13080d0c1d423bb2ab28742c91f719669eb
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/hairyhenderson/gomplate:v3.11.3@sha256:f61a1fdc240c68a95438896bf95464d74e69f349f0dde9d4210ab37580f51144
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

  Name:      docker.io/hairyhenderson/gomplate:v3.11.3@sha256:23c46b1a652b0eaeb8024587ca034acccf65d32beb54de354027aebc330de95c
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v6

  Name:      docker.io/hairyhenderson/gomplate:v3.11.3@sha256:b72a66ff98c0692fdc39846996220de8040337c2c552755833cdba1dc5c64e21
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7

  Name:      docker.io/hairyhenderson/gomplate:v3.11.3@sha256:b1c40aaf271ca5d2bd665904e4aed6398e29f772bfe652957688f6fc63e09ab2
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/ppc64le

  Name:      docker.io/hairyhenderson/gomplate:v3.11.3@sha256:972ee17b4d4ee1b0e9eb9583c418f45dc2e2af3395973f84d5c9fee940b791a5
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  windows/amd64

When inspecting the manifest for the affected image (docker buildx imagetools inspect hairyhenderson/gomplate:v3.11.4), attestation-manifests are included and the error message from the TaskRun status also reference those entries.

Name:      docker.io/hairyhenderson/gomplate:v3.11.4
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:418c33d559eca24ee820b416a12e7c2bb2a16f4e89ead548522f055c30a824fd

Manifests:
  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:dde9ec99e3039a93a100131ed107207f0fd10cb4e4a88b1c89953c96cd56a128
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:3c4312a22a827656d61809b1a74d6defe610c21a5f271b0002a5384c4476cde5
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:4698c176750f053ff14a16cffd225fe66628702f1221763adf67efd0569e8a0c
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v6

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:067790a20e78e549b3dbc2e1c9c393edf2f9b0a5d7217d8cd6855790423b5587
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v7

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:8fa4df46d5a9dfe20be1b9de543c50195179e739c8ead15f40e15ece80cc9cdf
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    windows/amd64

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:5f2142b3bfc923b4f34805f61e73b47b8948c22fe21578f2331ba0d6f006e5bd
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:dde9ec99e3039a93a100131ed107207f0fd10cb4e4a88b1c89953c96cd56a128
    vnd.docker.reference.type:   attestation-manifest

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:156980dbc3ee7a13b555440de275c4f3083c786b04a95f67fec5207a55b191eb
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:3c4312a22a827656d61809b1a74d6defe610c21a5f271b0002a5384c4476cde5
    vnd.docker.reference.type:   attestation-manifest

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:bfccb3f95faab4d53c0b80322746485dfc5bef11177df7238e0d78eb7038af58
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:4698c176750f053ff14a16cffd225fe66628702f1221763adf67efd0569e8a0c
    vnd.docker.reference.type:   attestation-manifest

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:8eeebd5c8f5f6285cdfe3c095e6b0cc5dfa85d076a60b86caab178802c295015
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:067790a20e78e549b3dbc2e1c9c393edf2f9b0a5d7217d8cd6855790423b5587
    vnd.docker.reference.type:   attestation-manifest

  Name:        docker.io/hairyhenderson/gomplate:v3.11.4@sha256:e61bd1f154772cbcb7187364f434277348bb319296f4d4de8f6c3a1afd282d6f
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations:
    vnd.docker.reference.digest: sha256:8fa4df46d5a9dfe20be1b9de543c50195179e739c8ead15f40e15ece80cc9cdf
    vnd.docker.reference.type:   attestation-manifest

I have to admit that I don´t know much about attestation-manifest but this seems to be an issue with tekton not (yet) supporting those images and not with the image itself?

Thanks,
Fabian

[chengjoey]

hi @Fabian-K ,This seems to be a problem with the image when building multiple systems with docker buildx. unknown/unknown seems a bit strange. In the process of creating pods, tekton will automatically use the entrypoint of the image itself if cmd and args are not displayed, causing this The root cause of a bug is that there are multiple unknown/unknown platforms and their sha IDs are different from the mirror inspection
I think unknown/unknown can be ignored, because the OS/architecture will not be unknown when entrypoint finally executed

[afrittoli]

The platform is set to unknown in the example from the docker docs as well: https://docs.docker.com/build/attestations/attestation-storage/#examples

This attestation storage format seems to be custom of docker, the attestation is linked back to the platform-specific image via the sha in the vnd.docker.reference.digest annotation.

I suppose ignoring entries with platform "unknown/unknown" would make sense anyways, regardless of the attestation storage format. @tektoncd/core-maintainers wdyt?

[vdemeester]

cc @imjasonh as well
@afrittoli yes, I tend to agree that we could just ignore unknown/unknown when going through the manifests.

[imjasonh]

This sounds right to me.

The combine program in plumbing will also need to handle unknown/unknown, probably by always including it in the combined output, rather than ignoring it.

[vdemeester]

Adding this for context/future : moby/buildkit#3610