Failed to pull OCI bundle from private registry

[iainsproat]

Expected Behavior

When using OCI bundles feature, I expect a Task Run with an OCI bundle Task reference will use the Docker Config Kubernetes secret (referenced in the Service Account) to access a private registry.

Actual Behavior

I receive a 401 error.

status:
  completionTime: "2023-03-15T17:44:43Z"
  conditions:
  - lastTransitionTime: "2023-03-15T17:44:43Z"
    message: 'failed to get task: error requesting remote resource: error getting
      "bundleresolver" "test-pipeline/bundles-d672fb3f2ec09ff198790535e8d424d8": cannot
      retrieve the oci image: GET https://api.digitalocean.com/v2/registry/auth?scope=repository%!A(MISSING)my-organisation%!F(MISSING)my-repository%!A(MISSING)pull&service=registry.digitalocean.com:
      unexpected status code 401 Unauthorized: {"id": "Unauthorized", "message": "Unable
      to authenticate you" }'
    reason: TaskRunResolutionFailed
    status: "False"
    type: Succeeded

Steps to Reproduce the Problem

Apply the following:

kind: Namespace
apiVersion: v1
metadata:
  name: test-pipeline
---
apiVersion: v1
data:
    .dockerconfigjson: yoursecrethereinbase64encoding
kind: Secret
metadata:
    name: docker-config-json
    namespace: test-pipeline
type: kubernetes.io/dockerconfigjson
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: test-pipeline
  namespace: test-pipeline
secrets: []
imagePullSecrets:
  - name: docker-config-json

kubectl create a TaskRun (file adapted from example):

apiVersion: tekton.dev/v1
kind: TaskRun
metadata:
  generateName: test-taskrun-
  namespace: test-pipeline
spec:
  serviceAccountName: test-pipeline
  taskRef:
    kind: Task
    params:
    - name: bundle
      value: private.registry.example.org/repository:tag
    - name: name
      value: yourtaskname
    - name: kind
      value: task
    - name: serviceAccountName
      value: test-pipeline
    resolver: bundles

The above works when the bundle is in a public registry, but not when the bundle is in a private registry.

The secret works as I am able to docker login and docker pull the bundle using the secret.
I am also able to run TektonTasks with an inline taskSpec. The Task Step referenced an image from the same private registry. Tekton is able to mount the same secret (which has entire registry read scope) and have Kubernetes pull the image to run in the Task Step. This issue therefore differs slightly from previously reported issues #3604 and #6276.

A difficulty I have had in attempting to debug this is that the error message does not indicate which service account, secret name, or namespace, it has attempted to mount secrets from.
As part of this debugging, I tried the above TaskRun with non-existent/invalid service account names. This also results in a 401 error. (unrelated to the core issue, but I would have expected a different error related to the non-existent service account.)

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: v1.26.2
Server Version: v1.25.4

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

v0.45.0

• Feature flags ConfigMap

apiVersion: v1
data:
  await-sidecar-readiness: "true"
  custom-task-version: v1beta1
  disable-affinity-assistant: "false"
  disable-creds-init: "false"
  enable-api-fields: alpha
  enable-provenance-in-status: "false"
  enable-tekton-oci-bundles: "true"
  enforce-nonfalsifiablity: none
  require-git-ssh-secret-known-hosts: "false"
  resource-verification-mode: skip
  running-in-environment-with-injected-sidecars: "true"
  send-cloudevents-for-runs: "false"
kind: ConfigMap
metadata:
  name: feature-flags
  namespace: tekton-pipelines

• Bundle Resolver ConfigMap

apiVersion: v1
data:
  default-kind: task
  default-service-account: test-pipeline
kind: ConfigMap
  name: bundleresolver-config
  namespace: tekton-pipelines-resolvers

• Resolver Feature Flags ConfigMap

apiVersion: v1
data:
  enable-bundles-resolver: "true"
  enable-cluster-resolver: "true"
  enable-git-resolver: "true"
  enable-hub-resolver: "true"
kind: ConfigMap
metadata:
  name: resolvers-feature-flags
  namespace: tekton-pipelines-resolvers

[liangyuanpeng]

Failed to pull OCI bundle from private registry

Same problem.

I follow the doc and tekton taskrun is not working for me .
https://tekton.dev/docs/pipelines/bundle-resolver/#parameters

My yamls:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton
imagePullSecrets:
  - name: tekton
---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: hello-task-run-private-bundle
spec:
  serviceAccountName: tekton
  taskRef:
    resolver: bundles
    params:
      - name: bundle
        value: ghcr.io/liangyuanpeng/tekton:task-hello
      - name: name
        value: hello
      - name: kind
        value: task
      - name: serviceAccount
        value: tekton

This is the error log:

           * failed to get task: resource request in progress
  Warning  Failed         14s  TaskRun  failed to get task: error requesting remote resource: error getting "bundleresolver" "default/bundles-95376cec82c97b72a22333a02a4b76de": cannot retrieve the oci image: GET https://ghcr.io/token?scope=repository%!A(MISSING)liangyuanpeng%!F(MISSING)tekton%!A(MISSING)pull&service=ghcr.io: UNAUTHORIZED: authentication required
  Warning  InternalError  14s  TaskRun  1 error occurred:
           * failed to get task: error requesting remote resource: error getting "bundleresolver" "default/bundles-95376cec82c97b72a22333a02a4b76de": cannot retrieve the oci image: GET https://ghcr.io/token?scope=repository%!A(MISSING)liangyuanpeng%!F(MISSING)tekton%!A(MISSING)pull&service=ghcr.io: UNAUTHORIZED: authentication required

Tekton Pipeline version:

0.44.0

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[iainsproat]

/remove-lifecycle stale this is still a problem

[vdemeester]

So looking at https://tekton.dev/docs/pipelines/auth/ a bit, Tekton usually looks at attached secrets from the ServiceAccount, and not necessarily the imagePullSecret.

Does it work with the following ?

secrets:
- name: tekton
imagePullSecrets:
- name: tekton

[AshleyDumaine]

I'm also experiencing the same issue with the v0.50.0 release, even with the secrets / imagePullSecrets specified on the ServiceAccount

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.