Unable to pull container images from the OpenShift internal registry

[cgruver]

Describe the bug

Within a running workspace, podman is able to pull images from external registries like quay.io but is unable to pull images from the internal OpenShift Registry.

The internal registry certificate is not trusted, and the workspace user is not authorized.

Che version

7.63@latest

Steps to reproduce

1. Create a workspace with https://github.com/eclipse-che-demo-app/che-demo-app.git

2. Start a terminal

3. Execute:

   podman pull quay.io/sclorg/postgresql-15-c9s:c9s

4. Observe success:

   WARN[0000] "/" is not a shared mount, this could cause issues or missing mounts with rootless containers 
   Trying to pull quay.io/sclorg/postgresql-15-c9s:c9s...
   Getting image source signatures
   Copying blob 204a508c7797 done  
   Copying blob 1a4b50973163 done  
   Copying blob eae77addda82 done  
   Copying config 034374e72d done  
   Writing manifest to image destination
   Storing signatures
   034374e72d2c12dacc9e3557f8752d3407b1b69ca9aed1e7ea709a31922f180c

5. Execute:

   podman pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest

6. Observe error:

   Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest...
   Error: initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift/cli:latest: pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": x509: certificate signed by unknown authority

7. Ignore TLS:

   podman pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest --tls-verify=false

8. Observe authentication error:

   Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest...
   Error: initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift/cli:latest: reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift/cli: unauthorized: authentication required

Expected behavior

Expect podman configuration to be setup for interacting with the internal image registry.

Runtime

OpenShift

Screenshots

No response

Installation method

OperatorHub

Environment

macOS

Eclipse Che Logs

No response

Additional context

OpenShift is OKD 4.12

[l0rd]

@cgruver makes sense. Do you know if the trusted certificates path can be specified in podman configuration?

[tolusha]

We can pull images from the OpenShift registry in the following steps:

1. mkdir $HOME/openshift-registry-certs
2. openssl s_client -showcerts -connect image-registry.openshift-image-registry.svc:5000 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > $HOME/openshift-registry-certs/ca.crt
3. podman login --cert-dir $HOME/openshift-registry-certs/ -u $(oc whoami) -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000
4. podman pull --cert-dir $HOME/openshift-registry-certs/ image-registry.openshift-image-registry.svc:5000/openshift/cli:latest

The OpenShift registry certificate can be put into /etc/docker/certs.d/image-registry.openshift-image-registry.svc:5000/ca.crt to avoid using --cert-dir flag. I believe all those steps can be automated with Dasboard.

[l0rd]

@tolusha this is important information. Isn't the registry certificate already included in the OpenShift trusted certificate bundle?

[tolusha]

It is even better, the certificate is already mounted into a container:

podman login --cert-dir /var/run/secrets/kubernetes.io/serviceaccount -u $(oc whoami) -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000
podman pull --cert-dir /var/run/secrets/kubernetes.io/serviceaccount image-registry.openshift-image-registry.svc:5000/openshift/cli:latest

[l0rd]

Ok so in UDI we could have a simbolic link like the following one to make things work automatically.

ln -s /var/run/secrets/kubernetes.io/serviceaccount/ca.crt  \
      /etc/docker/certs.d/image-registry.openshift-image-registry.svc:5000/ca.crt

[l0rd]

Or having the dashboard doing that when it creates the kubeconfig.

[cgruver]

I vote for having the dashboard handle it. A lot of users will build their own tools image rather than using UDI. So, that would be one less thing to have to build into their custom dev tools images.

[ibuziuk]

@l0rd it looks like the symlink approach not going to work due to lack of permissions, basically, /etc/docker/certs.d/image-registry.openshift-image-registry.svc:5000/ does not exist and we can not create it

[ibuziuk]

we can technically introduce another endpoint on the UD end that will do the podman login to the internal OpenShift registry and call it during the workspace startup

Draft PR - eclipse-che/che-dashboard#785

The problem that I found is that for some reason, login not always persist between terminal sessions e.g. if you login in one terminal window and open another one podman pull would not always work

[l0rd]

The PR has been merged.

[ibuziuk]

I checked on dogfooding and it does not seem to work

Requests are passing though

[ibuziuk]

after some investigation and discussions, the podman login injection is working correctly and the following command is executed during workspace startup:

podman login --cert-dir /var/run/secrets/kubernetes.io/serviceaccount -u $(oc whoami) -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000

However, in order to pull the image from the internal registry one still needs to explicitly provide --cert-dir /var/run/secrets/kubernetes.io/serviceaccount flag to the podman pull command (or ignore tls via --tls-verify=false).

@benoitf do you happen to know if there is a mechanism we can use for configuring podman via file / env var to point to the right certificate that would allow executing podman pull from the terminal without extra flags?

[benoitf]

@ibuziuk use the /etc/containers/certs.d directory ?

[ibuziuk]

@l0rd suggested smth. similar #22140 (comment)
but I don't think we have permission to create the symlink once workspace is started

[benoitf]

$HOME/.config/containers/certs.d is another path if it's easier to get permissions there

[ibuziuk]

@benoitf tried the approach with symlink but it does not seem to work

projects $ mkdir -p $HOME/.config/containers/certs.d/image-registry.openshift-image-registry.svc:5000
projects $ ln -s /var/run/secrets/kubernetes.io/serviceaccount/ca.crt $HOME/.config/containers/certs.d/image-registry.openshift-image-registry.svc:5000/ca.crt
projects $ ls -la $HOME/.config/containers/certs.d/image-registry.openshift-image-registry.svc:5000/ca.crt
lrwxrwxrwx. 1 user root 52 May 16 17:07 /home/user/.config/containers/certs.d/image-registry.openshift-image-registry.svc:5000/ca.crt -> /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
projects $ podman pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest
Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest...
Error: initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift/cli:latest: pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": x509: certificate signed by unknown authority

Checking the file permissions, but https://github.com/containers/image/blob/main/docs/containers-certs.d.5.md#directory-structure does not say anything about that
also bumped into that issue containers/podman#13823

[l0rd]

@ibuziuk you are linking the external routes certs CA (ca.crt) but you need to link the internal services certs CA (service-ca.crt) to make it work.

If I run the following commands on developer sandbox

export CERTS_SRC="/var/run/secrets/kubernetes.io/serviceaccount"
export CERTS_DEST="$HOME/.config/containers/certs.d/image-registry.openshift-image-registry.svc:5000"

mkdir -p ${CERTS_DEST} && \
ln -s ${CERTS_SRC}/service-ca.crt ${CERTS_DEST}/service-ca.crt && \
podman pull image-registry.openshift-image-registry.svc:5000/openshift/cli:latest

I get unauthorized: authentication required (which is normal I guess) rather than x509: certificate signed by unknown authority

In general we should link both certificates.

[ibuziuk]

PR has been sent - eclipse-che/che-dashboard#851

[ibuziuk]

PR has been merged eclipse-che/che-dashboard#851
Closing

@cgruver could you please clarify how you are using the images from the internal registry from the Eclipse Che workspace? Would be really interested in knowing the use-case

[KevinMGranger]

@ibuziuk (I'm not Charro but I'm on a team they've worked with)

We're building containers within Dev Spaces. Unless $HOME is persistent, we'll lose those containers when the workspace restarts-- so we push them to the internal registry.