When DevWorkspace is enabled zip projects download fail because the devfile regitry certificate is untrusted

[l0rd]

Is your enhancement related to a problem? Please describe

I am using a devfile that has a zip project:

projects:
  - name: spring-petclinic
    zip:
      location: https://192.168.64.10.nip.io/devfile-registry/resources/java-web-spring-java-web-spring-4953f87917b449a404a7f1f4e2457836b9eafbbc.zip

But when I start the workspace the project doesn't appear in Theia.

I can see a file /projects/project-clone-errors.log that contains a certificate error:

2021/10/29 10:02:08 Read DevWorkspace at /devworkspace-metadata/flattened.devworkspace.yaml
2021/10/29 10:02:08 Processing project spring-petclinic
2021/10/29 10:02:08 Downloading project archive from https://192.168.64.10.nip.io/devfile-registry/resources/java-web-spring-java-web-spring-4953f87917b449a404a7f1f4e2457836b9eafbbc.zip
2021/10/29 10:02:08 Encountered error while setting up project spring-petclinic: failed to download archive: Get "https://192.168.64.10.nip.io/devfile-registry/resources/java-web-spring-java-web-spring-4953f87917b449a404a7f1f4e2457836b9eafbbc.zip": x509: certificate signed by unknown authority

In the workpace namespace there is a secret with the tls.crt and tls.key for 192.168.64.10.nip.io that is named workspace763675abe4674548-endpoints. But this secret is not mounted in the workspace (it hasn't the annotations to be automounted by the devworkspce controller):

{
  "creationTimestamp": "2021-10-29T10:02:00Z",
  "name": "workspace763675abe4674548-endpoints",
  "namespace": "admin-che",
  "ownerReferences": [
    {
      "apiVersion": "controller.devfile.io/v1alpha1",
      "blockOwnerDeletion": true,
      "controller": true,
      "kind": "DevWorkspaceRouting",
      "name": "routing-workspace763675abe4674548",
      "uid": "3c845aa3-7193-4266-ae74-ce86172cd42d"
    }
  ],
  "resourceVersion": "14318",
  "uid": "6108ac05-b712-44d5-868d-67269469dc27"
}

Describe the solution you'd like

The secret should contain the right annotations and labels to be mounted in containers and used by curl and theia.

Describe alternatives you've considered

No response

Additional context

I have tried to edit the secret to have it automounted

  annotations:
    controller.devfile.io/mount-as: file
    controller.devfile.io/mount-path: /public-certs
  labels:
    controller.devfile.io/mount-to-devworkspace: "true"

As a result the secret get mounted and:

• curl still fails to download the zip file (I have the same error as above)
• if I refresh Theia downloads the zip and load the project successfully

[metlos]

Note that the namespace syncing code also creates a configmap suffixed with -trusted-ca-certs that wants to mount files to the same directory in the workspace pod. Not sure DWO handles that? @amisevsk could you please confirm?

[metlos]

@l0rd the <wsid>-endpoints secret is used for setting up TLS on the endpoints of the workspace - e.g. it is referenced in the configuration of the ingress(es) that expose workspace endpoints. Not sure it should be known inside the workspace pod.

The namespace syncing code mounts the contents of the the ca-certs-merged configmap from the che-cluster namespace into /public-certs of the workspace pod. If the devfile registry cert is not in that configmap, then the workspace won't know about it.

We need to look into how ca-certs-merged is composed. That codebase is "inherited" from che workspaces, so I am not that familiar with it.

[amisevsk]

Note that the namespace syncing code also creates a configmap suffixed with -trusted-ca-certs that wants to mount files to the same directory in the workspace pod. Not sure DWO handles that?

Currently, DWO creates volume mounts for secrets/configmaps, but the limitations are similar to k8s itself so it will result in a failed DevWorkspace (error message similar to auto-mounted volumes from secret 'test-secret' and configmap 'test-cm' have the same mount path).

For git credentials, we do some merging into one secret, but the general case is hard to manage as we'd need to create a new secret/configmap to hold both pieces of data. I've created devfile/devworkspace-operator#665 for some follow-up here.

In general, we should figure out a more robust way to manage certificates though. The project-clone container isn't picking up the secret for grabbing the project as it's not trusted on the OS-level (curl doesn't know to look in /public-certs and it's not a standard as far as I can tell)

[l0rd]

@metlos oh ok, yes the config map with the certs to trust is even better. I was using the information here #19318 (comment) to understand what and where should be mounted in a workspace pod.

curl doesn't know to look in /public-certs and it's not a standard as far as I can tell

@amisevsk I don't know if a standard exist

[l0rd]

The solution proposed yesterday by @benoitf: use Kubernetes internal service URL rather than route/ingress URL. That should work out of the box.

Anyway this issue depends on #20720. If we decide to use a git server we won't need zip files anymore.

[benoitf]

closing as devfile registry is making links to the internal links (no https, no proxy, no certificates)