CRW-579 add three new scripts for adding digests, building map of digests in registries, and introspecting containers (including scratch ones) to extract files

[nickboldt]

CRW-579 add three new scripts for adding digests, building map of digests in registries, and introspecting containers (including scratch ones) to extract files

Change-Id: I2a9c3ef028fd5e094cb247dfc85e2db3895b69a8
Signed-off-by: nickboldt [email protected]

[nickboldt]

To run this:

cd /tmp
git clone [email protected]:eclipse/che-operator.git && cd che-operator
hub checkout https://github.com/eclipse/che-operator/pull/202
cd olm

./addDigests.sh -s eclipse-che-preview-openshift/deploy/olm-catalog/eclipse-che-preview-openshift -n eclipse-che-preview-openshift -v 7.9.0 

Output will be in generated folder.

Console output looks like this:

[INFO] Get images from CSV /tmp/che-operator/olm/generated/eclipse-che-preview-openshift/7.9.0/eclipse-che-preview-openshift.v7.9.0.clusterserviceversion.yaml
[INFO] Extract images from registry quay.io/eclipse/che-plugin-registry:7.9.0 ... found 84
[INFO] Extract images from registry quay.io/eclipse/che-devfile-registry:7.9.0 ... found 15
[INFO] Get digest from quay.io/eclipse/che-operator:7.9.0
[INFO] Get digest from quay.io/eclipse/che-server:7.9.0
[INFO] Get digest from quay.io/eclipse/che-plugin-registry:7.9.0
[INFO] Get digest from quay.io/eclipse/che-devfile-registry:7.9.0
[INFO] Get digest from registry.access.redhat.com/ubi8-minimal:8.0-213
[INFO] Get digest from centos/postgresql-96-centos7:9.6
[WARN] Add 'docker.io/' prefix to centos/postgresql-96-centos7:9.6
[INFO] Get digest from quay.io/eclipse/che-keycloak:7.9.0
[INFO] Get digest from quay.io/eclipse/che-plugin-metadata-broker:v3.1.1
[INFO] Get digest from quay.io/eclipse/che-plugin-artifacts-broker:v3.1.1
[INFO] Get digest from quay.io/eclipse/che-jwtproxy:dbd0578
[INFO] Get digest from docker.io/chinodesuuu/coder:2.1523-vsc1.38.1-che
[INFO] Get digest from docker.io/chinodesuuu/coder:2.1650-vsc1.39.2-che
[INFO] Get digest from docker.io/chinodesuuu/coder:next
...
[INFO] Get digest from quay.io/eclipse/che-machine-exec:7.9.0
[INFO] Get digest from quay.io/eclipse/che-machine-exec:nightly
...
[INFO] Get digest from quay.io/eclipse/che-sidecar-python:3.7.3-8f39348
...
[INFO] Get digest from quay.io/eclipse/che-java11-maven:7.9.0
...
[INFO] Get digest from quay.io/eclipse/che-rust-1.39:7.9.0

[nickboldt]

Comparing the generated file (left) with the source file (right):

Now adding the relatedImages:

[nickboldt]

Note also since digests are not yet supported in Che I'm leaving the script to create content in generated/ folder instead of overwriting the actual CSV.

[nickboldt]

See also https://github.com/redhat-developer/codeready-workspaces-operator/pull/16 for downstream WIP

[tolusha]

@nickboldt
Thank you. You did my job. I was going to work on this in the current sprint)

[nickboldt]

You'll have to thank @davidfestal ... He wrote the first iteration last week

[nickboldt]

There are lots of bugs in CRW 2.1 you could investigate instead... ;)

[tolusha]

@nickboldt
It doesn't work for me

./addDigests.sh -s eclipse-che-preview-openshift/deploy/olm-catalog/eclipse-che-preview-openshift -n eclipse-che-preview-openshift -v 7.9.0 rm -rf generated 
[INFO] Get images from CSV /tmp/che-operator/olm/generated/eclipse-che-preview-openshift/7.9.0/eclipse-che-preview-openshift.v7.9.0.clusterserviceversion.yaml
>>>>>>>> start
[INFO] Pulling quay.io/eclipse/che-plugin-registry:7.9.0 ...
Trying to pull quay.io/eclipse/che-plugin-registry:7.9.0...
Getting image source signatures
Copying blob 30230d5fd633 done  
Copying blob 8eb9b1e99c64 done  
Copying blob e6a50b627bcb done  
Copying blob d67c4dfe3627 done  
Copying blob 1d9c8592f4f9 done  
Copying blob 50982eb4e7ff done  
Copying blob 58eb5d519e3a done  
Copying blob 8e15eac29301 done  
Copying blob 22d7ee08c96d done  
Copying blob 7cfff95d9858 done  
Copying blob 55d2b0950467 done  
Copying config ffc1dfd9d2 done  
Writing manifest to image destination
Storing signatures
ERRO[0024] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:54 for /run/lock/lockdev): lchown /run/lock/lockdev: invalid argument 
  ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:54 for /run/lock/lockdev): lchown /run/lock/lockdev: invalid argument
Error: error pulling image "quay.io/eclipse/che-plugin-registry:7.9.0": unable to pull quay.io/eclipse/che-plugin-registry:7.9.0: unable to pull image: Error committing the finished image: error adding layer with blob "sha256:e6a50b627bcb03d96996bb8e836ecb178eae7425636e3424d9e8d33a918768dd": ApplyLayer exit status 1 stdout:  stderr: there might not be enough IDs available in the namespace (requested 0:54 for /run/lock/lockdev): lchown /run/lock/lockdev: invalid argument

[tolusha]

And in case of docker

[INFO] Pulling quay.io/eclipse/che-plugin-registry:7.9.0 ...
Error: No such container: quay.io-eclipse-che-plugin-registry-7.9.0-1584608804
[INFO] Extract from container ...
tar: Pattern matching characters used in file names
tar: Use --wildcards to enable pattern matching, or --no-wildcards to suppress this warning
tar: var/www/html/*/external_images.txt: Not found in archive
tar: Exiting with failure status due to previous errors

[tolusha]

Updated scripts