CASM-4669 Upgrade Nexus to 3.67.1 to support image signatures #25

mtupitsyn · 2024-04-16T20:34:22Z

Summary and Scope

To support multi-platform container image uploads together with Sigstore attachments, we need to upgrade metal nexus to latest version 3.67.1. This also involves upgrade of cray-nexus-setup to 0.11.0.

Issue Type

Bugfix Pull Request
RFE Pull Request

Prerequisites

I have included documentation in my PR (or it is not required)
I tested this on internal system (if yes, please include results or a description of the test)
I tested this on a vshasta system (if yes, please include results or a description of the test)

Testing

Ran a few iterations of install/upgrade/uninstall RPM package on vShasta PIT node. Ensured that upgraded Nexus service starts up properly.
Ran /srv/cray/metal-provision/scripts/nexus/setup-nexus.sh script to initialize repositories and pour new content in Nexus
Ensured that newly uploaded images have all required components - versions for multiple platforms and sigstore attachments:

On the screenshot above, more then 1 manifest per tag denotes multi-platform images (aka "manifest lists"), and tag named sha256-*.sig denotes Sigstore attachment.

Idempotency

For better idempotency, fixed couple of bugs related to package upgrade flow:

RPM package install does not fail anymore if nexus or cray-nexus-setup images are already loaded into podman
Fixed cleanup in RPM package post-install script to remove nexus and cray-nexus-setup images properly

Risks and Mitigations

This introduces some risk since this change also brings in a newer version of Nexus, but otherwise a new functionality required by CSM 1.6 roadmap can not be achieved.

mtupitsyn · 2024-04-16T20:36:46Z

metal-nexus.spec

@@ -50,13 +50,14 @@ Provides: pit-nexus
 %define sonatype_nexus3_image artifactory.algol60.net/csm-docker/stable/docker.io/sonatype/nexus3:%{sonatype_nexus3_tag}
 %define sonatype_nexus3_file  sonatype-nexus3-%{sonatype_nexus3_tag}.tar

-%define cray_nexus_setup_tag   0.10.1
+%define cray_nexus_setup_tag   0.11.0


Version 0.11.0 of cray-setup-nexus makes requests to /v1 Nexus API where previous /beta was accessed: Cray-HPE/nexus-setup#29. The /beta is not supported anymore by Nexus 3.67.1.

mtupitsyn · 2024-04-16T20:39:14Z

metal-nexus.spec

-skopeo --override-arch amd64 --override-os linux copy docker://%{skopeo_image}           docker-archive:%{skopeo_file}:%{skopeo_image}:%{skopeo_tag}
+skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{sonatype_nexus3_image}  docker-archive:%{sonatype_nexus3_file}:%{sonatype_nexus3_image}
+skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{cray_nexus_setup_image} docker-archive:%{cray_nexus_setup_file}:%{cray_nexus_setup_image}
+skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{skopeo_source_image}    docker-archive:%{skopeo_file}:%{skopeo_image}:%{skopeo_tag}


Still package skopeo as quay.io/skopeo/stable:latest for backwards compatibility, but use different image artifactory.algol60.net/csm-docker/stable/quay.io/skopeo/stable:v1 as a source. Tag v1 ensures backward compatibility between skopeo versions. Also, we ship our image with special configuration option, which enables copying of Sigstore attachments.

mtupitsyn · 2024-04-16T20:40:21Z

metal-nexus.spec

-skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{sonatype_nexus3_image}  docker-archive:%{sonatype_nexus3_file}
-skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{cray_nexus_setup_image} docker-archive:%{cray_nexus_setup_file}
-skopeo --override-arch amd64 --override-os linux copy docker://%{skopeo_image}           docker-archive:%{skopeo_file}:%{skopeo_image}:%{skopeo_tag}
+skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{sonatype_nexus3_image}  docker-archive:%{sonatype_nexus3_file}:%{sonatype_nexus3_image}


Setting image name here allows to avoid searching for image name/tag after podman load. Image already comes properly tagged after podman load.

mtupitsyn · 2024-04-16T20:40:28Z

metal-nexus.spec

-skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{cray_nexus_setup_image} docker-archive:%{cray_nexus_setup_file}
-skopeo --override-arch amd64 --override-os linux copy docker://%{skopeo_image}           docker-archive:%{skopeo_file}:%{skopeo_image}:%{skopeo_tag}
+skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{sonatype_nexus3_image}  docker-archive:%{sonatype_nexus3_file}:%{sonatype_nexus3_image}
+skopeo --override-arch amd64 --override-os linux copy --src-creds=%(echo $ARTIFACTORY_USER:$ARTIFACTORY_TOKEN) docker://%{cray_nexus_setup_image} docker-archive:%{cray_nexus_setup_file}:%{cray_nexus_setup_image}


Setting image name here allows to avoid searching for image name/tag after podman load. Image already comes properly tagged after podman load.

mtupitsyn · 2024-04-16T20:40:55Z

metal-nexus.spec

@@ -121,8 +122,8 @@ rm -f \
 %service_del_postun nexus.service
 podman stop nexus || echo 'No nexus container was running, nothing to stop.'
 podman rm nexus || echo 'No nexus container was created, nothing to delete.'
-podman rmi %{sonatype_nexus3_image}:%{sonatype_nexus3_tag} || echo 'No nexus image was loaded, nothing to remove.'
-podman rmi %{cray_nexus_setup_image}:%{cray_nexus_setup_tag} || echo 'No nexus image was loaded, nothing to remove.'
+podman rmi %{sonatype_nexus3_image} || echo 'No nexus image was loaded, nothing to remove.'


Tag is already part of %{sonatype_nexus3_image} var.

mtupitsyn · 2024-04-16T20:41:09Z

metal-nexus.spec

-podman rmi %{sonatype_nexus3_image}:%{sonatype_nexus3_tag} || echo 'No nexus image was loaded, nothing to remove.'
-podman rmi %{cray_nexus_setup_image}:%{cray_nexus_setup_tag} || echo 'No nexus image was loaded, nothing to remove.'
+podman rmi %{sonatype_nexus3_image} || echo 'No nexus image was loaded, nothing to remove.'
+podman rmi %{cray_nexus_setup_image} || echo 'No nexus image was loaded, nothing to remove.'


Tag is already part of %{scray_nexus_setup_image} var.

mtupitsyn · 2024-04-16T20:42:19Z

systemd/nexus-init.sh

        podman load -i "$NEXUS_IMAGE_PATH" || exit
-        # get the tag
-        NEXUS_IMAGE_ID=$(podman images --noheading --format "{{.Id}}" --filter label="name=Nexus Repository Manager")


This may not work properly, if there are more then 1 nexus images in podman. Anyway it's not needed anymore, because image comes already tagged after change in packaging.

mtupitsyn · 2024-04-16T20:42:41Z

systemd/nexus-init.sh

        podman load -i "$NEXUS_IMAGE_PATH"
-        # get the tag
-        NEXUS_IMAGE_ID=$(podman images --noheading --format "{{.Id}}" --filter label="name=Nexus Repository Manager")


This may not work properly, if there are more then 1 nexus images in podman. Anyway it's not needed anymore, because image comes already tagged after change in packaging.

mtupitsyn · 2024-04-16T20:43:04Z

systemd/nexus-setup.sh

    podman load -i "$NEXUS_SETUP_IMAGE_PATH" || exit
-    # get the image id
-    CRAY_NEXUS_SETUP_ID=$(podman images --noheading --format "{{.Id}}" --filter label="org.label-schema.name=cray-nexus-setup")


This may not work properly, if there are more then 1 cray-nexus-setup images in podman. Anyway it's not needed anymore, because image comes already tagged after change in packaging.

CASM-4669 Upgrade Nexus to 3.67.1 to support image signatures

783f97a

mtupitsyn requested a review from a team as a code owner April 16, 2024 20:34

mtupitsyn requested review from rustydb, manderson-hpe and jpdavis-prof April 16, 2024 20:34

mtupitsyn commented Apr 16, 2024

View reviewed changes

mtupitsyn mentioned this pull request Apr 16, 2024

CASM-4669 Support container image signatures Cray-HPE/metal-provision#664

Merged

4 tasks

rustydb approved these changes Apr 17, 2024

View reviewed changes

jpdavis-prof approved these changes Apr 17, 2024

View reviewed changes

jpdavis-prof merged commit 783f97a into main Apr 17, 2024
2 of 3 checks passed

jpdavis-prof deleted the feature/image-signatures branch April 17, 2024 17:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CASM-4669 Upgrade Nexus to 3.67.1 to support image signatures #25

CASM-4669 Upgrade Nexus to 3.67.1 to support image signatures #25

mtupitsyn commented Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

mtupitsyn Apr 16, 2024

CASM-4669 Upgrade Nexus to 3.67.1 to support image signatures #25

CASM-4669 Upgrade Nexus to 3.67.1 to support image signatures #25

Conversation

mtupitsyn commented Apr 16, 2024

Summary and Scope

Issue Type

Prerequisites

Testing

Idempotency

Risks and Mitigations

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment