add multi-arch build of upstream skopeo image via Travis #1066

barthy1 · 2020-10-07T09:40:12Z

This PR is connected to #1010 to get skopeo multi-arch container images.

To build multi-arch upstream container image Travis is used, as it has native hardware to run the build for many architectures (amd64, s390x, ppc64le). Docker is used as build and manifest tool. quay.io/skopeo/upstream:master, quay.io/skopeo/stable:v1.2.0 and quay.io/containers/skopeo:v1.2.0 are specified as target multi-arch images.
Travis config file has 3 stages:

local-build to do the local binary build and test for linux/amd64 and osx, as it was in
the initial code
image-build-push to build and push images for specific architectures (amd64, s390x, ppc64le)
multi-arch-manifest-push to create and push manifest for multi-arch image - quay.io/skopeo/upstream:master, quay.io/skopeo/stable:v1.2.0 and quay.io/containers/skopeo:v1.2.0

2 last stages are not done for pull request event or for non-master branch.

2 env variables in Travis are expected - QUAY_USERNAME and QUAY_PASSWORD to push the images to quay.io.
As a result multi-arch image for 3 architectures is published.

The code is tested using another quay registry to push https://travis-ci.com/github/barthy1/skopeo/builds/194396214

Signed-off-by: Yulia Gaponenko [email protected]

rhatdan · 2020-10-08T09:20:05Z

@TomSweeneyRedHat @mtrmac @cevich @edsantiago @vrothberg PTAL

If this works, this is great. would love to see this for Buildah and Podman as well.

Would be nice to get something like this to work for the stable branch.

barthy1 · 2020-10-08T09:30:15Z

@rhatdan thanks for the answer. I've created this PR to get understanding if this approach is acceptable. If it's ok, I can do the same for stable branch. Just want to go step by step :)

rhatdan · 2020-10-08T14:23:03Z

@barthy1 I want multi arch images available in quay.io and to use manifests correctly. So I am fine with this. I want to get feedback @mtrmac @vrothberg @TomSweeneyRedHat though.

TomSweeneyRedHat · 2020-10-08T21:44:20Z

Makefile

@@ -82,6 +82,10 @@ ifeq ($(DISABLE_CGO), 1)
 	override BUILDTAGS = exclude_graphdriver_devicemapper exclude_graphdriver_btrfs containers_image_openpgp
 endif

+ARCHES = amd64 s390x ppc64le
+MA_IMAGE := quay.io/skopeo/$(IMAGE_TYPE):ma


IDK, but we may also want to push to quay.io/containers/skopeo:ma

I see. I believe quay.io/containers/skopeo:ma should be based on the stable version, not on the upstream, which is target of this PR for now?
In general I'd suggest to add one more env variable - EXTRA_REPO (EXTRA_REPOS?) and when it exists - retag and push multi-arch image also to the extra quay.io repo(s).

Yes, I'd suggest that the quay.io/containers/skope:ma should be based on the stable version. Currently, quay.io/containers/skopeo:latest and quay.io/skopeo/stable:latest are equivalent.

TomSweeneyRedHat · 2020-10-08T21:47:08Z

I love the concept, but unfortunately, I've been buried in other stuff the past couple of days. I do want to dig into this though.
@edsantiago @cevich if you've a spare cycle or three to eyeball, input would be appreciated.

barthy1 · 2020-10-12T17:02:52Z

@TomSweeneyRedHat I've updated the PR, adding image build for stable version and pushing it to quay.io/containers/skopeo:ma and quay.io/skopeo/stable:ma

barthy1 · 2020-10-26T10:14:58Z

@TomSweeneyRedHat any suggestions for this PR?

TomSweeneyRedHat · 2020-10-27T14:19:20Z

Thanks for the ping @barthy1 . @edsantiago or @cevich, could one of you take a looksee at this please?

edsantiago · 2020-10-27T14:59:11Z

I will not have time to look this week. My recollection from last month is that I had a strong averse reaction to the :ma : I think it is not futureproof. Is there any reason not to just build and push a multiarch skopeo:version?

barthy1 · 2020-10-27T15:10:59Z

@edsantiago thank you for input! I can change tag to any value community prefers to have. ma was chosen mostly to use some free meaningful tag. As I was not sure that it's ok to reuse existing tags.. Which tags you would like to see?
for instance:
quay.io/skopeo/upstream:master for upstream
quay.io/skopeo/stable:v1.2.0 and quay.io/containers/skopeo:v1.2.0 for stable ?

rhatdan · 2020-10-27T19:41:30Z

@barthy1 That works for me.

edsantiago

Thank you for your patience. At first glance it LGTM, but (caveat) this was not a thorough review, and I'm not well versed in Travis anyway.

Did you look for ways to refactor the multiple image-build-push stanzas? Maybe using with_items? I worry that the duplication will become unmaintainable.

And yes, using :master and/or :v-actual-version sounds ideal to me.

edsantiago · 2020-10-27T20:20:00Z

.travis.yml

+      os: osx
+      install:
+        # Ideally, the (brew update) should not be necessary and Travis would have fairly
+        # frequenstly updated OS images; that�~@~Ys not been the case historically.


Typo ("frequently", no "s"). And, weird localization characters. Please make sure your editor is set to UTF-8.

hey, I just copied frequenstly from existing text :) fixed

barthy1 · 2020-10-28T12:00:14Z

@edsantiago

Did you look for ways to refactor the multiple image-build-push stanzas? Maybe using with_items? I worry that the duplication will become unmaintainable.

Iterating via archs is not doable in case when we need to keep order and archs should be applied only to one stage.
I just refactored a little bit the code to have script for image-build-push in one place.

barthy1 · 2020-11-05T11:20:40Z

@edsantiago hey! any time this week to review the PR?

cevich · 2020-11-05T14:03:32Z

Suggestion 1: Since both the skopeo binary build and the container images follow after package updates, it probably makes sense to enable some cron-triggered (in Travis settings) builds. Otherwise the binaries/images could miss out on important dependency security updates if the pace of PR merges (as written) aren't rapid enough.

Suggestion 2: (forgive me if I missed it) It would be helpful to maintainers, if just the "build" steps also ran for PRs (i.e. only skip the push). This way any code/script changes which affect the mechanics will be noticed right away, instead of being delayed to branch-push time.

edsantiago · 2020-11-05T21:04:40Z

Quick note first: the CI failure is not your PR's fault; it is being investigated in #1096.

I've been trying to review this today, but it's a big job for me because I don't know anything about Travis. It's now late and my brain is fried; and I'm still not much closer to understanding the full details of this PR. I'm sorry.

barthy1 · 2020-11-06T18:47:20Z

@cevich thank you for suggestions!

Suggestion 1: Since both the skopeo binary build and the container images follow after package updates, it probably makes sense to enable some cron-triggered (in Travis settings) builds. Otherwise the binaries/images could miss out on important dependency security updates if the pace of PR merges (as written) aren't rapid enough.

Good point. You can enable Travis cron jobs in your Travis project settings - https://docs.travis-ci.com/user/cron-jobs/ Nothing is configured in the code.

Suggestion 2: (forgive me if I missed it) It would be helpful to maintainers, if just the "build" steps also ran for PRs (i.e. only skip the push). This way any code/script changes which affect the mechanics will be noticed right away, instead of being delayed to branch-push time.

That's just brilliant! You are absolute right, sorry that I didn't do that from the beginning. Now I've added image builds (upstream and stable) to PR pipeline. That's you can see already in Travis setup for my PR - https://travis-ci.org/github/containers/skopeo/builds/741935428 . The push part won't be done, as it's still disabled for PR case.

cevich · 2020-11-06T19:37:35Z

You can enable Travis cron jobs in your Travis project settings

Yep, I'm happy to set this up.

That's just brilliant! You are absolute right, sorry that I didn't do that from the beginning.

You wouldn't be saying that if you knew how many years it took me to arrive at that 😞

Presumably w/o much/any effort, this will also be covered by cron-triggered builds. I believe they simply execute as if branch-push happened, so the full build + push workflow should happen.

Sorry my Travis-Foo isn't what it use to be

.travis.yml

mtrmac · 2020-11-07T18:05:59Z

Suggestion 2: (forgive me if I missed it) It would be helpful to maintainers, if just the "build" steps also ran for PRs (i.e. only skip the push). This way any code/script changes which affect the mechanics will be noticed right away, instead of being delayed to branch-push time.

That’s definitely useful, OTOH I’m worried about the update-test cycle getting longer; it’s already too long for much comfort. Is there a way to measure the time spent running the tests but not waiting for Travis to queue the first one?

Travis reports “total time”, which just seems to sump the duration of all tasks, and probably does not account for parallelism; and a “Ran for” which I can’t interpret (it is now shorter than before, at least in one instance?!), so neither seems to be a

mtrmac

I’m afraid I don’t know much about Travis, so mostly focusing on maintainability over time.

.travis.yml

mtrmac · 2020-11-07T18:10:58Z

.travis.yml

+      # Dont's push if it's PR check
+      - if [[ "$TRAVIS_EVENT_TYPE" != "pull_request" ]]; then make push-arch-container ; fi
+      # skopeo stable image build and push
+      - export SOURCE_TYPE=stable; export REPO=skopeo/stable; export TAG=v1.2.0; export EXTRA_REPO=containers/skopeo


The tag value should, if at all possible, be dynamically computed, or it’s going to stay at v1.2.0 forever.

(Alternatively, create a new “what to do on a release” document or script, but there’s high risk it will not be always followed.)

it's a little bit tricky. as we have several architectures (and builds) + multi-arch manifest creation after. In general it's possible for image-build-push step, I mean just build container image and run it to get the version, but not doable for multi-arch-manifest-push as no build is done there and no container image is available to check the version.
I've follow the second option - with “what to do on a release” document, thank you for input

Makefile

.travis.yml

mtrmac · 2020-11-07T18:32:50Z

Makefile

+push-ma-manifest:
+	echo "${QUAY_PASSWORD}" | docker login ${REGISTRY} -u "${QUAY_USERNAME}" --password-stdin
+	sudo chmod 0755 /etc/docker
+	DOCKER_CLI_EXPERIMENTAL="enabled" docker manifest create "${MA_IMAGE}" $(foreach arch,${ARCHES}, ${MA_IMAGE}-${arch})


Podman experts, should we be using Docker for all of this?

docker is default container tool for travis. podman and buildah, as far as I know, are questionable to install with some hacks (especially for non amd64 archs), see https://travis-ci.community/t/podman-libpod-support/6823/6. I can play with it, if it's hard requirement, however would prefer to use docker, as it's more stable and verified approach for Travis

I'd prefer podman, but @barthy1 is right, travis at this point defaults pretty hard to docker from what I've seen. @edsantiago might have other thoughts.

Makefile

.travis.yml

Travis is used, as it has native hardware to run the build for many architectures (amd64, s390x, ppc64le). Docker is used as build and manifest tool. `quay.io/skopeo/upstream:master`, `quay.io/skopeo/stable:v1.2.0` and `quay.io/containers/skopeo:v1.2.0` are specified as target multi-arch upstream image. Travis config file has 3 stages: - local-build to do the local test for linux/amd64 and osx, as it was in the initial code - image-build-push to build and push images for specific architectures (amd64, s390x, ppc64le) - manifest-multiarch-push to create and push manifest for multi-arch image - `quay.io/skopeo/upstream:master`, `quay.io/skopeo/stable:v1.2.0` and `quay.io/containers/skopeo:v1.2.0` last stage amnd image push step are not done for pull request. 2 env variables specified in Travis settings are expected - QUAY_USERNAME and QUAY_PASSWORD to push the images to quay.io. As a result multi-arch images for 3 architectures are published. README about build setup id prepared Signed-off-by: Yulia Gaponenko <[email protected]>

rhatdan · 2020-11-12T11:24:17Z

I am going to merge, and we can look at the Podman versus Docker in a different PR.

barthy1 · 2020-11-12T11:34:11Z

@rhatdan thank you for merging! Don't forget to specify QUAY_USERNAME and QUAY_PASSWORD in Travis settings for this repo -> https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings

rhatdan · 2020-11-12T18:49:26Z

@TomSweeneyRedHat Do you know what these are?

barthy1 · 2020-11-12T19:13:04Z

Just in case - QUAY_USERNAME and QUAY_PASSWORD are used to push container images to quay.io registry. Usually you can get user and password for registry login via quay.io robot account -> https://docs.quay.io/glossary/robot-accounts.html
For skopeo case you need robot account with write permissions to quay.io/skopeo and quay.io/containers

cevich · 2020-11-12T20:15:46Z

Don't forget to specify QUAY_USERNAME and QUAY_PASSWORD

@rhatdan I know what they are, I'll see if I can set them...

cevich · 2020-11-12T20:25:35Z

...rats. Two major problems:

I don't have access to either skopeo or containers, @TomSweeneyRedHat can you invite me?
Quay hard-codes the repository name as a prefix to the robot user name. i.e. It's impossible to use the same account for both skopeo and containers 😞

cevich · 2020-11-12T20:36:37Z

...double checked. Indeed, the robot accounts are all namespaced to the user or orginization account that owns them. There's no way around this 😢 i.e. no matter what "name" I give the robot, quay will create skopeo+name and containers+name 😭

I'll open up a PR to split QUAY_USERNAME and QUAY_PASSWORD across the two quay namespaces. I'm pretty sure that's the only option. Once I get permissions to actually create those robot accounts, I'll setup the env. vars. in travis.

cevich · 2020-11-12T21:07:33Z

Opened #1104

TomSweeneyRedHat · 2020-11-13T01:06:34Z

@cevich I've just added you to quay.io/containers /buildah /podman and /skopeo. Why 2 when I can give you 4?

I'm hoping you can manage this without dropping a password in clear text, the other issues you noted sound "fun".

@barthy1 thanks again and again for putting this all together.

cevich · 2020-11-16T15:57:09Z

Lol @TomSweeneyRedHat okay, why not. Yes, I'm happy to manage this stuff as I've got my fingers in all the automation pies anyway. Yummy.

barthy1 force-pushed the skopeo_upstream branch 2 times, most recently from 729fda6 to 413e163 Compare October 7, 2020 09:45

barthy1 mentioned this pull request Oct 7, 2020

Build quay.io/skopeo/stable as multi-arch image #1010

Closed

TomSweeneyRedHat reviewed Oct 8, 2020

View reviewed changes

barthy1 force-pushed the skopeo_upstream branch from 413e163 to 73072e2 Compare October 12, 2020 16:59

barthy1 mentioned this pull request Oct 19, 2020

s390x related fixes for tekton pipeline tektoncd/pipeline#3064

Closed

16 tasks

barthy1 force-pushed the skopeo_upstream branch from 73072e2 to b7f2d1b Compare October 26, 2020 10:18

edsantiago reviewed Oct 27, 2020

View reviewed changes

barthy1 force-pushed the skopeo_upstream branch 2 times, most recently from 566dcc3 to 83003c7 Compare October 28, 2020 11:52

barthy1 force-pushed the skopeo_upstream branch from 83003c7 to d2a17e3 Compare November 5, 2020 10:30

barthy1 force-pushed the skopeo_upstream branch from d2a17e3 to 0bcb883 Compare November 6, 2020 17:29

cevich reviewed Nov 6, 2020

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

mtrmac reviewed Nov 7, 2020

View reviewed changes

barthy1 force-pushed the skopeo_upstream branch from 0bcb883 to 39f8117 Compare November 9, 2020 19:20

rhatdan merged commit 8d2c20f into containers:master Nov 12, 2020

TomSweeneyRedHat mentioned this pull request Nov 19, 2020

Build multi-arch rootless-cni-infra on GitHub Actions containers/podman#8415

Closed

github-actions bot added the locked - please file new issue/PR label Sep 27, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 27, 2023

add multi-arch build of upstream skopeo image via Travis #1066

add multi-arch build of upstream skopeo image via Travis #1066

Conversation

barthy1 commented Oct 7, 2020 • edited Loading

rhatdan commented Oct 8, 2020

barthy1 commented Oct 8, 2020

rhatdan commented Oct 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomSweeneyRedHat commented Oct 8, 2020

barthy1 commented Oct 12, 2020

barthy1 commented Oct 26, 2020

TomSweeneyRedHat commented Oct 27, 2020

edsantiago commented Oct 27, 2020

barthy1 commented Oct 27, 2020

rhatdan commented Oct 27, 2020

edsantiago left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

barthy1 Oct 28, 2020 • edited Loading

Choose a reason for hiding this comment

barthy1 commented Oct 28, 2020

barthy1 commented Nov 5, 2020

cevich commented Nov 5, 2020

edsantiago commented Nov 5, 2020

barthy1 commented Nov 6, 2020

cevich commented Nov 6, 2020

mtrmac commented Nov 7, 2020

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhatdan commented Nov 12, 2020

barthy1 commented Nov 12, 2020

rhatdan commented Nov 12, 2020

barthy1 commented Nov 12, 2020

cevich commented Nov 12, 2020

cevich commented Nov 12, 2020

cevich commented Nov 12, 2020

cevich commented Nov 12, 2020

TomSweeneyRedHat commented Nov 13, 2020

cevich commented Nov 16, 2020

barthy1 commented Oct 7, 2020 •

edited

Loading

barthy1 Oct 28, 2020 •

edited

Loading