Support namespaced logins for quay.io

[cevich]

Service accounts (a.k.a. robots) in quay.io are forcably namespaced
to the user or orginization under which they are created. Therefore,
it is impossible to use a common login/password to push images for
both skopeo and containers namespaces. Worse, because the
authentication is recorded against quay.io, multiple login sessions
are required.

Fix this by adding a function definition which verifies non-empty
username/password arguments, before logging in. Call this function
as needed from relevant targets, prior to pushing images.

Signed-off-by: Chris Evich [email protected]

[cevich]

@barthy1 PTAL, is this acceptable? And don't feel bad, this is an unfortunate quirk of quay which just occurred to me while trying to add $QUAY_USERNAME into the travis settings.

[barthy1]

Hi @cevich, well, still my bad :( I tested the code with my own quay.io org, and obviously didn't have to use 2 accounts, so missed this problem :( Thank you so much for your help here!

Regarding the proposed code, I tried to do small test with sample Makefile on s390x and amd64 archs

err_if_empty = $(if $(strip $($(1))),$(strip $($(1))),$(error Required $(1) variable is undefined or empty))

define quay_login
        @echo "$(call err_if_empty,$(1))" | \
                docker login quay.io -u "$(call err_if_empty,$(2))" --password-stdin
endef

test:
        $(call quay_login,SKOPEO_QUAY_USERNAME,SKOPEO_QUAY_PASSWORD)

and when I ran it I got an error

# SKOPEO_QUAY_USERNAME=<..> SKOPEO_QUAY_PASSWORD=<..> make test
Error response from daemon: Get https://quay.io/v2/: unauthorized: Invalid Username or Password
Makefile:9: recipe for target 'test' failed
make: *** [test] Error 1

When I replace $(call err_if_empty,$(1)) and $(call err_if_empty,$(2)) to just env variables, same command works.
Did I miss something?

[cevich]

well, still my bad

Nope, this is super-duper easy to overlook...and not at all obvious, even from the docs, until you actually go to create the account and see it forces a user/org. prefix.

and when I ran it I got an error

Assuming you used a valid username/password...yeah that's a problem, thanks for testing. I now have access to create the real robot accounts, so I'll use those with a fix...

[cevich]

...oh, I see the problem. I mixed up the username and password parameters in quay_login definition.

[cevich]

Fixed, and I added the necessary env. vars. for the two new robot accounts.

[cevich]

@barthy1 After fixing my u/p mixup, I tried similar test w/ both accounts, using actual robot account values and get:

$ export CONTAINERS_QUAY_USERNAME= ...etc...
$ make test
Login Succeeded!
Login Succeeded!

[cevich]

@rhatdan @TomSweeneyRedHat PTAL, I think this is ready. Once it merges, I'll activate a travis cron job to run master...say, weely?

(Daily, Monthly are also options)

[rhatdan]

LGTM

[TomSweeneyRedHat]

@cevich I'd not do it weely, weekly maybe.
LGTM

[cevich]

I'm assuming you mean "monthly"

[TomSweeneyRedHat]

No, I'd prefer "Weekly".

[cevich]

Oh okay, no problem. For ref., it's simply a drop-down box in the Travis settings (bottom of the page).

[barthy1]

Just had a look at first successful multi-arch build and push https://travis-ci.org/github/containers/skopeo/builds/744183408 🎆 🎆 🎆

Thank you @cevich for quick fix!
Thank you @rhatdan and @TomSweeneyRedHat for support of this initiative!

[TomSweeneyRedHat]

@barthy1 You're the driver on this one and a big thank you back to you and to @cevich too for the follow up assist.

@barthy1, any chance I could ask you to write a blog post and/or an addition to https://github.com/containers/skopeo/tree/master/contrib/skopeoimage?

[rhatdan]

Can we get this to happen in podman and buildah images?

[barthy1]

@rhatdan

Can we get this to happen in podman and buildah images?

I can work on podman and buildah next month, when I get back access to Travis, as this month I spent already all free resources for my account and cannot run any jobs there :( Or would be happy just see, that somebody from the team will do it :)

@TomSweeneyRedHat you are absolutely right, documentation also should be updated! I will prepare update to https://github.com/containers/skopeo/blob/master/contrib/skopeoimage/README.md, mentioning about multi-arch images availability. If you have a good place to post a blog, I can also do that. Or feel free to do it on your own (however I guess better to cover podman, buildah and skopeo together in the future)

[TomSweeneyRedHat]

@barthy1 Probably the best place to point the log at would be enable sysadmin. Basically, a post introducing the new images and then a bit on how to use them. I think you're right, probably best to do the post after Buildah and Podman are done too. I'm more than happy to help with the blog and to get you in contact with the folks at that site.

[barthy1]

@TomSweeneyRedHat fyi PR to update https://github.com/containers/skopeo/blob/master/contrib/skopeoimage/README.md

[barthy1]

@TomSweeneyRedHat sorry for touching this old PR, but here we discussed buildah/podman multi-arch builds, similar to what was done for skopeo. Is it still something you are still interested in? I have some time and can prepare the code to use Github actions for multi-arch buildah/podman builds pretty fast (I hope).

[rhatdan]

This is definitely something we would want.

[barthy1]

ok, great, then I will work on it.

[cevich]

@barthy1 I just checked, and don't see any non-x86 hardware supported by Github Actions...IIRC, that's specifically why Travis is used here, no? If host architecture doesn't matter, whatever can be done in Github Actions, can be done in Cirrus-CI but with a greater choice of VMs & distro. versions. For the sake of simplicity and ease of future maintenance, would you consider working with me to put this into Cirrus-CI instead?

Unless of course I'm mistaken, and you have some magic way to access non-x86 VMs in github-actions...then please ignore me 😄

[barthy1]

@cevich Github actions has predefined code to use QEMU emulation(via docker buildx) to run multi-arch builds on x86 architecture - https://github.com/docker/build-push-action#multi-platform-image . It's very easy and short way to get multi-arch builds.
It didn't work for skopeo, because build fails in emulation mode (sometimes it happens, emulation is still not perfect for some cases), but so far my tests for buildah/podman + emulation were successful.
I am not familiar with Cirrus, probably it's possible to do the setup similar to Github actions. However I tried to search for "multi-arch" or "buildx" at https://cirrus-ci.org/ and didn't find anything. So looks like we need to reimplement the wheel :)
It's up to you to choose the best CI platform here and I understand your reasons to keep everything in one place. Let me know if you'd like to try with Cirrus-CI, I can provide the steps how to run multi-arch image build on x86 host, having docker installed.

[cevich]

First-off, thanks for the detailed explanations and effort to understand my position. It makes life easier for everybody.

So looks like we need to reimplement the wheel

I understand, and yes, there is no built-in magic-sauce similar to "actions", within cirrus (it's more general-purpose). Though it seems easily possible to pull this off, it would be wheel-re-implementation. At the same time, I (and probably the containers community) would look negatively on relying on Docker long-term. So we may end up implementing this in Cirrus anyway....but that's up to the future.

build fails in emulation mode (sometimes it happens, emulation is still not perfect for some cases), but so far my tests for buildah/podman + emulation were successful.

Interesting...since it sounds like the reasons are unknown, would it make sense to take a similar approach as skopeo (Makefile driven) and just use Travis? This also brings the benefit of native architectures and a more flexible design based on commonly-understood tooling.

---

All that said, I'm okay now with you doing this in github actions + buildx. It sounds like you've already done some work here, and I understand that CI system well enough to offer reviews/feedback. It also sounds like it would bring these images to the community sooner, rather than later. Again, thanks for taking the time to explain your reasoning, I appreciate it. Please let me know what secure variables you need set (e.g. for quay.io push access), I'm happy to create them for you.

[barthy1]

@cevich yeah Travis was the best CI tool in the past for real multi-architecture support. And it would be easy to move the same code from skopeo to other repos.
However at the end of last year I was exhausted using this tool with job start delays and their new limitation to usage for free. + it was mentioned in one of the discussions, that you removed Travis from podman and preferred not to have it again.

Also I truly believe that github actions is the best tool in your case for multi-arch builds in terms of simplicity and usability.

To sum up :) I can use Travis, if you prefer it, or Github Actions. Cirrus-CI is interesting option to learn for me, but I suspect it will take much more time and effort, so not brave enough here :(
Please, let me know, which tool you'd choose - Github Actions or Travis, and I will use it.

[rhatdan]

If this is being done via emulation, we could theoretically also do this in cirrus with buildah. Which would seem to be better.

buildah bud --arch arm64 ....

Should work as long as you have qemu-user-static installed.

If one of you wants to try this out.

Latest podman should also support this.

The issue I have with it is doing this on Fedora, since the dnf -y update takes a huge amount of time in emulation mode.

[barthy1]

I tried buildah bud --arch s390x for buildah upstream with qemu-user-static installed.
The build hangs at first large step and nothing happened during 2 hours.

Tried to run the build via container with quay.io/buildah/stable and build actually finished, but the image was still for amd64 arch, not s390x arch.

I guess my env has some problems. If you have instructions how to setup working env from scratch I can try one more time

[barthy1]

Created separate issue in buildah repo, just to have proper place for this discussion containers/buildah#2958
Meanwhile I will finish with Github action workflow for buildah and let's see.