Action is not able to take image and pull into Podman image storage #37

andmos · 2021-04-21T17:13:09Z

Hi guys 👋

I'm just wrapping up my workflows with the new s2i action supporting multiple tags (redhat-actions/s2i-build#29) the image builds just fine with multiple tags, but when I try to push with the push-to-registry action, the action fails when podman should pull the image from the normal Docker context:

Run redhat-actions/push-to-registry@v2
  with:
    image: domstolene/ip-avgivelser
    tags: 2.0.0-128 253b187
    registry: ghcr.io
    username: domstolene
    password: ***
    tls-verify: true
  env:
    APP_NAME: ip-avgivelser
    JAR_NAME: ip-avgivelser-2.0.0.jar
    VERSION: 2.0.0
    IMAGE_TAG: test
    JAR_TARGET_FOLDER: target/
    BASE_IMAGE: registry.access.redhat.com/fuse7/fuse-java-openshift:1.2
    IMAGE_PUSH_REPOSITORY: ghcr.io
    OUTPUT_IMAGE: domstolene/ip-avgivelser
    IMAGE_SHA_TAG: 253b187
Checking if "domstolene/ip-avgivelser" with tags "2.0.0-128, 253b187" is present in Podman image storage
/usr/bin/podman version
/usr/bin/podman image exists domstolene/ip-avgivelser:2.0.0-128
/usr/bin/podman image exists domstolene/ip-avgivelser:253b187
Checking if "domstolene/ip-avgivelser" with tags "2.0.0-128, 253b187" is present in Docker image storage
/usr/bin/podman pull docker-daemon:domstolene/ip-avgivelser:2.0.0-128
/usr/bin/podman pull docker-daemon:domstolene/ip-avgivelser:253b187
Tags "2.0.0-128, 253b187" of "domstolene/ip-avgivelser" found in Docker image storage
"docker.io/library/domstolene/ip-avgivelser" was found in the Docker image storage, but not in the Podman image storage. The image(s) will be pulled into Podman image storage, pushed, and then removed from the Podman image storage.
Pushing "docker.io/library/domstolene/ip-avgivelser" with tags "2.0.0-128, 253b187" to "ghcr.io" as "domstolene"
/usr/bin/podman push --quiet --digestfile docker.io-library-domstolene-ip-avgivelser-2.0.0-128_digest.txt docker.io/library/domstolene/ip-avgivelser:2.0.0-128 ghcr.io/domstolene/ip-avgivelser:2.0.0-128 --tls-verify=true --creds=domstolene:***
Error: unable to find 'docker.io/library/domstolene/ip-avgivelser:2.0.0-128' in local storage: no such image
Error: Error: podman exited with code 125
Error: unable to find 'docker.io/library/domstolene/ip-avgivelser:2.0.0-128' in local storage: no such image

Removing "docker.io/library/domstolene/ip-avgivelser" from the Podman image storage
/usr/bin/podman rmi docker.io/library/domstolene/ip-avgivelser:2.0.0-128
Error: 1 error occurred:
	* unable to find 'docker.io/library/domstolene/ip-avgivelser:2.0.0-128' in local storage: no such image
(node:1860) UnhandledPromiseRejectionWarning: Error: podman exited with code 1
Error: 1 error occurred:
	* unable to find 'docker.io/library/domstolene/ip-avgivelser:2.0.0-128' in local storage: no such image

As you can see,

Tags "2.0.0-128, 253b187" of "domstolene/ip-avgivelser" found in Docker image storage
"docker.io/library/domstolene/ip-avgivelser" was found in the Docker image storage

But it fails when trying to pull into Podman image storage.

The target registry is ghcr, and I have tried building the image as ghcr/domstolene/ip-avgivelser as well - same result.
What am I not getting here? 😅

The text was updated successfully, but these errors were encountered:

tetchel · 2021-04-21T17:22:50Z

hi, is there a public link to this workflow run so we can look at it and try and reproduce it?

~~Otherwise, can you set the secret ACTIONS_STEP_DEBUG to true (ref) and run it again? I think the error in the podman pull may be getting logged only at the debug level in this catch block~~

edit: I have upgraded that logging to warning level so if you run your workflow again (targeting v2.2 or v2) then you should see a better warning

andmos · 2021-04-21T17:35:06Z

I'll give it a run with ~~v2.2~~ @main. The repo is private unfortunately, så I can't share it. I'll give it a go!

tetchel · 2021-04-21T17:44:40Z

Note the podman pull output is grouped (collapsed) so make sure to expand those arrows to see / copy everything.

andmos · 2021-04-21T17:54:36Z

@tetchel here is the entire build log (with expanded output) in a gist:
https://gist.github.com/andmos/61aa36d22c657995030fa965788b5950

tetchel · 2021-04-21T17:57:32Z

okay, it looks like the assumption about prefixing the image URL with docker.io/library is not correct

andmos · 2021-04-21T18:01:24Z

Yeah, that would make sense for operating with Dockerhub I guess. In my case, the complete repository URL would be ghcr.io/domstolene/ip-avgivelser. I was not sure if I needed the complete address when building (since the destination registry is ghcr.io it should not be needed in my understanding).

divyansh42 · 2021-04-21T19:32:22Z

Hi @andmos I see your inputs are bit different. Can you change these two inputs?

image: ip-avgivelser
registry: ghcr.io/domstolene

instead of

image: domstolene/ip-avgivelser
registry: ghcr.io

divyansh42 · 2021-04-21T19:34:02Z

Input registry should be in the form of ghcr.io/user
Ref: https://github.com/redhat-actions/push-to-registry#action-inputs

andmos · 2021-04-22T06:23:07Z

Yes having ghcr.io/user did the trick! I guess an action point should be some updates in the doc to make it clear. Thanks for the help, guys.

divyansh42 · 2021-04-22T06:25:41Z

Yeah will do that 👍

andmos · 2021-04-27T11:57:16Z

@divyansh42 / @tetchel hm I ran into a new error today with v2.2 and main:

Run redhat-actions/[email protected]
  with:
    image: ip-avgivelser
    tags: 2.0.0-144 ae9a555
    registry: ghcr.io/domstolene
    username: domstolene
    password: ***
    tls-verify: true
  env:
    APP_NAME: ip-avgivelser
    JAR_NAME: ip-avgivelser-2.0.0.jar
    VERSION: 2.0.0
    IMAGE_TAG: test
    JAR_TARGET_FOLDER: target/
    BASE_IMAGE: registry.access.redhat.com/fuse7/fuse-java-openshift:1.2
    IMAGE_PUSH_REPOSITORY: ghcr.io/domstolene
    OUTPUT_IMAGE: ip-avgivelser
    IMAGE_SHA_TAG: ae9a555
Checking if "ip-avgivelser" with tags "2.0.0-144, ae9a555" is present in Podman image storage
/usr/bin/podman version
  /usr/bin/podman version
  Version:      3.1.0
  API Version:  3.1.0
  Go Version:   go1.15.2
  Built:        Thu Jan  1 00:00:00 1970
  OS/Arch:      linux/amd64
/usr/bin/podman image exists ip-avgivelser:2.0.0-144
/usr/bin/podman image exists ip-avgivelser:ae9a555
Checking if "ip-avgivelser" with tags "2.0.0-144, ae9a555" is present in Docker image storage
/usr/bin/podman pull docker-daemon:ip-avgivelser:2.0.0-144
  /usr/bin/podman pull docker-daemon:ip-avgivelser:2.0.0-144
  Getting image source signatures
  Copying blob sha256:fb15b60ae93220334096891a970266fad8dbd924c154310edefdb87e20ccc272
  Copying blob sha256:bb4eeb13935401f15e2901ced5eb675546e17807bb9f2c78b0465cc0c8887819
  Copying blob sha256:4fa07caa9dc14bebc49deaaefc81844c428082cd352bb6eea3244c0ee541c937
  Copying blob sha256:c919f00c19ddd9f4ba2a1cf1d0e26cd1c3b0babc2f85cb4dbb646f01b43dd374
  Copying blob sha256:9bbc657f4fb00461e2754fbfc8e11a15ed24c9055e14902c2a18fbc0989f0e8e
  Copying blob sha256:050c734bd2868bcd3b69ab0ca033aa3bc95a00a4a1e5317e732394e1c36ef59e
  Copying config sha256:e9e448b72167ce56507f65b4d2a75f7bb34debdd50defd08e2cb566de375e22f
  Writing manifest to image destination
  Storing signatures
    Error processing tar file(exit status 1): operation not permitted
  Error: 1 error occurred:
  	* Error committing the finished image: error adding layer with blob "sha256:4fa07caa9dc14bebc49deaaefc81844c428082cd352bb6eea3244c0ee541c937": Error processing tar file(exit status 1): operation not permitted
/usr/bin/podman pull docker-daemon:ip-avgivelser:ae9a555
  /usr/bin/podman pull docker-daemon:ip-avgivelser:ae9a555
  Getting image source signatures
  Copying blob sha256:bb4eeb13935401f15e2901ced5eb675546e17807bb9f2c78b0465cc0c8887819
  Copying blob sha256:fb15b60ae93220334096891a970266fad8dbd924c154310edefdb87e20ccc272
  Copying blob sha256:050c734bd2868bcd3b69ab0ca033aa3bc95a00a4a1e5317e732394e1c36ef59e
  Copying blob sha256:c919f00c19ddd9f4ba2a1cf1d0e26cd1c3b0babc2f85cb4dbb646f01b43dd374
  Copying blob sha256:4fa07caa9dc14bebc49deaaefc81844c428082cd352bb6eea3244c0ee541c937
  Copying blob sha256:9bbc657f4fb00461e2754fbfc8e11a15ed24c9055e14902c2a18fbc0989f0e8e
  Copying config sha256:e9e448b72167ce56507f65b4d2a75f7bb34debdd50defd08e2cb566de375e22f
  Writing manifest to image destination
  Storing signatures
    Error processing tar file(exit status 1): operation not permitted
  Error: 1 error occurred:
  	* Error committing the finished image: error adding layer with blob "sha256:4fa07caa9dc14bebc49deaaefc81844c428082cd352bb6eea3244c0ee541c937": Error processing tar file(exit status 1): operation not permitted
Error: Error: All tags for "ip-avgivelser" were not found in either Podman image storage, or Docker image storage. Tags "2.0.0-144, ae9a555" not found in Podman image storage, and tags "2.0.0-144, ae9a555" not found in Docker image storage.

Any ideas? Worked fine when we closed this issue 🙈

divyansh42 · 2021-04-27T12:14:32Z

Can you paste the complete logs in a github gist?

andmos · 2021-04-27T12:19:07Z

Here you are:
https://gist.github.com/andmos/4c3db2e7455d0a071bd0d9e563d0ad79
If you need some more debugging, I think you should add that to the code - I don't have permission to add the ACTIONS_STEP_DEBUG secret to the repo 😢

andmos · 2021-04-27T12:30:50Z

Here is a gist from 5 days ago when it worked like a charm:
https://gist.github.com/andmos/811586109c8831d191b5db40dcde231b

tetchel · 2021-04-27T12:45:55Z

I am guessing, this is the 20.04 runner? It looks like the podman version has been upgraded to 3.1.0.

andmos · 2021-04-27T12:48:23Z

Correct, Environment: ubuntu-20.04. Any way to pin podman?

tetchel · 2021-04-27T13:10:35Z

Well, our tests succeeded last night on 3.1.0 so now I'm not certain that's the case.

However your issue looks like containers/podman#9936
and this https://bugzilla.redhat.com/show_bug.cgi?id=1943321

which, apparently is fixed in 3.1.2 but in the meantime has this workaround: containers/podman#9834 (comment)

tetchel · 2021-04-27T13:11:59Z

it may also be worth trying 3.1.2 from https://podman.io/getting-started/installation
(see Ubuntu section, noting podman is not in the official repositories until Ubuntu 20.10)

tetchel · 2021-04-27T13:20:32Z

Any way to pin podman?

I asked on your behalf: containers/podman#10148

since we've wanted a way to do this before, too, with buildah.

andmos · 2021-04-27T13:27:07Z

Yeah, sounds like the sensible thing to do. If we want these actions to be truly versioned and reproducible, it should be a way to pin podman 👍

tetchel · 2021-04-27T13:29:03Z

agreed, I wish the virtual environments were a little more flexible in that sense because you can either get no upgrades (18.04) or all of them without warning (20.04).

andmos · 2021-04-27T17:28:11Z

I tried

      - name: Podman workaround
        run: | 
          . /etc/os-release
          echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.10/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
          curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.10/Release.key | sudo apt-key add -
          sudo apt-get update
          sudo apt-get -y upgrade
          sudo apt-get -y install podman

In my Action, failed with a new error:

/usr/bin/podman pull docker-daemon:ip-avgivelser:207ebfe
 /usr/bin/podman pull docker-daemon:ip-avgivelser:207ebfe
 /usr/bin/podman: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/podman)
"ip-avgivelser" was found in the Podman image storage, but not in the Docker image storage. The image(s) will be pushed from Podman image storage.
⏳ Pushing "ip-avgivelser" with tags "2.0.0-149, 207ebfe" to "ghcr.io/domstolene" as "domstolene"
Combining image name "ip-avgivelser" and registry "ghcr.io/domstolene" to form registry path "ghcr.io/domstolene/ip-avgivelser"
/usr/bin/podman push --quiet --digestfile ip-avgivelser-2.0.0-149_digest.txt ip-avgivelser:2.0.0-149 ghcr.io/domstolene/ip-avgivelser:2.0.0-149 --tls-verify=true --creds=domstolene:***
/usr/bin/podman: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/podman)
Error: podman exited with code 1
/usr/bin/podman: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/podman)

This is really not something I would like to think about 😸

Dit not fail on the pull-step tough

tetchel · 2021-04-27T17:30:30Z

It looks like you used the kubic repository for ubuntu 20.10, but you're on 20.04 if you're using the GitHub runner

fwiw, if you don't need "new" versions of any of the other tools, reverting to ubuntu-18.04 runner will lock in the podman version to an older one while we wait for GitHub to upgrade to 3.1.2

andmos · 2021-04-27T17:32:16Z

Ah my bad. Got the versions confused there, let's see if this does the trick.

tetchel · 2021-04-27T17:36:37Z

this workaround may also be easier (though I have not personally confirmed it works) containers/podman#9834 (comment)

andmos · 2021-04-27T17:47:31Z

Can confirm

      - name: Podman workaround
        run: | 
          . /etc/os-release
          echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
          curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/Release.key | sudo apt-key add -
          sudo apt-get update
          sudo apt-get -y upgrade
          sudo apt-get -y install podman

works, installs podman 3.1.2

tetchel · 2021-04-27T17:49:40Z

great. you'll see I linked this issue to a request for github to update the podman version so if/when they get around to that you can remove your workaround.

andmos · 2021-04-27T17:52:08Z

Yes I noticed! Thanks for being on the ball here 👍 🚀

andmos · 2021-06-03T06:18:57Z

Can now confirm that actions/runner-images#3282 is fixed, so we have the needed version of podman installed on Ubuntu 20.04.

andmos added the question Further information is requested label Apr 21, 2021

tetchel assigned divyansh42 Apr 21, 2021

divyansh42 mentioned this issue Apr 22, 2021

Fail the action if input image contains any / in it's name. #38

Closed

divyansh42 closed this as completed Apr 22, 2021

tetchel mentioned this issue Apr 22, 2021

Warn user if input image and registry both has / in it's name #39

Merged

7 tasks

tetchel mentioned this issue Apr 27, 2021

Update podman to 3.1.2 actions/runner-images#3282

Closed

7 tasks

scottyhq mentioned this issue Apr 28, 2021

notes on github action development (pushing images to quay.io) uwhackweek/docker-template#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Action is not able to take image and pull into Podman image storage #37

Action is not able to take image and pull into Podman image storage #37

andmos commented Apr 21, 2021

tetchel commented Apr 21, 2021 •

edited

Loading

andmos commented Apr 21, 2021 •

edited

Loading

tetchel commented Apr 21, 2021

andmos commented Apr 21, 2021

tetchel commented Apr 21, 2021

andmos commented Apr 21, 2021

divyansh42 commented Apr 21, 2021

divyansh42 commented Apr 21, 2021

andmos commented Apr 22, 2021

divyansh42 commented Apr 22, 2021

andmos commented Apr 27, 2021

divyansh42 commented Apr 27, 2021

andmos commented Apr 27, 2021

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021 •

edited

Loading

tetchel commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021 •

edited

Loading

tetchel commented Apr 27, 2021 •

edited

Loading

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021 •

edited

Loading

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021 •

edited

Loading

andmos commented Jun 3, 2021

Action is not able to take image and pull into Podman image storage #37

Action is not able to take image and pull into Podman image storage #37

Comments

andmos commented Apr 21, 2021

tetchel commented Apr 21, 2021 • edited Loading

andmos commented Apr 21, 2021 • edited Loading

tetchel commented Apr 21, 2021

andmos commented Apr 21, 2021

tetchel commented Apr 21, 2021

andmos commented Apr 21, 2021

divyansh42 commented Apr 21, 2021

divyansh42 commented Apr 21, 2021

andmos commented Apr 22, 2021

divyansh42 commented Apr 22, 2021

andmos commented Apr 27, 2021

divyansh42 commented Apr 27, 2021

andmos commented Apr 27, 2021

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021 • edited Loading

tetchel commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021 • edited Loading

tetchel commented Apr 27, 2021 • edited Loading

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021 • edited Loading

andmos commented Apr 27, 2021

tetchel commented Apr 27, 2021

andmos commented Apr 27, 2021 • edited Loading

andmos commented Jun 3, 2021

tetchel commented Apr 21, 2021 •

edited

Loading

andmos commented Apr 21, 2021 •

edited

Loading

tetchel commented Apr 27, 2021 •

edited

Loading

andmos commented Apr 27, 2021 •

edited

Loading

tetchel commented Apr 27, 2021 •

edited

Loading

tetchel commented Apr 27, 2021 •

edited

Loading

andmos commented Apr 27, 2021 •

edited

Loading