podman-4.6.x - image tagging regression #19594

srcshelton · 2023-08-11T11:11:08Z

Issue Description

In podman-4.6.x I'm getting the error below… but only if the image docker.io/gentoo/stage3:amd64-nomultilib-openrc already exists - if I delete the image first, then the sequence completes successfully:

+ ref=sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a
+ echo -e 'INFO:  Using '\''docker.io/gentoo/stage3'\'' pin '\''sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a'\'''
INFO:  Using 'docker.io/gentoo/stage3' pin 'sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a'
+ podman image pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a
Trying to pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a...
Getting image source signatures
Copying blob e59faa1f516b skipped: already exists  
Copying config 2bef20376e done  
Writing manifest to image destination
2bef20376ec5fb38c225c5bc8c98caa5109dd3824c696b117dfe4fe64de773c4
+ podman image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc
+ echo -e 'INFO:  Tagging '\''sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a'\'' as '\''docker.io/gentoo/stage3:amd64-nomultilib-openrc'\'''
INFO:  Tagging 'sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a' as 'docker.io/gentoo/stage3:amd64-nomultilib-openrc'
+ podman --debug image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called untag.PersistentPreRunE(podman --debug image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc) 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Using graph driver btrfs                     
DEBU[0000] Using graph root /mnt/podman-storage         
DEBU[0000] Using run root /var/run/podman               
DEBU[0000] Using static dir /space/podman/static        
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /space/podman/volumes      
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "btrfs" 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Looking up image "docker.io/gentoo/stage3:amd64-nomultilib-openrc" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/gentoo/stage3:amd64-nomultilib-openrc" ... 
DEBU[0000] reference "[btrfs@/mnt/podman-storage+/var/run/podman]docker.io/gentoo/stage3:amd64-nomultilib-openrc" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/gentoo/stage3:amd64-nomultilib-openrc" ... 
DEBU[0000] reference "[btrfs@/mnt/podman-storage+/var/run/podman]docker.io/gentoo/stage3:amd64-nomultilib-openrc" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/gentoo/stage3:amd64-nomultilib-openrc" ... 
Error: docker.io/gentoo/stage3:amd64-nomultilib-openrc: image not known
DEBU[0000] Shutting down engines                        
+ podman --debug image tag docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a docker.io/gentoo/stage3:amd64-nomultilib-openrc
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called tag.PersistentPreRunE(podman --debug image tag docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a docker.io/gentoo/stage3:amd64-nomultilib-openrc) 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Using graph driver btrfs                     
DEBU[0000] Using graph root /mnt/podman-storage         
DEBU[0000] Using run root /var/run/podman               
DEBU[0000] Using static dir /space/podman/static        
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /space/podman/volumes      
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "btrfs" 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Looking up image "docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a" ... 
DEBU[0000] reference "[btrfs@/mnt/podman-storage+/var/run/podman]docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a" ... 
DEBU[0000] reference "[btrfs@/mnt/podman-storage+/var/run/podman]docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a" does not resolve to an image ID 
DEBU[0000] Trying "docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a" ... 
Error: docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a: image not known
DEBU[0000] Shutting down engines

Steps to reproduce the issue

# Remove any prior images...
podman image rm docker.io/gentoo/stage3:amd64-nomultilib-openrc docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a || :
podman image prune -f

# Works first time…
#
podman image pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a

# Next command is expected to fail if there isn't already a tagged image...
podman image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc || :

podman image tag docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a docker.io/gentoo/stage3:amd64-nomultilib-openrc

# … but errors when we then run the same sequence again (new behaviour in 4.6.x):
#
podman image pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a
podman image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc || :
podman image tag docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a docker.io/gentoo/stage3:amd64-nomultilib-openrc

Describe the results you received

$ podman image pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a
Trying to pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a...
Getting image source signatures
Copying blob e59faa1f516b skipped: already exists  
Copying config 2bef20376e done  
Writing manifest to image destination
2bef20376ec5fb38c225c5bc8c98caa5109dd3824c696b117dfe4fe64de773c4
$ podman image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc || :
Error: docker.io/gentoo/stage3:amd64-nomultilib-openrc: image not known
$ podman image tag docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a docker.io/gentoo/stage3:amd64-nomultilib-openrc

$ podman image pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a
Trying to pull docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a...
Getting image source signatures
Copying blob e59faa1f516b skipped: already exists  
Copying config 2bef20376e done  
Writing manifest to image destination
2bef20376ec5fb38c225c5bc8c98caa5109dd3824c696b117dfe4fe64de773c4
$ podman image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc || :
$ podman image tag docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a docker.io/gentoo/stage3:amd64-nomultilib-openrc
Error: docker.io/gentoo/stage3@sha256:16438f24e2e49c5bd23e8aa601c7e7df158b1e0a6237d4bff1d4b4646353626a: image not known

Describe the results you expected

With every release of podman from 1.9.3(!), the above sequence successfully tagged the downloaded image.

Has podman untag behaviour changed in 4.6.x so that untagging a tag also affects the underlying SHA256 in a way which it didn't before?

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.1.7
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d'
  cpuUtilization:
    idlePercent: 88.88
    systemPercent: 6.59
    userPercent: 4.53
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: gentoo
    version: "2.13"
  eventLogger: file
  freeLocks: 2024
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.4.2-gentoo
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3152285696
  memTotal: 67266355200
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: app-containers/aardvark-dns-1.7.0
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: app-containers/netavark-1.7.0
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: app-containers/crun-1.8.6
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /var/run/crun
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 42948067328
  swapTotal: 42949652480
  uptime: 236h 3m 26.00s (Approximately 9.83 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
  - public.ecr.aws
  - registry.fedoraproject.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 22
    paused: 0
    running: 20
    stopped: 2
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /mnt/podman-storage
  graphRootAllocated: 2000397795328
  graphRootUsed: 72675807232
  graphStatus:
    Build Version: Btrfs v6.3.3
    Library Version: "102"
  imageCopyTmpDir: /var/tmp/.private/root
  imageStore:
    number: 345
  runRoot: /var/run/podman
  transientStore: false
  volumePath: /space/podman/volumes
version:
  APIVersion: 4.6.1
  Built: 1691749526
  BuiltTime: Fri Aug 11 11:25:26 2023
  GitCommit: f3069b3ff48e30373c33b3f5976f15abf8cfee20
  GoVersion: go1.20.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1



### Podman in a container

No

### Privileged Or Rootless

Privileged

### Upstream Latest Release

Yes

### Additional environment details

Additional environment details

### Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

The text was updated successfully, but these errors were encountered:

vrothberg · 2023-08-11T11:30:56Z

Thanks for reaching out, @srcshelton!

You're observing the changes made in containers/common#1505. In short, Podman now requires the specified digest to match the referenced image. So for something@digest, the something image must exist AND also match digest.

Some background conversation can be found in containers/common#1248. We decided to do that to follow Docker as specifying the digest of an image that either doesn't match or exist can be seen as a security issue (or lead to some).

@srcshelton, is the causing huge pains or your end or is it something easy to fix?

vrothberg · 2023-08-11T11:32:14Z

So in your reproducer, the untag removes the tag (entirely stripping off all names from the image) and that's why the subsequent tag fails.

srcshelton · 2023-08-11T11:35:30Z

So in your reproducer, the untag removes the tag (entirely stripping off all names from the image) and that's why the subsequent tag fails.

I'm coding-up a workaround now which I think should work - just noting this (including all names now being stripped) as a change in behaviour compared to all previous podman releases, which I didn't see mentioned in the release-notes!

vrothberg · 2023-08-11T11:36:03Z

which I didn't see mentioned in the release-notes

Thank you for pointing this out. I will take a look and make sure it's being added.

vrothberg · 2023-08-11T11:39:19Z

@ashley-cui, can you add something similar to the suggested text below to the release notes of 4.6?

When looking up an image by digest, make sure that the entire repository of the specified value is considered. This aligns with Docker's behavior since version v20.10.20. Previously, both the repository and the tag have been ignored and Podman looked for an image with a matching digest. Ignoring the name, repository and tag of the specified value can lead to security issues and is considered harmful.

ashley-cui · 2023-08-11T13:33:58Z

Aready updated on the GitHub page, here's the PR for the regular release notes: #19597

I'll open a PR to update main once that merges.

vrothberg · 2023-08-11T14:00:11Z

Thanks a ton, @ashley-cui !

srcshelton added the kind/bug Categorizes issue or PR as related to a bug. label Aug 11, 2023

ashley-cui mentioned this issue Aug 11, 2023

[CI:DOCS] [4.6] Update Release Notes #19597

Merged

vrothberg closed this as completed Aug 11, 2023

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Nov 10, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman-4.6.x - image tagging regression #19594

podman-4.6.x - image tagging regression #19594

srcshelton commented Aug 11, 2023 •

edited

Loading

vrothberg commented Aug 11, 2023

vrothberg commented Aug 11, 2023

srcshelton commented Aug 11, 2023

vrothberg commented Aug 11, 2023

vrothberg commented Aug 11, 2023 •

edited

Loading

ashley-cui commented Aug 11, 2023 •

edited

Loading

vrothberg commented Aug 11, 2023

podman-4.6.x - image tagging regression #19594

podman-4.6.x - image tagging regression #19594

Comments

srcshelton commented Aug 11, 2023 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

vrothberg commented Aug 11, 2023

vrothberg commented Aug 11, 2023

srcshelton commented Aug 11, 2023

vrothberg commented Aug 11, 2023

vrothberg commented Aug 11, 2023 • edited Loading

ashley-cui commented Aug 11, 2023 • edited Loading

vrothberg commented Aug 11, 2023

srcshelton commented Aug 11, 2023 •

edited

Loading

vrothberg commented Aug 11, 2023 •

edited

Loading

ashley-cui commented Aug 11, 2023 •

edited

Loading