arch: arm instead of arch: arm32v7 implies official images need to be referenced by archtecture explicitely

[enedil]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm running Podman on Raspberry Pi:

$ uname -a
Linux raspberrypi 5.4.51-v7l+ containers/podman#1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux

The problem is, that podman info tells that my architecture is

  arch: arm

However official docker.io images have architectures named a bit differently, for instance arm32v7. This is caused by using runtime.GOARCH (libpod/info.go, function hostInfo), which defines the name as arm. In order to pull images fropm docker.io, I need to reference them by hand, for instance FROM arm32v7/fedora.

Steps to reproduce the issue:

1. Get Raspberry Pi and install Raspberry Pi OS.

2. Install podman, as detailed in installation instructions on podman.io.

3. Try building FROM fedora

Describe the results you received:

Error I get:

* "docker.io/library/fedora:latest": Error choosing an image from manifest list docker://fedora:latest: no image found in manifest list for architecture arm, variant "", OS linux

Describe the results you expected:

I expected that my architecture is recognized correctly.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

pi@raspberrypi ~> podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/arm

Output of podman info --debug:

host:
  arch: arm
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 4
  distribution:
    distribution: raspbian
    version: "10"
  eventLogger: journald
  hostname: raspberrypi
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.51-v7l+
  linkmode: dynamic
  memFree: 5561524224
  memTotal: 8258523136
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0~rc6+dfsg1
      commit: 1.0.0~rc6+dfsg1-3
      spec: 1.0.1
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 55832576
  swapTotal: 104853504
  uptime: 6h 52m 41.83s (Approximately 0.25 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/pi/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/pi/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/pi/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/arm
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 2.1.1~2 armhf [residual-config]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

I didn't try, but I inspected latest GitHub source to confirm it.

Additional environment details (AWS, VirtualBox, physical, etc.):

As mentioned, Raspberry Pi 4.

[mheon]

@rhatdan @QiWang19 @mtrmac I'm going to guess the issue here is that we're not properly setting the variant field?

[rhatdan]

@vrothberg PTAL

[mtrmac]

However official docker.io images have architectures named a bit differently, for instance arm32v7.

That’s not what I see:

$ bin/skopeo inspect --raw docker://docker.io/library/fedora:latest | python -mjson.tool
…
            "digest": "sha256:ccf05790983018082b4021cc0f76e22228b794d9b6e624a2a4417a4de4c1fc07",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "arm",
                "os": "linux",
                "variant": "v7"
            },
…

In order to pull images fropm docker.io, I need to reference them by hand, for instance FROM arm32v7/fedora.

That suggests that the images are also available at arm32v7/fedora, but that string does not factor into the code’s decisions for library/fedora.

Try building FROM fedora

(For future reference, in steps to reproduce please provide specific commands and specific files. It’s not trivially clear what this step was.)

* "docker.io/library/fedora:latest": Error choosing an image from manifest list docker://fedora:latest: no image found in manifest list for architecture arm, variant "", OS linux

This suggests that getCPUVariantArm has failed. Is /proc/cpuinfo available in that environment?

---

Some Podman commands have a an --override-variant option, --override-variant=v7 might be a workaround. (Of course this should ideally work automatically.)

[enedil]

Sorry for incomplete information. This is not an issue of library/fedora:

pi@raspberrypi ~/m/z/issue> cat Dockerfile.ubuntu 
FROM ubuntu
pi@raspberrypi ~/m/z/issue> cat Dockerfile.fedora 
FROM fedora
pi@raspberrypi ~/m/z/issue> cat Dockerfile.debian 
FROM debian
pi@raspberrypi ~/m/z/issue> podman build -f Dockerfile.ubuntu
STEP 1: FROM ubuntu
Error: error creating build container: The following failures happened while trying to pull image specified by "ubuntu" based on search registries in /etc/containers/registries.conf:
* "localhost/ubuntu": Error initializing source docker://localhost/ubuntu:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
* "docker.io/library/ubuntu": Error choosing an image from manifest list docker://ubuntu:latest: no image found in manifest list for architecture arm, variant "", OS linux
* "quay.io/ubuntu": Error initializing source docker://quay.io/ubuntu:latest: Error reading manifest latest in quay.io/ubuntu: StatusCode: 404, <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final/...
pi@raspberrypi ~/m/z/issue> podman build -f Dockerfile.debian 
STEP 1: FROM debian
Error: error creating build container: The following failures happened while trying to pull image specified by "debian" based on search registries in /etc/containers/registries.conf:
* "localhost/debian": Error initializing source docker://localhost/debian:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
* "docker.io/library/debian": Error choosing an image from manifest list docker://debian:latest: no image found in manifest list for architecture arm, variant "", OS linux
* "quay.io/debian": Error initializing source docker://quay.io/debian:latest: Error reading manifest latest in quay.io/debian: StatusCode: 404, <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final/...
pi@raspberrypi ~/m/z/issue> podman build -f Dockerfile.fedora 
STEP 1: FROM fedora
Error: error creating build container: The following failures happened while trying to pull image specified by "fedora" based on search registries in /etc/containers/registries.conf:
* "localhost/fedora": Error initializing source docker://localhost/fedora:latest: error pinging docker registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
* "docker.io/library/fedora": Error choosing an image from manifest list docker://fedora:latest: no image found in manifest list for architecture arm, variant "", OS linux
* "quay.io/fedora": Error initializing source docker://quay.io/fedora:latest: Error reading manifest latest in quay.io/fedora: StatusCode: 404, <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final/...

However after executing podman pull fedora by hand (which finishes without errors), the last command is successful too:

pi@raspberrypi ~/m/z/issue> podman pull fedora
Trying to pull docker.io/library/fedora...
Getting image source signatures
Copying blob 86cb080cbf6c [--------------------------------------] 0.0b / 0.0b
Copying config beb015c712 done  
Writing manifest to image destination
Storing signatures
beb015c712ccdedf88a02d1499d6449b1e6f10a192387a436e578c455ca639ab
pi@raspberrypi ~/m/z/issue> podman build -f Dockerfile.fedora 
STEP 1: FROM fedora
STEP 2: COMMIT
--> beb015c712c
beb015c712ccdedf88a02d1499d6449b1e6f10a192387a436e578c455ca639ab

/proc/cpuinfo is available:

processor	: 0
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 108.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

processor	: 1
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 108.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

processor	: 2
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 108.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

processor	: 3
model name	: ARMv7 Processor rev 3 (v7l)
BogoMIPS	: 108.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd08
CPU revision	: 3

Hardware	: BCM2711
Revision	: d03114
Serial		: [redacted, albeit I'm not sure if it's sensitive]
Model		: Raspberry Pi 4 Model B Rev 1.4

If there is something else I could provide you with, just ask.

[vrothberg]

Could be podman-build bug. @nalind @TomSweeneyRedHat PTAL

[jameseck]

I'm experiencing the same problem with the caddy builder image on a rpi4 (running official 32-bit Raspberry OS):

# podman build -t caddy .
STEP 1: FROM docker.io/caddy:builder AS builder
STEP 2: FROM docker.io/caddy:latest
Error: error creating build container: Error choosing an image from manifest list docker://caddy:builder: no image found in manifest list for architecture arm, variant "", OS linux

Like the OP, if I "podman pull docker.io/caddy:builder" manually, the "podman build" command works fine.

Same cpuinfo and podman version as OP.

[vrothberg]

@jameseck, does buildah bud work? Need to revive my rpi for testing soon.

[jameseck]

"buildah bud" gives the same error.

# buildah bud .
STEP 1: FROM docker.io/caddy:builder AS builder
error creating build container: Error choosing an image from manifest list docker://caddy:builder: no image found in manifest list for architecture arm, variant "", OS linux

And just like "podman build", "buildah bud" works if I "podman pull" the image first.

[vrothberg]

Thanks for checking, @jameseck!

Clearly a Buildah issue then. @nalind @rhatdan PTAL

[mtrmac]

Just searching the code without testing the hypothesis,

buildah/pkg/cli/common.go

Line 284 in af10f8c

fs.StringVar(&flags.OverrideArch, "override-arch", runtime.GOARCH, "prefer `ARCH` instead of the architecture of the machine when pulling images")

(and other copies of that code) set the override-flag value with a non-empty default, which causes

buildah/pkg/parse/parse.go

Lines 613 to 615 in af10f8c

	if arch, err := c.Flags().GetString("override-arch"); err == nil {
	ctx.ArchitectureChoice = arch
	}

to set an override, and that triggers the https://github.com/containers/image/blob/d623481380d073ec6ac9a066a33a9da649eef06e/internal/pkg/platform/platform_matcher.go#L140-L149 logic and turns off variant detection.

AFAICS both override-arch and override-os should default to “no override”.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[firesoft-de]

Same problem for me. Trying to get a python image on a rpi3. Arch is correctly set to arm but the variant is emtpy and therefore no matching image can be found

Pulling is possible manually with podman using
podman pull python:3.8.7-slim-buster --override-variant v7

With buildah bud there is no possibilty to override or enforcing the variant.

[rhatdan]

#2868 should help fix this.

[rhatdan]

Do you know if there is a way to figure out what the variant should be? IE Should we have a table that states
if arch=arm and variant=="" then force it to v7 by default?
Similar rules for other Arches?

[mtrmac]

https://github.com/containers/image/blob/master/internal/pkg/platform/platform_matcher.go has some internal tables, but the code should just pull the right image by default, without every caller having to care. Does that fail?

[umohnani8]

@rhatdan did #2868 fix this or is something else needed?

[rhatdan]

@jameseck @firesoft-de @jameseck Could you test against the main branch, I believe this is fixed. Reopen if I am mistaken.