[v20.10] Platform architecture detection is broken #1170

shaderecker · 2020-12-22T02:52:51Z

This is a bug report
This is a feature request
I searched existing issues before opening this one

Expected behavior

Docker correctly identifies and uses the image's platform architecture.
In my example: It should detect and use armhf on the Raspberry Pi.

Actual behavior

Docker complains about the image platform arch.
This happens only since docker-ce version 20.10. Up until 19.03.14 this still worked correctly (or at least there was no complaint/warning).

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested

Steps to reproduce the behavior

For details see here: pi-hole/docker-pi-hole#735
In short:

Install Docker 20.10 on Raspbian Buster
Run a test command docker run -it --rm --entrypoint pihole-FTL pihole/pihole:v5.2.1 -vv
You will see the warning

The image/tag from the test command is a multi-arch image, so normally it should auto-select the armhf arch from that.
I also tried directly addressing this by explicitely using an armhf image (e.g. v5.2.1-armhf-buster), but this still produces the same warning.
Also notice: Even as this really is an armhf image, in Docker Hub it gets displayed as an amd64 image for some reason.

Output of docker version:

i@pihole-2:~ $ docker version
Client: Docker Engine - Community
 Version:           20.10.1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        831ebea
 Built:             Tue Dec 15 04:35:27 2020
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f001486
  Built:            Tue Dec 15 04:33:02 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

pi@pihole-2:~ $ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.0-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.79-v7+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 924.8MiB
 Name: pihole-2
 ID: 3FB3:6WLC:2DV7:QUL5:7MPF:SBBL:B2MT:BJUN:37X2:G4DD:DCBN:NF3R
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details (AWS, VirtualBox, physical, etc.)

Raspberry Pi 3 Model B Plus Rev 1.3
Docker is installed with an Ansible role which follows the official guide: https://github.com/shaderecker/ansible-pihole/blob/master/roles/docker/tasks/main.yaml

The text was updated successfully, but these errors were encountered:

thaJeztah · 2020-12-23T13:37:05Z

Just to double-check; there was no image present yet before the docker run, so it was freshly pulled, correct?

Wondering if this is related to containerd/containerd#4530 - looking at that image; https://hub.docker.com/r/pihole/pihole/tags?page=1&ordering=last_updated&name=v5.2.1 I see it supports linux/amd64, linux/arm/v7, and linux/arm64/v8. If it's indeed related to that pull request, it may be trying to match a linux/arm/v6 (which is not available).

Could you post the output of

uname -a
uname -m
cat /proc/cpuinfo

/cc @tianon

thaJeztah · 2020-12-23T13:57:53Z

As a workaround, you should probably be able to run the image when explicitly passing --platform=linux/arm/v7 on docker run / docker pull.

gerrodlop · 2020-12-23T14:45:28Z

Same problem here:

pi@node03:~ $ docker run -it --rm balenalib/raspberrypi4-64:sid-20200801 /bin/bash                                                        
Unable to find image 'balenalib/raspberrypi4-64:sid-20200801' locally                                                                     
sid-20200801: Pulling from balenalib/raspberrypi4-64
6f4fd5193f3b: Pull complete
6fc3f88940fc: Pull complete
35c55595a169: Pull complete
6a0d4aaa43d8: Pull complete
e09b3e013617: Pull complete
611e6f942dc1: Pull complete
05b547aa64aa: Pull complete
8e1177087eb0: Pull complete
97afb9e2bf6c: Pull complete
0144ae10cee1: Pull complete
e9f1ccb76158: Pull complete
bbcbd264b7cb: Pull complete
a5f6e3179a06: Pull complete
ee061025e236: Pull complete
1047c2becd1a: Pull complete
b27718082afc: Pull complete
Digest: sha256:65ce6d8e66926134dfa0e2acfa47030f8977245b14d1d77c0860423965076292                                                           
Status: Downloaded newer image for balenalib/raspberrypi4-64:sid-20200801                                                                 
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
root@941d8be89e48:/#

hello-world is ok:

pi@node03:~ $ docker run -it --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
256ab8fe8778: Pull complete
Digest: sha256:1a523af650137b8accdaed439c17d684df61ee4d74feac151b5b337bd29e7eec                                                           
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.                                                                

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.                                                                 
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the                                                              
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it                                                            
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:                                                                    
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

thaJeztah · 2020-12-23T14:58:21Z

@gerrodlop looks like different issue; the balenalib/raspberrypi4-64:sid-20200801 is not a multi-arch image; looking at https://hub.docker.com/r/balenalib/raspberrypi4-64/tags?page=1&ordering=last_updated&name=sid-20200801, it's reported as linux/amd64;

I suspect the image is built incorrectly (specifying the wrong architecture)

shaderecker · 2020-12-23T14:58:49Z

@thaJeztah
Yes, I cleared all images before pulling during my testing.

uname -a

pi@pihole-2:~ $ uname -a
Linux pihole-2 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 armv7l GNU/Linux

uname -m

pi@pihole-2:~ $ uname -m
armv7l

cat /proc/cpuinfo

pi@pihole-2:~ $ cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 1
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 2
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

processor	: 3
model name	: ARMv7 Processor rev 4 (v7l)
BogoMIPS	: 38.40
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xd03
CPU revision	: 4

Hardware	: BCM2835
Revision	: a020d3
Serial		: 00000000f17aab07
Model		: Raspberry Pi 3 Model B Plus Rev 1.3

shaderecker · 2020-12-23T15:04:52Z

I just tried with:
pi@pihole-2:~ $ docker pull pihole/pihole:v5.2.1 --platform=linux/arm/v7

But then a docker image inspect pihole/pihole:v5.2.1 still shows:

"Architecture": "amd64",

thaJeztah · 2020-12-23T15:26:37Z

Interesting. That's odd. If I try to do the same for (e.g.) the hello-world image, it correctly pulls the variant I specify, but not for the pihole image

Thanks for the /proc/cpuinfo output #1170 (comment). Wondering if the issue is that the selection tries a case-sensitive match for Cpu architecture (therefore not matching CPU architecture) https://github.com/containerd/containerd/blob/2055e12953bb538228d8d9fe627fa545d7cf82be/platforms/cpuinfo.go#L93

That wouldn't explain why the docker pull with --platform doesn't match though 🤔

thaJeztah · 2020-12-23T15:36:21Z

Hmmm... so if I specify --platform=linux/arm64 I DO get a different image than when I specify --platform=linux/amd64;

docker pull --platform=linux/arm64 pihole/pihole:v5.2.1

[
    {
        "Id": "sha256:ddcd770c66339bae8f2e9f89940b7f1d3300538fff48e5443d1244ba21d63885",
        "RepoTags": [
            "pihole/pihole:v5.2.1"
        ],
        "RepoDigests": [
            "pihole/pihole@sha256:7888e89f4d435673c75896c546fefe3177379c99c8766c89be25a905f8def533"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2020-12-03T02:06:14.680174992Z",
        "Container": "35f3f315a63da09485218661dc7c1e335121dc5b3b10191c3ba2e8f3f5c718ac",
        "ContainerConfig": {
            "Hostname": "35f3f315a63d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ARCH=arm64",
                "UBUNTU_SUITE=buster",
                "DOCKER_REPO=multiarch/debian-debootstrap",
                "PIHOLE_ARCH=arm64",
                "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",
                "PIHOLE_INSTALL=/root/ph_install.sh",
                "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
                "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
                "IPv6=True",
                "S6_LOGGING=0",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "ServerIP=0.0.0.0",
                "FTL_CMD=no-daemon",
                "DNSMASQ_USER=root",
                "VERSION=v5.2.1"
            ],
            "Cmd": [
                "/bin/bash",
                "-c",
                "#(nop) ",
                "SHELL [/bin/bash -c]"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "Image": "sha256:b4459ac3099e2fd70e9c2aa5ef710e853da11624d65f0b353f88da8c39dbd34f",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": null,
            "Labels": {
                "image": "pihole/pihole:v5.2.1_arm64",
                "maintainer": "[email protected]",
                "url": "https://www.github.com/pi-hole/docker-pi-hole"
            },
            "Shell": [
                "/bin/bash",
                "-c"
            ]
        },
        "DockerVersion": "19.03.13+azure",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ARCH=arm64",
                "UBUNTU_SUITE=buster",
                "DOCKER_REPO=multiarch/debian-debootstrap",
                "PIHOLE_ARCH=arm64",
                "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",
                "PIHOLE_INSTALL=/root/ph_install.sh",
                "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
                "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
                "IPv6=True",
                "S6_LOGGING=0",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "ServerIP=0.0.0.0",
                "FTL_CMD=no-daemon",
                "DNSMASQ_USER=root",
                "VERSION=v5.2.1"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "Image": "sha256:b4459ac3099e2fd70e9c2aa5ef710e853da11624d65f0b353f88da8c39dbd34f",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": null,
            "Labels": {
                "image": "pihole/pihole:v5.2.1_arm64",
                "maintainer": "[email protected]",
                "url": "https://www.github.com/pi-hole/docker-pi-hole"
            },
            "Shell": [
                "/bin/bash",
                "-c"
            ]
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 386958209,
        "VirtualSize": 386958209,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/5a3e83f7f79c72d97966cbf65f8667fc9ce69573b782785a65bb60f06fbf9d3e/diff:/var/lib/docker/overlay2/4513f8e22e61cf6940d46663389a823af08d99851a942effb447b94c6823918d/diff:/var/lib/docker/overlay2/71ddf41a3376105bf9cbaa86e15e824d43c690944d9517f7995105088422689e/diff:/var/lib/docker/overlay2/c4f94d77d21d3ba107a1eb733bb3c4dccfe583a8c25c078c712da75efeeec7af/diff:/var/lib/docker/overlay2/e8c92f5e4b9699403271db8bffdc008b02a7bf70fb01361687d8b3b5f8f85226/diff:/var/lib/docker/overlay2/975d9d9b0cd48b73553e80fd5ef1c111bb6614fc4d54ec4bfda90995918fa06c/diff:/var/lib/docker/overlay2/c31a030288e3818d99ba1d8fd75847d2e56c45e33af280d07850760155dfe021/diff",
                "MergedDir": "/var/lib/docker/overlay2/e180a4554865f5a757a84525b9db1c546be58d64ccdeb04ea60dda9a8da3fef5/merged",
                "UpperDir": "/var/lib/docker/overlay2/e180a4554865f5a757a84525b9db1c546be58d64ccdeb04ea60dda9a8da3fef5/diff",
                "WorkDir": "/var/lib/docker/overlay2/e180a4554865f5a757a84525b9db1c546be58d64ccdeb04ea60dda9a8da3fef5/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:cc77256f2d56e07985cba58506e433bb21ef7b3a06bd3a688ac6abfdfc010344",
                "sha256:c2c03ee92cd94dc4721023ebe9e6b1a1c6e3588a470d65bdabc589d68bd38395",
                "sha256:beb456e5587da0c7e993017a51b77dac721a6fc290b0c23b63fda47fa262488c",
                "sha256:038c127208f1ddfa51eab576f6de0f44c4c9991bcdb66e432d654d280db597f6",
                "sha256:7554d616a82bfa378acdc7ae9e5c3e885f29efb6cf06059071ca8648fbc9ec42",
                "sha256:110d37e83a24488a1565c2d8e5307f676a72a19d6779d0f1e97a7203f7aa03e2",
                "sha256:9657d8d1b438d67e784aaf3c96d95c943b9138efbd029453f6e7227d304c2553",
                "sha256:6fa10c09468998df7084f6affeb8d3d2b5e02ce2aaccc02249c48b41ee323392"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

docker pull --platform=linux/amd64 pihole/pihole:v5.2.1

[
    {
        "Id": "sha256:52e72f4b6a617b430cf14c6fa919eb477c48aee4f7f87cc1c55247c1d602a66f",
        "RepoTags": [
            "pihole/pihole:v5.2.1"
        ],
        "RepoDigests": [],
        "Parent": "",
        "Comment": "",
        "Created": "2020-12-03T01:59:33.41222977Z",
        "Container": "6c475f695cacb914ab84e7f07d978bd5180bbb6b5b102c099eebb185f54473e3",
        "ContainerConfig": {
            "Hostname": "6c475f695cac",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PIHOLE_ARCH=amd64",
                "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-amd64.tar.gz",
                "PIHOLE_INSTALL=/root/ph_install.sh",
                "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
                "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
                "IPv6=True",
                "S6_LOGGING=0",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "ServerIP=0.0.0.0",
                "FTL_CMD=no-daemon",
                "DNSMASQ_USER=root",
                "VERSION=v5.2.1"
            ],
            "Cmd": [
                "/bin/bash",
                "-c",
                "#(nop) ",
                "SHELL [/bin/bash -c]"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "Image": "sha256:929daab554acb2a02ab991ec85bba37b6934eaad91adec262b35ab36b151a03c",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": null,
            "Labels": {
                "image": "pihole/pihole:v5.2.1_amd64",
                "maintainer": "[email protected]",
                "url": "https://www.github.com/pi-hole/docker-pi-hole"
            },
            "Shell": [
                "/bin/bash",
                "-c"
            ]
        },
        "DockerVersion": "19.03.13+azure",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "PIHOLE_ARCH=amd64",
                "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-amd64.tar.gz",
                "PIHOLE_INSTALL=/root/ph_install.sh",
                "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
                "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
                "IPv6=True",
                "S6_LOGGING=0",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "ServerIP=0.0.0.0",
                "FTL_CMD=no-daemon",
                "DNSMASQ_USER=root",
                "VERSION=v5.2.1"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "Image": "sha256:929daab554acb2a02ab991ec85bba37b6934eaad91adec262b35ab36b151a03c",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": null,
            "Labels": {
                "image": "pihole/pihole:v5.2.1_amd64",
                "maintainer": "[email protected]",
                "url": "https://www.github.com/pi-hole/docker-pi-hole"
            },
            "Shell": [
                "/bin/bash",
                "-c"
            ]
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 334142811,
        "VirtualSize": 334142811,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/c81092a13f818ccbd34d1dad0fbe0defef4b23cd71571b478dd1b2f8682002c2/diff:/var/lib/docker/overlay2/519ba27e5d31dc80a78b2dd4bb18aa8b0a459628263f5bbdcf358dd130f1c341/diff:/var/lib/docker/overlay2/122d90537c4950d38f96cbc46124301a013d2cf5d8e0b13fae4122b8589ad42a/diff:/var/lib/docker/overlay2/f0d6f913616a133ee8f0c1aba8551bc96414c456b62ae7f14759ae32471e5e10/diff:/var/lib/docker/overlay2/189819e86e683b4eb22fc53b27203935dd5bd89e5441e425f46992308a12a92c/diff:/var/lib/docker/overlay2/3de9aa9bcc4e0d3b932dd87da347058634301022fef4e5adc179440764ebdaea/diff:/var/lib/docker/overlay2/c413fbc60fc3e94ceb18205e325e519cd307680d21a8e31e37424fcca73db35f/diff:/var/lib/docker/overlay2/58416db361d34f20459d4545be1df8a312cf4d2c5a7303f8ff0687214f2171fa/diff:/var/lib/docker/overlay2/c84b697f7135ad21fc9763767e562e6cb152f591cde178c5e8aa7cb627b76840/diff:/var/lib/docker/overlay2/d72c77397d63398400b1f298f957e8f61ba99025bac1bcf2805634debbbc504e/diff:/var/lib/docker/overlay2/7df3bb0df183a896c0a13608577ba295aeb58d35e56beba520a9d2edcbea9ba6/diff:/var/lib/docker/overlay2/0699598278b65549bfc219f04617a9d4c1bd04e053899dcccd1b9454354e01e8/diff",
                "MergedDir": "/var/lib/docker/overlay2/27baecf20f210d74c6b5519b4ff58588030c500a1dc3f982a7554e0f5b308b30/merged",
                "UpperDir": "/var/lib/docker/overlay2/27baecf20f210d74c6b5519b4ff58588030c500a1dc3f982a7554e0f5b308b30/diff",
                "WorkDir": "/var/lib/docker/overlay2/27baecf20f210d74c6b5519b4ff58588030c500a1dc3f982a7554e0f5b308b30/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:f5600c6330da7bb112776ba067a32a9c20842d6ecc8ee3289f1a713b644092f8",
                "sha256:333ec1b36502156108bb75d5372e9bd01660ef212c63c6852afb719f2578a135",
                "sha256:7337aaa591164c259c17bb8de3b91c5aa856aa8b60bd7ed4029fcb340864134c",
                "sha256:2ad2d3172ff73c6009e9386e1a826bee4ebf6b6767bd275657b4269f771c8ff5",
                "sha256:a75feaf7b4ef0e0a2a9f2d1c7ea3737e34e2bfaac268ef854d9c13a4bd69bd95",
                "sha256:40821dc7d051aa9e92ce9de97b50d1ed3590e9ab7f9281ebabb3a2c631057d57",
                "sha256:4e1202651bb25616397108cc90af3940c2fd93b8d292978a6791046c97f347bc",
                "sha256:f7e072f2db4609884ad427f42b4a7ddab30c0ebe208d8e98ce2b38b8efc03f0c",
                "sha256:2a617d1b6a13f220d026a4a509ec0ff5fa536f6899a72f4350f6820ce654ac74",
                "sha256:7554d616a82bfa378acdc7ae9e5c3e885f29efb6cf06059071ca8648fbc9ec42",
                "sha256:c3e81aeb769d23e17e7bc675f016b640bc2a9bb2214cebf089521f554102f367",
                "sha256:9657d8d1b438d67e784aaf3c96d95c943b9138efbd029453f6e7227d304c2553",
                "sha256:6fa10c09468998df7084f6affeb8d3d2b5e02ce2aaccc02249c48b41ee323392"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

Could it be that these images were cross-compiled, and have the wrong architecture set in their metadata?

thaJeztah · 2020-12-23T15:38:14Z

Looks indeed that the image itself is the right variant;

"S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",

"S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-amd64.tar.gz",

thaJeztah · 2020-12-23T15:42:07Z

So.. the manifest list ("multi arch") image has the correct architectures set, but the individual images either don't have the right architecture set, or have no architecture set (and the daemon sets the default linux/amd64 which could be for backward-compat), or something in the daemon code overwrites the architecture 🤔

docker manifest inspect pihole/pihole:v5.2.1

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3035,
         "digest": "sha256:5281797ed544ae438cd3c458171212d1e0cbf5c219a1cffe28b042a9f9821791",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1988,
         "digest": "sha256:d899e498cb7beb5784aef9d62e1bb7f28b4ec3306cf96f4d1bc6cd62e26bf599",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1988,
         "digest": "sha256:b2bdc30dd2024d1770f4e1358ea4fdf507ed9e63ae79d9e6fac9ececa2da00e6",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      }
   ]
}

thaJeztah · 2020-12-23T16:00:43Z

Narrowing down what the cause is, Inspecting the image by pulling it using https://github.com/moby/moby/blob/master/contrib/download-frozen-image-v2.sh

mkdir /arm64
./contrib/download-frozen-image-v2.sh /arm64 pihole/pihole:v5.2.1@sha256:d899e498cb7beb5784aef9d62e1bb7f28b4ec3306cf96f4d1bc6cd62e26bf599
Downloading 'pihole/pihole:v5.2.1@sha256:d899e498cb7beb5784aef9d62e1bb7f28b4ec3306cf96f4d1bc6cd62e26bf599' (8 layers)...
....

Download of images into '/arm64' complete.
Use something like the following to load the result into a Docker daemon:
  tar -cC '/arm64' . | docker load

cat /arm64/manifest.json
[
  {
    "Config": "ddcd770c66339bae8f2e9f89940b7f1d3300538fff48e5443d1244ba21d63885.json",
    "RepoTags": [
      "pihole/pihole:v5.2.1"
    ],
    "Layers": [
      "e1f5c9cf823075c9232453c9d7053867d3cf3f439168f98dd93d05cd21e44811/layer.tar",
      "ae605a65daf61f97f709b3e112a732c75059e575ec134f1dca3cf7575c5410d7/layer.tar",
      "e53dae905ccec45ab20632a393fc05011657c3fc1b2a77ca0bf17d10edcb7f12/layer.tar",
      "6e10c020957b38ca48574a9034a89bedf1b1319288f859eba7a7b927f39bab25/layer.tar",
      "1a19ac15b4e2d81665ae836dd62841f344ec34bae33664f04ff34b2371c1c7e7/layer.tar",
      "04d6b9264033223437559e633a15eabb193d85fb8e354f7ee84002fd0db67b29/layer.tar",
      "dee573760ae01fd1cca624a652d3478d771e6ff383a3f18fa519edc19f716dc6/layer.tar",
      "0fb2bd2b7ffd7c57f4c302f4d959de6edbd58358e6c1fd1c844aace30f26dc41/layer.tar"
    ]
  }
]

cat /arm64/ddcd770c66339bae8f2e9f89940b7f1d3300538fff48e5443d1244ba21d63885.json

{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
      "443/tcp": {},
      "53/tcp": {},
      "53/udp": {},
      "67/udp": {},
      "80/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "ARCH=arm64",
      "UBUNTU_SUITE=buster",
      "DOCKER_REPO=multiarch/debian-debootstrap",
      "PIHOLE_ARCH=arm64",
      "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",
      "PIHOLE_INSTALL=/root/ph_install.sh",
      "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
      "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
      "IPv6=True",
      "S6_LOGGING=0",
      "S6_KEEP_ENV=1",
      "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
      "ServerIP=0.0.0.0",
      "FTL_CMD=no-daemon",
      "DNSMASQ_USER=root",
      "VERSION=v5.2.1"
    ],
    "Cmd": null,
    "Healthcheck": {
      "Test": [
        "CMD-SHELL",
        "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
      ]
    },
    "Image": "sha256:b4459ac3099e2fd70e9c2aa5ef710e853da11624d65f0b353f88da8c39dbd34f",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": [
      "/s6-init"
    ],
    "OnBuild": null,
    "Labels": {
      "image": "pihole/pihole:v5.2.1_arm64",
      "maintainer": "[email protected]",
      "url": "https://www.github.com/pi-hole/docker-pi-hole"
    },
    "Shell": [
      "/bin/bash",
      "-c"
    ]
  },
  "container": "35f3f315a63da09485218661dc7c1e335121dc5b3b10191c3ba2e8f3f5c718ac",
  "container_config": {
    "Hostname": "35f3f315a63d",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
      "443/tcp": {},
      "53/tcp": {},
      "53/udp": {},
      "67/udp": {},
      "80/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "ARCH=arm64",
      "UBUNTU_SUITE=buster",
      "DOCKER_REPO=multiarch/debian-debootstrap",
      "PIHOLE_ARCH=arm64",
      "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",
      "PIHOLE_INSTALL=/root/ph_install.sh",
      "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
      "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
      "IPv6=True",
      "S6_LOGGING=0",
      "S6_KEEP_ENV=1",
      "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
      "ServerIP=0.0.0.0",
      "FTL_CMD=no-daemon",
      "DNSMASQ_USER=root",
      "VERSION=v5.2.1"
    ],
    "Cmd": [
      "/bin/bash",
      "-c",
      "#(nop) ",
      "SHELL [/bin/bash -c]"
    ],
    "Healthcheck": {
      "Test": [
        "CMD-SHELL",
        "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
      ]
    },
    "Image": "sha256:b4459ac3099e2fd70e9c2aa5ef710e853da11624d65f0b353f88da8c39dbd34f",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": [
      "/s6-init"
    ],
    "OnBuild": null,
    "Labels": {
      "image": "pihole/pihole:v5.2.1_arm64",
      "maintainer": "[email protected]",
      "url": "https://www.github.com/pi-hole/docker-pi-hole"
    },
    "Shell": [
      "/bin/bash",
      "-c"
    ]
  },
  "created": "2020-12-03T02:06:14.680174992Z",
  "docker_version": "19.03.13+azure",
  "history": [
    {
      "created": "2020-09-21T13:14:16.064296896Z",
      "created_by": "/bin/sh -c #(nop) ADD file:9dec6ab242477dcf4ffa603bdfd5e27075862c7686626da9d17e46d36bdfa680 in / "
    },
    {
      "created": "2020-09-21T13:14:17.245076223Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
      "empty_layer": true
    },
    {
      "created": "2020-09-21T13:14:17.399948599Z",
      "created_by": "/bin/sh -c #(nop)  ENV ARCH=arm64 UBUNTU_SUITE=buster DOCKER_REPO=multiarch/debian-debootstrap",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:07.657658869Z",
      "created_by": "/bin/sh -c #(nop)  ARG PIHOLE_ARCH",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:08.366627676Z",
      "created_by": "/bin/sh -c #(nop)  ENV PIHOLE_ARCH=arm64",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:09.353406418Z",
      "created_by": "/bin/sh -c #(nop)  ARG S6_ARCH",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:10.341364661Z",
      "created_by": "/bin/sh -c #(nop)  ARG S6_VERSION",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:11.364166552Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:13.000183821Z",
      "created_by": "/bin/sh -c #(nop) COPY file:dcb7fff06ca7f56b62b19a51693c1fbe9cab7bbdd8d7a1f2527267fd29841558 in /usr/local/bin/install.sh "
    },
    {
      "created": "2020-12-03T01:58:14.000270627Z",
      "created_by": "/bin/sh -c #(nop) COPY file:e4764efc839663d763445f0ed35e56861f7c1801ef50ac7bbc53f057b8f32a18 in /etc/docker-pi-hole-version "
    },
    {
      "created": "2020-12-03T01:58:14.354399312Z",
      "created_by": "/bin/sh -c #(nop)  ENV PIHOLE_INSTALL=/root/ph_install.sh",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:40.705887232Z",
      "created_by": "|2 S6_ARCH=aarch64 S6_VERSION=v1.22.1.0 /bin/sh -c bash -ex install.sh 2>&1 &&     rm -rf /var/cache/apt/archives /var/lib/apt/lists/*"
    },
    {
      "created": "2020-12-03T02:05:45.742997182Z",
      "created_by": "/bin/sh -c #(nop)  ENTRYPOINT [\"/s6-init\"]",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:47.000238082Z",
      "created_by": "/bin/sh -c #(nop) ADD dir:04ba41115e115e65f57dd6ee9d6f24290644d70968720eb39690830b87003cbf in / "
    },
    {
      "created": "2020-12-03T02:05:48.000235495Z",
      "created_by": "/bin/sh -c #(nop) COPY file:5f42354bc5364cecacf611938520aa2e01ab65d469aac487bcd0e27abc5de7d5 in /usr/local/bin/service "
    },
    {
      "created": "2020-12-03T02:05:48.650120484Z",
      "created_by": "/bin/sh -c #(nop)  ARG PHP_ENV_CONFIG",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:49.639148111Z",
      "created_by": "/bin/sh -c #(nop)  ENV PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:50.654615615Z",
      "created_by": "/bin/sh -c #(nop)  ARG PHP_ERROR_LOG",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:51.629011331Z",
      "created_by": "/bin/sh -c #(nop)  ENV PHP_ERROR_LOG=/var/log/lighttpd/error.log",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:53.000296372Z",
      "created_by": "/bin/sh -c #(nop) COPY file:c4d677569a4b094380c73d2adcf27b933b8c602c200b3bc617b71aa04ad68209 in / "
    },
    {
      "created": "2020-12-03T02:05:54.000189749Z",
      "created_by": "/bin/sh -c #(nop) COPY file:f14f0f49d9cfd5e91767a9f90081149a0e1603990adec0b6336267836fbffd58 in / "
    },
    {
      "created": "2020-12-03T02:05:54.732175479Z",
      "created_by": "/bin/sh -c #(nop)  ENV IPv6=True",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:55.742387817Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 53 53/udp",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:56.637736561Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 67/udp",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:57.676535684Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 80",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:58.754594269Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 443",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:05:59.718176164Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6_LOGGING=0",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:00.635321537Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6_KEEP_ENV=1",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:01.620055766Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:02.625719432Z",
      "created_by": "/bin/sh -c #(nop)  ENV ServerIP=0.0.0.0",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:03.624323345Z",
      "created_by": "/bin/sh -c #(nop)  ENV FTL_CMD=no-daemon",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:04.611857977Z",
      "created_by": "/bin/sh -c #(nop)  ENV DNSMASQ_USER=root",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:05.61224819Z",
      "created_by": "/bin/sh -c #(nop)  ARG PIHOLE_VERSION",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:06.612663998Z",
      "created_by": "/bin/sh -c #(nop)  ENV VERSION=v5.2.1",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:07.647574535Z",
      "created_by": "/bin/sh -c #(nop)  ENV PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:08.641198085Z",
      "created_by": "/bin/sh -c #(nop)  ARG NAME",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:09.657814486Z",
      "created_by": "/bin/sh -c #(nop)  LABEL image=pihole/pihole:v5.2.1_arm64",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:10.653781641Z",
      "created_by": "/bin/sh -c #(nop)  ARG MAINTAINER",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:11.669868426Z",
      "created_by": "/bin/sh -c #(nop)  LABEL [email protected]",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:12.662539748Z",
      "created_by": "/bin/sh -c #(nop)  LABEL url=https://www.github.com/pi-hole/docker-pi-hole",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:13.657085876Z",
      "created_by": "/bin/sh -c #(nop)  HEALTHCHECK &{[\"CMD-SHELL\" \"dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1\"] \"0s\" \"0s\" \"0s\" '\\x00'}",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T02:06:14.680174992Z",
      "created_by": "/bin/bash -c #(nop)  SHELL [/bin/bash -c]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:cc77256f2d56e07985cba58506e433bb21ef7b3a06bd3a688ac6abfdfc010344",
      "sha256:c2c03ee92cd94dc4721023ebe9e6b1a1c6e3588a470d65bdabc589d68bd38395",
      "sha256:beb456e5587da0c7e993017a51b77dac721a6fc290b0c23b63fda47fa262488c",
      "sha256:038c127208f1ddfa51eab576f6de0f44c4c9991bcdb66e432d654d280db597f6",
      "sha256:7554d616a82bfa378acdc7ae9e5c3e885f29efb6cf06059071ca8648fbc9ec42",
      "sha256:110d37e83a24488a1565c2d8e5307f676a72a19d6779d0f1e97a7203f7aa03e2",
      "sha256:9657d8d1b438d67e784aaf3c96d95c943b9138efbd029453f6e7227d304c2553",
      "sha256:6fa10c09468998df7084f6affeb8d3d2b5e02ce2aaccc02249c48b41ee323392"
    ]
  }
}

and for the amd64 image:

mkdir /amd64
./contrib/download-frozen-image-v2.sh /amd64 pihole/pihole:v5.2.1@sha256:5281797ed544ae438cd3c458171212d1e0cbf5c219a1cffe28b042a9f9821791
Downloading 'pihole/pihole:v5.2.1@sha256:5281797ed544ae438cd3c458171212d1e0cbf5c219a1cffe28b042a9f9821791' (13 layers)...
...

Download of images into '/amd64' complete.
Use something like the following to load the result into a Docker daemon:
  tar -cC '/amd64' . | docker load

cat /amd64/manifest.json
[
  {
    "Config": "52e72f4b6a617b430cf14c6fa919eb477c48aee4f7f87cc1c55247c1d602a66f.json",
    "RepoTags": [
      "pihole/pihole:v5.2.1"
    ],
    "Layers": [
      "484d6ff7582f78e0c8c70410fef1e167a82f68df64895a4318689bf265f323f9/layer.tar",
      "19e0f39c9873847b2b787aa59d8ad44189327a0815444ebeacd6b3afcf858587/layer.tar",
      "2ef4549fc54ab95a45c8649ce51fc9715eb6b951b3ff8d67688d1182d3f2bb5d/layer.tar",
      "25b5949f8916972ee72de299abbb1abc26e7fd530dc0ca31d206c108abcb23b1/layer.tar",
      "93370434bdba1941d53616635f25abfe3dbad4eff47cf10d76aa194c88a005d5/layer.tar",
      "84bfbf2cb68120795b9a2e1d8e07de393a60294cae1e6f6e3470b2f5fd0d4465/layer.tar",
      "0427eb918d977916581b1691b75f3e307e2147ac4a0f0de3606d50be4614c09e/layer.tar",
      "aecb639f13655a0c9fb3ebdf36c02c5dfe67dc25514df9cf460bf75dd1c2c9ec/layer.tar",
      "55da22151a39bc4c81879de27f6987bbc621973555660d75f7ac68ffca367126/layer.tar",
      "99a94529e89780bf856193d9c78a6dc561ad27ba97bbe16420375a15bfcd8fd2/layer.tar",
      "8459e6c5e2bf3da7a66ebb99e8f4f36cd93dc3ce6a03708e1f9ac02c6fe75e88/layer.tar",
      "d8a1fe5e592b7aa0242a563e7335d0ad870556fc633bc3e205071d65761550ec/layer.tar",
      "12375d48619baf76e3cbb78676c0fd96cc51432812dc58156e7695eff86cef5f/layer.tar"
    ]
  }
]

cat /amd64/52e72f4b6a617b430cf14c6fa919eb477c48aee4f7f87cc1c55247c1d602a66f.json | jq .

{
  "architecture": "amd64",
  "config": {
    "Hostname": "",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
      "443/tcp": {},
      "53/tcp": {},
      "53/udp": {},
      "67/udp": {},
      "80/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "PIHOLE_ARCH=amd64",
      "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-amd64.tar.gz",
      "PIHOLE_INSTALL=/root/ph_install.sh",
      "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
      "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
      "IPv6=True",
      "S6_LOGGING=0",
      "S6_KEEP_ENV=1",
      "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
      "ServerIP=0.0.0.0",
      "FTL_CMD=no-daemon",
      "DNSMASQ_USER=root",
      "VERSION=v5.2.1"
    ],
    "Cmd": null,
    "Healthcheck": {
      "Test": [
        "CMD-SHELL",
        "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
      ]
    },
    "Image": "sha256:929daab554acb2a02ab991ec85bba37b6934eaad91adec262b35ab36b151a03c",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": [
      "/s6-init"
    ],
    "OnBuild": null,
    "Labels": {
      "image": "pihole/pihole:v5.2.1_amd64",
      "maintainer": "[email protected]",
      "url": "https://www.github.com/pi-hole/docker-pi-hole"
    },
    "Shell": [
      "/bin/bash",
      "-c"
    ]
  },
  "container": "6c475f695cacb914ab84e7f07d978bd5180bbb6b5b102c099eebb185f54473e3",
  "container_config": {
    "Hostname": "6c475f695cac",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "ExposedPorts": {
      "443/tcp": {},
      "53/tcp": {},
      "53/udp": {},
      "67/udp": {},
      "80/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "PIHOLE_ARCH=amd64",
      "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-amd64.tar.gz",
      "PIHOLE_INSTALL=/root/ph_install.sh",
      "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
      "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
      "IPv6=True",
      "S6_LOGGING=0",
      "S6_KEEP_ENV=1",
      "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
      "ServerIP=0.0.0.0",
      "FTL_CMD=no-daemon",
      "DNSMASQ_USER=root",
      "VERSION=v5.2.1"
    ],
    "Cmd": [
      "/bin/bash",
      "-c",
      "#(nop) ",
      "SHELL [/bin/bash -c]"
    ],
    "Healthcheck": {
      "Test": [
        "CMD-SHELL",
        "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
      ]
    },
    "Image": "sha256:929daab554acb2a02ab991ec85bba37b6934eaad91adec262b35ab36b151a03c",
    "Volumes": null,
    "WorkingDir": "",
    "Entrypoint": [
      "/s6-init"
    ],
    "OnBuild": null,
    "Labels": {
      "image": "pihole/pihole:v5.2.1_amd64",
      "maintainer": "[email protected]",
      "url": "https://www.github.com/pi-hole/docker-pi-hole"
    },
    "Shell": [
      "/bin/bash",
      "-c"
    ]
  },
  "created": "2020-12-03T01:59:33.41222977Z",
  "docker_version": "19.03.13+azure",
  "history": [
    {
      "created": "2020-11-17T20:21:17.570073346Z",
      "created_by": "/bin/sh -c #(nop) ADD file:d2abb0e4e7ac1773741f51f57d3a0b8ffc7907348842d773f8c341ba17f856d5 in / "
    },
    {
      "created": "2020-11-17T20:21:17.865210281Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
      "empty_layer": true
    },
    {
      "created": "2020-11-29T02:35:40.042063952Z",
      "created_by": "/bin/sh -c #(nop) COPY multi:adc45a1285c053f33230963c1773a349cd47bf9e84d05ea8bd222937ba7c5540 in / "
    },
    {
      "created": "2020-11-29T02:35:41.000158482Z",
      "created_by": "/bin/sh -c #(nop) COPY dir:cca3648dbdd7cf158510aa3f34162b26231a77144248f8be9cb4f249540fad23 in /etc/apt/ "
    },
    {
      "created": "2020-11-29T02:35:42.000180586Z",
      "created_by": "/bin/sh -c #(nop) COPY dir:b6fe344f2ba9a4fe86666c76dd73eba7aa0db7b37b1111a04138cd870fa7f7aa in /usr/local/repos/pam "
    },
    {
      "created": "2020-11-29T02:35:43.000150571Z",
      "created_by": "/bin/sh -c #(nop) COPY file:262f9221f7411cc3dedc913091495a2aedb98b1f29eee680ba84523cb2c322ab in /usr/local/bin/ "
    },
    {
      "created": "2020-11-29T02:35:44.578550307Z",
      "created_by": "/bin/sh -c bash -ex setup.sh && rm /usr/local/bin/setup.sh"
    },
    {
      "created": "2020-12-03T01:57:59.794627738Z",
      "created_by": "/bin/sh -c #(nop)  ARG PIHOLE_ARCH",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:00.221441121Z",
      "created_by": "/bin/sh -c #(nop)  ENV PIHOLE_ARCH=amd64",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:01.238142861Z",
      "created_by": "/bin/sh -c #(nop)  ARG S6_ARCH",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:02.240996623Z",
      "created_by": "/bin/sh -c #(nop)  ARG S6_VERSION",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:03.225666432Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-amd64.tar.gz",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:58:05.000102827Z",
      "created_by": "/bin/sh -c #(nop) COPY file:dcb7fff06ca7f56b62b19a51693c1fbe9cab7bbdd8d7a1f2527267fd29841558 in /usr/local/bin/install.sh "
    },
    {
      "created": "2020-12-03T01:58:06.000250421Z",
      "created_by": "/bin/sh -c #(nop) COPY file:e4764efc839663d763445f0ed35e56861f7c1801ef50ac7bbc53f057b8f32a18 in /etc/docker-pi-hole-version "
    },
    {
      "created": "2020-12-03T01:58:06.216505719Z",
      "created_by": "/bin/sh -c #(nop)  ENV PIHOLE_INSTALL=/root/ph_install.sh",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:00.888073076Z",
      "created_by": "|2 S6_ARCH=amd64 S6_VERSION=v1.22.1.0 /bin/sh -c bash -ex install.sh 2>&1 &&     rm -rf /var/cache/apt/archives /var/lib/apt/lists/*"
    },
    {
      "created": "2020-12-03T01:59:05.339708593Z",
      "created_by": "/bin/sh -c #(nop)  ENTRYPOINT [\"/s6-init\"]",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:06.00020376Z",
      "created_by": "/bin/sh -c #(nop) ADD dir:04ba41115e115e65f57dd6ee9d6f24290644d70968720eb39690830b87003cbf in / "
    },
    {
      "created": "2020-12-03T01:59:07.000145186Z",
      "created_by": "/bin/sh -c #(nop) COPY file:5f42354bc5364cecacf611938520aa2e01ab65d469aac487bcd0e27abc5de7d5 in /usr/local/bin/service "
    },
    {
      "created": "2020-12-03T01:59:07.400666373Z",
      "created_by": "/bin/sh -c #(nop)  ARG PHP_ENV_CONFIG",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:08.396185483Z",
      "created_by": "/bin/sh -c #(nop)  ENV PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:09.398617479Z",
      "created_by": "/bin/sh -c #(nop)  ARG PHP_ERROR_LOG",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:10.399860424Z",
      "created_by": "/bin/sh -c #(nop)  ENV PHP_ERROR_LOG=/var/log/lighttpd/error.log",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:12.00016337Z",
      "created_by": "/bin/sh -c #(nop) COPY file:c4d677569a4b094380c73d2adcf27b933b8c602c200b3bc617b71aa04ad68209 in / "
    },
    {
      "created": "2020-12-03T01:59:13.000219618Z",
      "created_by": "/bin/sh -c #(nop) COPY file:f14f0f49d9cfd5e91767a9f90081149a0e1603990adec0b6336267836fbffd58 in / "
    },
    {
      "created": "2020-12-03T01:59:13.420108554Z",
      "created_by": "/bin/sh -c #(nop)  ENV IPv6=True",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:14.401150447Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 53 53/udp",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:15.39856308Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 67/udp",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:16.413183067Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 80",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:17.423655155Z",
      "created_by": "/bin/sh -c #(nop)  EXPOSE 443",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:18.403633214Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6_LOGGING=0",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:19.419131426Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6_KEEP_ENV=1",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:20.403816404Z",
      "created_by": "/bin/sh -c #(nop)  ENV S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:21.400794455Z",
      "created_by": "/bin/sh -c #(nop)  ENV ServerIP=0.0.0.0",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:22.400339818Z",
      "created_by": "/bin/sh -c #(nop)  ENV FTL_CMD=no-daemon",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:23.416784726Z",
      "created_by": "/bin/sh -c #(nop)  ENV DNSMASQ_USER=root",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:24.396713912Z",
      "created_by": "/bin/sh -c #(nop)  ARG PIHOLE_VERSION",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:25.411531635Z",
      "created_by": "/bin/sh -c #(nop)  ENV VERSION=v5.2.1",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:26.41904171Z",
      "created_by": "/bin/sh -c #(nop)  ENV PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:27.404895307Z",
      "created_by": "/bin/sh -c #(nop)  ARG NAME",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:28.414446456Z",
      "created_by": "/bin/sh -c #(nop)  LABEL image=pihole/pihole:v5.2.1_amd64",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:29.399637486Z",
      "created_by": "/bin/sh -c #(nop)  ARG MAINTAINER",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:30.395105252Z",
      "created_by": "/bin/sh -c #(nop)  LABEL [email protected]",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:31.411684327Z",
      "created_by": "/bin/sh -c #(nop)  LABEL url=https://www.github.com/pi-hole/docker-pi-hole",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:32.41615888Z",
      "created_by": "/bin/sh -c #(nop)  HEALTHCHECK &{[\"CMD-SHELL\" \"dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1\"] \"0s\" \"0s\" \"0s\" '\\x00'}",
      "empty_layer": true
    },
    {
      "created": "2020-12-03T01:59:33.41222977Z",
      "created_by": "/bin/bash -c #(nop)  SHELL [/bin/bash -c]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:f5600c6330da7bb112776ba067a32a9c20842d6ecc8ee3289f1a713b644092f8",
      "sha256:333ec1b36502156108bb75d5372e9bd01660ef212c63c6852afb719f2578a135",
      "sha256:7337aaa591164c259c17bb8de3b91c5aa856aa8b60bd7ed4029fcb340864134c",
      "sha256:2ad2d3172ff73c6009e9386e1a826bee4ebf6b6767bd275657b4269f771c8ff5",
      "sha256:a75feaf7b4ef0e0a2a9f2d1c7ea3737e34e2bfaac268ef854d9c13a4bd69bd95",
      "sha256:40821dc7d051aa9e92ce9de97b50d1ed3590e9ab7f9281ebabb3a2c631057d57",
      "sha256:4e1202651bb25616397108cc90af3940c2fd93b8d292978a6791046c97f347bc",
      "sha256:f7e072f2db4609884ad427f42b4a7ddab30c0ebe208d8e98ce2b38b8efc03f0c",
      "sha256:2a617d1b6a13f220d026a4a509ec0ff5fa536f6899a72f4350f6820ce654ac74",
      "sha256:7554d616a82bfa378acdc7ae9e5c3e885f29efb6cf06059071ca8648fbc9ec42",
      "sha256:c3e81aeb769d23e17e7bc675f016b640bc2a9bb2214cebf089521f554102f367",
      "sha256:9657d8d1b438d67e784aaf3c96d95c943b9138efbd029453f6e7227d304c2553",
      "sha256:6fa10c09468998df7084f6affeb8d3d2b5e02ce2aaccc02249c48b41ee323392"
    ]
  }
}

So for both images, the image that's in the registry has "amd64" set as "architecture". I suspect the image is cross-compiled, and has the incorrect architecture set.

thaJeztah · 2020-12-23T16:20:11Z

So concluding;

looks like for this case, the issue is actually with the images having the incorrect architecture set in their metadata
docker 20.10 now prints a warning ("WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested"). Older versions of docker silently tried to run the image (which in this case would work, because the image is an arm image, just "labeled" incorrectly
docker 20.10 now produces an error if the architecture of the image isn't supported by the engine (which it bases on the architecture specified in the image's manifest)

docker run -it --rm --platform=linux/arm64 --entrypoint /bin/sh pihole/pihole:v5.2.1
Unable to find image 'pihole/pihole:v5.2.1' locally
v5.2.1: Pulling from pihole/pihole
aceecb32d1c3: Pull complete
5619849a9c12: Pull complete
e998d3f61df7: Pull complete
b1ab287d11d2: Pull complete
f45b672ae3e8: Pull complete
efe0958f28c6: Pull complete
1292910fb137: Pull complete
a54bdd0520cd: Pull complete
Digest: sha256:7888e89f4d435673c75896c546fefe3177379c99c8766c89be25a905f8def533
Status: Downloaded newer image for pihole/pihole:v5.2.1
docker: Error response from daemon: image with reference pihole/pihole:v5.2.1 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.
See 'docker run --help'.

Perhaps a "I know what I'm doing; ignore what the image says" flag would be needed?

thaJeztah · 2020-12-23T16:23:04Z

~~Note that the above is on Docker Desktop, which can run non-matching architectures (through qemu), but for that requires the image metadata to reflect the correct architecture.~~

Ignore this; see below

thaJeztah · 2020-12-23T21:03:17Z

Note that the above is on Docker Desktop, which can run non-matching architectures (through qemu), but for that requires the image metadata to reflect the correct architecture.

Actually the above isn't correct; Docker Desktop uses binfmt for this, which looks for the architecture of the binary itself, so doesn't need the metadata to be correct.

thaJeztah · 2020-12-23T21:19:36Z

So, the error (and failure) is only in situations where --platform was explicitly set, which forces the platform to match what's specified; the error is produced in this part of the code;

https://github.com/moby/moby/blob/8a4671fb1f89d222bd65cb07ea6a0de1ec7a1a2b/daemon/images/image.go#L46-L57

$ docker run --rm --platform=linux/arm64 --entrypoint /bin/sh pihole/pihole:v5.2.1 -c 'uname -m'
Unable to find image 'pihole/pihole:v5.2.1' locally
v5.2.1: Pulling from pihole/pihole
Digest: sha256:7888e89f4d435673c75896c546fefe3177379c99c8766c89be25a905f8def533
Status: Image is up to date for pihole/pihole:v5.2.1
docker: Error response from daemon: image with reference pihole/pihole:v5.2.1 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.
See 'docker run --help'.

In the above;

docker checks if pihole/pihole:v5.2.1 is present in the image cache AND matches the specified platform
it discovers that an image exists, but not with the right platform ("Unable to find image 'pihole/pihole:v5.2.1' locally")
therefore pulls the image ("v5.2.1: Pulling from pihole/pihole")
this pulls the linux/arm64 variant of the image (which is actually the one we already pulled)
5 the image is pulled successfully, and is now in the local image cache
docker proceeds to create / run a container using that image;
when creating the container, it (again) looks up the pihole/pihole:v5.2.1 image in the image cache, and checks if the platform matches what's passed in the --platform flag
This is what fails, because even though the manifest list specifies the variant that was pulled to be linux/arm64, the actual image (incorrectly) has architecture: linux/amd64
docker produces an error because of the above ("image with reference pihole/pihole:v5.2.1 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64")

If, on the other hand, we first pull the image with the desired platform, then create/run a container (without explicitly setting --platform), then docker will pick whatever variant of the image is present (irregardless of the architecture), and will run it;

$ docker run --rm --entrypoint /bin/sh pihole/pihole:v5.2.1 -c 'uname -m'
aarch64

thaJeztah · 2020-12-23T21:25:03Z

Orthogonal to the above, but there's some (error) messages that could be improved:

Unable to find image 'pihole/pihole:v5.2.1' locally

This could include the desired os/architecture, e.g.:

Unable to find image 'pihole/pihole:v5.2.1' for platform linux/arm64

The pull output could also include a line with the platform/architecture that was pulled;

v5.2.1: Pulling from pihole/pihole
aceecb32d1c3: Pull complete
...
Digest: sha256:7888e89f4d435673c75896c546fefe3177379c99c8766c89be25a905f8def533
Status: Downloaded newer image for pihole/pihole:v5.2.1

e.g.;

v5.2.1: Pulling from pihole/pihole (platform: linux/arm64)
aceecb32d1c3: Pull complete
...
Digest: sha256:7888e89f4d435673c75896c546fefe3177379c99c8766c89be25a905f8def533
Platform: linux/arm64
Status: Downloaded newer image for pihole/pihole:v5.2.1

Or splitting the fields (not sure if "os-features" and "variant" are part of the OCI spec now)

Digest: sha256:7888e89f4d435673c75896c546fefe3177379c99c8766c89be25a905f8def533
Platform: os: linux, architecture: arm64, variant: xxx, os-features: xxx
Status: Downloaded newer image for pihole/pihole:v5.2.1

edit: opened moby/moby#41838 for tracking the enhancements above

tonistiigi · 2020-12-28T17:39:51Z

Warning on non-native default arch is correct as we don't want to give an assumption that qemu is something that can be used in production or for long running jobs. If users know what they are doing they can ignore the warning.

Erroring if --platform is set but does not match the image is correct as well. If users asked for one architecture we can't just give them another one. The exception here could be the architectures that are compatible with each other. Eg. same way as we will fall back to a lower arm variant on pull we should fall back to a lower variant here without erroring. Not sure if this is the case atm. @cpuguy83

If the image is corrupt and manifest lists a different architecture than config then these images need to be fixed and we should not try to ignore errors. Images that are built for non-native architecture need to be built with --platform flag. You can't just copy in a binary for another architecture while building image for your native architecture and then rewrite only the manifest metadata with an external tool. Eg. as seen in estesp/mquery#9 . Things we could do is update the open-source registry to validate for this case and refuse push. We could also show a warning on pull if corruption is detected.

gruijter · 2021-01-08T12:26:07Z

I hope this can be fixed soon! Since docker 20.10 I am unable to run some important images on my Rpi4 with raspberry OS (32bit). The same images used to work fine with docker 19. Weird thing is, images that were already running when I installed them using docker 19 kept working after upgrading to docker 20. But trying to run from the same image again gives an error. Using --platform=linux/arm/v7 does not help either.

thaJeztah · 2021-01-08T13:02:30Z

@gruijter do you have names for the images that don't work? This issue is cause by images that are invalid, so effectively, it's not a bug (but more a bug-fix that the invalid images were not caught before).

gruijter · 2021-01-08T13:07:16Z

https://hub.docker.com/r/uifd/ui-for-docker

https://hub.docker.com/r/portainer/portainer-ce

Especially portainer is crucial for me :)

thaJeztah · 2021-01-08T16:15:29Z

https://hub.docker.com/r/uifd/ui-for-docker doesn't look to be a multi-arch image, and doesn't have a variant for arm machines; https://hub.docker.com/r/uifd/ui-for-docker/tags?page=1&ordering=last_updated, so that's expected.

The portainer image looks to be broken; it shows it's a linux/amd64 image;

$ docker image pull portainer/portainer-ce@sha256:ffd59292a3253137569609fba72ad2e572b050ba9deb1a7b9a94d388693f236a

docker.io/portainer/portainer-ce@sha256:ffd59292a3253137569609fba72ad2e572b050ba9deb1a7b9a94d388693f236a: Pulling from portainer/portainer-ce
b890dbc4eb27: Pull complete
d39f4eb29d9f: Pull complete
Digest: sha256:ffd59292a3253137569609fba72ad2e572b050ba9deb1a7b9a94d388693f236a
Status: Downloaded newer image for portainer/portainer-ce@sha256:ffd59292a3253137569609fba72ad2e572b050ba9deb1a7b9a94d388693f236a
docker.io/portainer/portainer-ce@sha256:ffd59292a3253137569609fba72ad2e572b050ba9deb1a7b9a94d388693f236a

$ docker image inspect --format '{{.Os}}/{{.Architecture}}' portainer/portainer-ce@sha256:ffd59292a3253137569609fba72ad2e572b050ba9deb1a7b9a94d388693f236a

linux/amd64

Looking at how it's built; they cross-compile binaries; https://github.com/portainer/portainer/blob/cd475a533835aa923970bfe9c7c5b2a46c3040c0/build.sh#L24-L48

Then to create the image https://github.com/portainer/portainer/blob/cd475a533835aa923970bfe9c7c5b2a46c3040c0/build/linux/Dockerfile#L1

It start with portainer/base, which is a "from scratch" image; https://hub.docker.com/r/portainer/base/dockerfile

Then it copies the binaries into that image; https://github.com/portainer/portainer/blob/cd475a533835aa923970bfe9c7c5b2a46c3040c0/build/linux/Dockerfile#L3

So, yes, that's definitely the wrong way to get that done, because it's an amd64 (x86) image, in which they put an arm binary; I'd recommend opening a ticket in their repository, because that's a bug in their build process.

cpuguy83 · 2021-01-08T18:39:47Z

This error only occurs when the specified platform on container create does not match the one on the image.
If the user does not specify a platform it should only have a warning.

Are you sending a platform to use along with the container creation?

thaJeztah · 2021-02-23T13:39:15Z

Closing this ticket as this will be fixed in docker 20.10.4 (to be released soon)

piksel · 2021-02-23T14:11:07Z

@thaJeztah Well, the problem with the blog post you linked is that the "hard way" mentioned (using docker manifest) is that it only produces manifests with the given platform, the images' platform might still be wrong. That's because docker build sets the platform to the hosts platform for FROM scratch base images.

I think that is the source of these mismatched image/manifests.
If the above behaviour is actually what is happening, perhaps that could be included in the documentation somewhere? This is especially a gotcha for go projects since the go compiler can do cross platform compilation "natively" (and produces statically compiled binaries, so no additional files are necessary in the image).

thaJeztah · 2021-02-23T15:50:41Z

Right, if "building" the image is done by building the binaries separately and then only copying them to a FROM scratch image, then assembling a multi-arch image using docker manifest, it's important to make sure that the images have the correct architecture set.

Generally, the recommendation would be to perform the cross-build as part of the Dockerfile, but if binaries are built outside of the Dockerfile you can still use the automatic variables that BuildKit provide (TARGETPLATFORM, TARGETOS etc.). The list of variables is documented in the builder reference, and a short example in the buildx docs.

For example, with this Dockerfile;

FROM scratch
ARG TARGETOS
ARG TARGETARCH
COPY ./mybinary-${TARGETOS}-${TARGETARCH} ./binary

Mimicking some binaries that were cross-compiled on the host:

touch mybinary-linux-amd64 mybinary-linux-arm64

export DOCKER_BUILDKIT=1

export OS=linux ARCH=amd64
docker build -t myimage:$OS-$ARCH --platform=$OS/$ARCH .
docker image inspect --format '# Architecture: {{.Architecture}} OS: {{.Os}}' myimage:$OS-$ARCH
# Architecture: amd64 OS: linux

export OS=linux ARCH=arm64
docker build -t myimage:$OS-$ARCH --platform=$OS/$ARCH .
docker image inspect --format '# Architecture: {{.Architecture}} OS: {{.Os}}' myimage:$OS-$ARCH
# Architecture: arm64 OS: linux

piksel · 2021-02-24T01:04:04Z

Sure, that would work, but I am not looking for a way to set up the builds, that has worked all the time. I'm just pointing out that people have been able to build images this way for a long time without even knowing that the images themselves had another platform, and that to solve the issue they just need to add the --platform flag to docker build to set up the image correctly.

But as you pointed out above, the problem has been "fixed" for now, I just figured that that piece of information would speed up the actual fixing of the mismatched repo images. 🤷‍♀️

pdcastro · 2021-03-18T00:41:57Z

This issue is considered to be fixed in Docker Engine v20.10.4, but it doesn't seem to be the end of the story. :-) The fix results in docker build errors as detailed in issue moby/moby/issues/42158.

thaJeztah · 2021-03-18T19:59:26Z

@pdcastro that's the expected behavior; I left a comment on that issue; moby/moby#42158 (comment)

jamshid · 2021-04-20T18:57:52Z

Ah thanks, I also ran into this because I was building my multi-arch image FROM a base image that was only x86.
Seemed like I was doing everything right with builds and my image and this didn't report any errors so I couldn't figure out why --platform was being ignored.

# Running on an M1 Docker for Mac 3.3.1
 % docker run -ti --platform linux/arm64 192.168.1.50:5000/myelasticsearch:7.5.2b bash
[root@fbc1f90407fa /]# uname -a
Linux fbc1f90407fa 5.10.25-linuxkit #1 SMP PREEMPT Tue Mar 23 09:24:45 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I'm surprised that didn't report an error, that the image didn't match the requested platform, and that docker buildx build --platform linux/arm64,linux/amd64 --push ... didn't report an error that the base image was not arm64.

thaJeztah · 2021-04-21T11:06:03Z

I'm surprised that didn't report an error, that the image didn't match the requested platform, and that docker buildx build --platform linux/arm64,linux/amd64 --push ... didn't report an error that the base image was not arm64.

@jamshid hmm.. yes, I agree. It's possible buildx either follows a different code-path and/or that the warning doesn't "bubble up" to the client (I know there's some improvements to be made to print "non-fatal" messages during build).

If would be helpful if you have a (preferably "minimal as possible") example of such a build, so that we can check your scenario (and if a fix is needed for that)
Could you open a ticket in the buildx issue tracker? https://github.com/docker/buildx/issues

thaJeztah · 2021-04-21T11:09:04Z

a (preferably "minimal as possible") example of such a build

Probably should've worded "a build" somewhat better; I tried to describe "reproduction steps" (Dockerfile that's using a public image, doesn't depend on (e.g.) a private GitHub repository, and the exact docker buildx command to run), so that we can reproduce the scenario. 😅

laotoutou · 2021-12-01T07:02:43Z

I add --platform=linux/x86_64 in my dockerfile, like this:
FROM --platform=linux/x86_64 golang:1.14 as build
I add --platform=linux/x86_64 when docker build:
docker build --platform=linux/x86_64

In this way it can run

thaJeztah changed the title ~~docker-ce v20: Platform architecture detection is broken~~ [v20.10] Platform architecture detection is broken Dec 23, 2020

shaderecker mentioned this issue Dec 23, 2020

Wrong platform architecture on Docker Hub Images pi-hole/docker-pi-hole#735

Closed

thaJeztah mentioned this issue Dec 23, 2020

UX: improve messages / warnings /errors to include platform (os/arch/xx) moby/moby#41838

Open

This was referenced Jan 8, 2021

Docker image for arm/v7 is broken- it uses a linux/amd64 base image and cannot be run on RPi portainer/portainer#4690

Closed

Portainer 2.0 with RPI 4 and Raspberry Pi OS (64 bit) problem portainer/portainer#4663

Closed

nouaman mentioned this issue Jan 10, 2021

support for apple silicon linux/arm64/v8 platform google/cadvisor#2763

Closed

sozercan mentioned this issue Jan 12, 2021

Fix non-amd64 arch config for debian images kubernetes/release#1839

Merged

thaJeztah mentioned this issue Jan 14, 2021

Fix builder inconsistent error on buggy platform moby/moby#41873

Merged

thaJeztah closed this as completed Feb 23, 2021

mvadu mentioned this issue Mar 5, 2021

[v20.10.5 ] Platform architecture detection #1214

Closed

3 tasks

CheckeredFlag mentioned this issue Mar 15, 2021

Warning when starting on Raspberry Pi 4 jaymoulin/docker-plex#31

Closed

toochevere mentioned this issue Mar 16, 2021

[v20.10.5] Inconsistent behavior of --platform flag results in build failure moby/moby#42158

Closed

This was referenced Mar 17, 2021

build, deploy: Add support for multi-architecture base images balena-io/balena-cli#2145

Closed

balena push selects incorrect image (multiarch images, Docker manifest) balena-io/balena-cli#1408

Closed

meinname mentioned this issue Mar 19, 2021

Add support for arm64 in Docker image openbullet/OpenBullet2#131

Closed

thaJeztah mentioned this issue Apr 26, 2021

docker pull --platform=linux/arm64 silently falls back to linux/amd64 image docker/for-mac#5625

Closed

3 tasks

marvin-robot mentioned this issue Aug 30, 2021

DockerStorage not running on different architecture (x86 vs M1) PrefectHQ/prefect#4918

Closed

This was referenced Sep 7, 2021

Unable to generate arm image for bumping Alpine to 3.14.2 openzipkin/docker-alpine#14

Closed

Possible fix for multi-arch build failures openzipkin/docker-alpine#15

Merged

dionjwa mentioned this issue Oct 22, 2021

Inconsistent or wrong architecture tags, cannot run or build on a raspberry pi 4 nodejs/docker-node#1589

Open

wdiechmann mentioned this issue Dec 20, 2021

how to install on M1 (running in a container) rails/tailwindcss-rails#104

Closed

ShahanaFarooqui mentioned this issue Feb 9, 2022

Wrong architecture on docker arm32v7 image Ride-The-Lightning/RTL#940

Closed

barbeau mentioned this issue Apr 21, 2022

Docker doesn't work on M1 Mac MobilityData/gtfs-realtime-validator#148

Closed

boojack mentioned this issue Jul 3, 2022

Memos unsupported arm on raspberry pi usememos/memos#101

Closed

amraln mentioned this issue Aug 10, 2022

fix: resolve arm build inside Dockerfile appvantageasia/html2pdf-service#173

Merged

hubertdeng123 mentioned this issue Aug 29, 2022

Fix fresh install of self-hosted for ARM by removing the --platform parameter getsentry/self-hosted#1671

Closed

gracegoo-stripe mentioned this issue Nov 14, 2022

Removing the platform tag stripe/stripe-cli#1002

Merged

thaJeztah mentioned this issue Aug 1, 2024

c8d/image: Simplify presentImages and better "platform not found" error moby/moby#48276

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v20.10] Platform architecture detection is broken #1170

[v20.10] Platform architecture detection is broken #1170

shaderecker commented Dec 22, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

gerrodlop commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

shaderecker commented Dec 23, 2020

shaderecker commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020 •

edited

Loading

thaJeztah commented Dec 23, 2020 •

edited

Loading

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020 •

edited

Loading

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020 •

edited

Loading

tonistiigi commented Dec 28, 2020

gruijter commented Jan 8, 2021 •

edited

Loading

thaJeztah commented Jan 8, 2021

gruijter commented Jan 8, 2021 •

edited

Loading

thaJeztah commented Jan 8, 2021

cpuguy83 commented Jan 8, 2021

thaJeztah commented Feb 23, 2021

piksel commented Feb 23, 2021 •

edited

Loading

thaJeztah commented Feb 23, 2021

piksel commented Feb 24, 2021

pdcastro commented Mar 18, 2021

thaJeztah commented Mar 18, 2021

jamshid commented Apr 20, 2021

thaJeztah commented Apr 21, 2021

thaJeztah commented Apr 21, 2021

laotoutou commented Dec 1, 2021

[v20.10] Platform architecture detection is broken #1170

[v20.10] Platform architecture detection is broken #1170

Comments

shaderecker commented Dec 22, 2020

Expected behavior

Actual behavior

Steps to reproduce the behavior

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

gerrodlop commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

shaderecker commented Dec 23, 2020

shaderecker commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020 • edited Loading

thaJeztah commented Dec 23, 2020 • edited Loading

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020 • edited Loading

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020 • edited Loading

tonistiigi commented Dec 28, 2020

gruijter commented Jan 8, 2021 • edited Loading

thaJeztah commented Jan 8, 2021

gruijter commented Jan 8, 2021 • edited Loading

thaJeztah commented Jan 8, 2021

cpuguy83 commented Jan 8, 2021

thaJeztah commented Feb 23, 2021

piksel commented Feb 23, 2021 • edited Loading

thaJeztah commented Feb 23, 2021

piksel commented Feb 24, 2021

pdcastro commented Mar 18, 2021

thaJeztah commented Mar 18, 2021

jamshid commented Apr 20, 2021

thaJeztah commented Apr 21, 2021

thaJeztah commented Apr 21, 2021

laotoutou commented Dec 1, 2021

thaJeztah commented Dec 23, 2020 •

edited

Loading

thaJeztah commented Dec 23, 2020 •

edited

Loading

thaJeztah commented Dec 23, 2020 •

edited

Loading

thaJeztah commented Dec 23, 2020 •

edited

Loading

gruijter commented Jan 8, 2021 •

edited

Loading

gruijter commented Jan 8, 2021 •

edited

Loading

piksel commented Feb 23, 2021 •

edited

Loading