docker pull --platform=linux/arm64 silently falls back to linux/amd64 image

[jsleeio]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features (had not enabled any)
• I have uploaded Diagnostics
• Diagnostics ID:

Expected behavior

docker pull --platform=linux/arm64 ... pulls image for linux/arm64 platform, or fails with an error if it is not available

Actual behavior

Docker appears to fall back to linux/amd64 which was extremely surprising behaviour

Information

• macOS Version: 11.2.3
• Intel chip or Apple chip: Apple M1 (Macbook Pro)
• Docker Desktop Version: 3.3.1 (63152)

Also reproduced with 3.3.0 and 3.3.1 on an M1 Mac Mini

Steps to reproduce the behavior

### make sure no images are lying around to confuse things
$ docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B


### no images present
$ docker images -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE


### try to pull an arm64 Terraform image --- which doesn't exist, but linux/amd64 does
$ docker pull --platform=linux/arm64 hashicorp/terraform:0.12.30
0.12.30: Pulling from hashicorp/terraform
801bfaa63ef2: Pull complete
99202a18a200: Pull complete
f982975ff37a: Pull complete
Digest: sha256:e47cb5dff9984c39affb320610879b817f98fa89835178cf05707d9f299ce4cc
Status: Downloaded newer image for hashicorp/terraform:0.12.30
docker.io/hashicorp/terraform:0.12.30


$ docker images -a
REPOSITORY            TAG       IMAGE ID       CREATED        SIZE
hashicorp/terraform   0.12.30   70de4cb15e1d   3 months ago   98.1MB


$ docker inspect 70de4cb15e1d | jq .[].Architecture
"amd64"

[thaJeztah]

Thanks for reporting! The --platform flag is used to select a specific variant when pulling a multi-arch image, in which case it will produce an error if no matching platform (os/arch) was found.

For non-multi-arch images, we had to be slightly more "permissive" for backward compatibility, and it is (currently) ignored, although we probably could make this an "error" if --platform is explicitly specified, and the image that was pulled was a single-arch image, and does not match the requested os/arch.

I agree that this scenario at least should print a warning, and I'm pushing a pull request in the upstream "moby" repository for this (possibly to be included in a patch release), and will open a tracking issue to consider making this specific scenario an error instead of a warning (but pulling a single-arch, non-matching image without specifying --platform will still be allowed for backward compatibility).

Some issues / pull requests related to the above:

• [v20.10] Platform architecture detection is broken for-linux#1170
• Fix builder inconsistent error on buggy platform moby/moby#41873
• Fallback to manifest list when no platform match moby/moby#41955

[thaJeztah]

Pull request; moby/moby#42325

[jsleeio]

thanks for chasing this up, @thaJeztah! I don't have the wider view of compatibility hazards that Docker developers clearly do, but in my narrow little world I think it should be an error

[expressrussian]

I confirm the issue on Raspberry Pi 4b, Ubuntu 20.04 aarch64 (arm64).

docker pull --platform=linux/arm64 guacamole/guacamole
Docker pulls it and does not care that it pulls invalid images! Then when i run it:

docker run --rm guacamole/guacamole /opt/guacamole/bin/initdb.sh --mysql > initdb.sql
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
standard_init_linux.go:219: exec user process caused: exec format error

This is not user-friendly.
Arm and Docker: Better Together?

[jonathanalevi]

Now having this issue trying to build a docker of charge-lnd on a Raspberry Pi 4B as well:
accumulator/charge-lnd#35

Has anyone figured out a way around this? It's driving me crazy.

[wasabii]

I don't know if anybody is still watching this issue, but it's still open. So I'll add something else to the mix.

If you have locally cached images, it will consider them a match, instead of checking for a multi-arch build:

failed to get destination image "sha256:e1c866cab3f50207e609f38461e2136c505a212746964e0e90982ed341b5dabb": image with reference sha256:e1c866cab3f50207e609f38461e2136c505a212746964e0e90982ed341b5dabb was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64

So, this appeared on one of my build servers. As part of initialization of that build server, we pre-cache some upstream images by running docker pull {image}. When actually running builds on this machine, we use docker-compose with platform: to specify ARM64. This prints out the error. Adding '--pull' to docker-compose makes the problem go away.

So I'm pretty sure the precaching is precaching the local platform. And then it gets later confused when a request comes in for a specific platform.

[dvaldivia]

is it possible to pull all the multi-arch images so they can be tagged and pushed to another registry?

[thaJeztah]

So, this appeared on one of my build servers. As part of initialization of that build server, we pre-cache some upstream images by running docker pull {image}. When actually running builds on this machine, we use docker-compose with platform: to specify ARM64. This prints out the error. Adding '--pull' to docker-compose makes the problem go away.

What version of docker are you running there, and are you using buildkit for builds? Does the same happen with a plain docker build, or only with docker compose?

Tried reproducing that situation;

Pull the linux/arm64 variant of the hello-world image;

docker pull --platform=linux/arm64 hello-world
Using default tag: latest
latest: Pulling from library/hello-world
Digest: sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685
Status: Image is up to date for hello-world:latest
docker.io/library/hello-world:latest

Building an image that uses that as a base-image, but specifying the target platform to build (amd64); BuildKit should pull the correct variant;

docker build --platform=linux/amd64 -t foo -<<EOF
FROM hello-world:latest
LABEL foo=bar
EOF

[+] Building 2.6s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                          0.0s
 => => transferring dockerfile: 80B                                                                                           0.0s
 => [internal] load .dockerignore                                                                                             0.0s
 => => transferring context: 2B                                                                                               0.0s
 => [internal] load metadata for docker.io/library/hello-world:latest                                                         2.2s
 => [auth] library/hello-world:pull token for registry-1.docker.io                                                            0.0s
 => [1/1] FROM docker.io/library/hello-world:latest@sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685   0.2s
 => => resolve docker.io/library/hello-world:latest@sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685   0.0s
 => => sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54 2.48kB / 2.48kB                                0.2s
 => => sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685 2.56kB / 2.56kB                                0.0s
 => => sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4 525B / 525B                                    0.0s
 => => sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412 1.47kB / 1.47kB                                0.0s
 => => extracting sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54                                     0.0s
 => exporting to image                                                                                                        0.0s
 => => exporting layers                                                                                                       0.0s
 => => writing image sha256:463ca4dc1fb39ebf35a4368b52fc5e2b72516379dd06e312022fa3f8bf46217f                                  0.0s
 => => naming to docker.io/library/foo                                                                                        0.0s

docker inspect --format='{{.Architecture}}' foo
amd64

Even if no platform was specified, it looks like it's checking with the registry and pulling / using the correct variant;

docker pull --platform=linux/arm64 hello-world

docker build -t foo -<<EOF
FROM hello-world:latest
LABEL foo=bar
EOF

[+] Building 1.0s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 80B                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                   0.0s
 => => transferring context: 2B                                                                                                     0.0s
 => [internal] load metadata for docker.io/library/hello-world:latest                                                               0.9s
 => CACHED [1/1] FROM docker.io/library/hello-world:latest@sha256:cc15c5b292d8525effc0f89cb299f1804f3a725c8d05e158653a563f15e4f685  0.0s
 => exporting to image                                                                                                              0.0s
 => => exporting layers                                                                                                             0.0s
 => => writing image sha256:463ca4dc1fb39ebf35a4368b52fc5e2b72516379dd06e312022fa3f8bf46217f                                        0.0s
 => => naming to docker.io/library/foo                                                                                              0.0s

docker inspect --format='{{.Architecture}}' foo-amd64
amd64

[thaJeztah]

is it possible to pull all the multi-arch images so they can be tagged and pushed to another registry?

Currently not; the local image cache can only store a single architecture per image, so if you pull the same image but with a different platform specified, then the latter will overwrite the former.

Or, more factually, the image-layers will still be there (until you docker prune), but the image name/tag will point to what's pulled last.

There are various tools that can help if you want to copy images (including all os/architectures) between registries (e.g. to mirror content), for example https://github.com/regclient/regclient

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked