[v20.10.5 ] Platform architecture detection

[mvadu]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Since 20.10.5 (with fix for #1170 platform detection is broken other way on SBCs running 64bit kernel and 32bit user land. Raspberry Pi 4 in my case.

Expected behavior

docker pull should use the highest available platform. Trying to pull an image with arm64 support should work on a 64 bit kernel engine.

Actual behavior

docker fails to detect the arm64 part, and tries to pull arm/v7, which is not supported by image.

Steps to reproduce the behavior

1. Install fresh raspi os (either get 64 bit image, or enable kernal using arm_64bit=1 in /boot/config.txt
2. install docker
3. issue docker pull influxdb

docker pull influxdb
Using default tag: latest
latest: Pulling from library/influxdb
no matching manifest for linux/arm/v7 in the manifest list entries

4. Manually setting platform to arm64 works fine

docker pull influxdb --platform=arm64
Using default tag: latest
latest: Pulling from library/influxdb
Digest: sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe
Status: Image is up to date for influxdb:latest
docker.io/library/influxdb:latest

5. Trying to add platform: arm64 to a compose file and trying to use in docker stack deploy fails with services.influxdb Additional property platform is not allowed

Output of docker version:

 docker version
Client: Docker Engine - Community
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:18:46 2021
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:16:18 2021
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

 docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 9
  Running: 4
  Paused: 0
  Stopped: 5
 Images: 10
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: mjp1ksry2ijdrvc6sbc00a1w7
  Is Manager: true
  ClusterID: 8011srp12bt8pu7fvhw60lvqo
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.23.3
  Manager Addresses:
   192.168.23.3:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.17-v8+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.806GiB
 Name: raspberrypi4
 ID: KAH2:56LY:CSA5:PWR4:LY5D:IBZR:Q2PG:2PBN:VZU4:U3FW:USBL:NXWT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

[cpuguy83]

Can you paste logs associated with the pull?

[mvadu]

Are you referring to CLI output?

I cleared all setup, fresh start - sudo service docker stop ; sudo rm -r /var/lib/docker/; sudo service docker start

>docker pull influxdb
Using default tag: latest
latest: Pulling from library/influxdb
no matching manifest for linux/arm/v7 in the manifest list entries

>docker pull influxdb --platform=arm64
Using default tag: latest
latest: Pulling from library/influxdb
c78c297fb0d0: Pull complete
06af62193c25: Pull complete
8b846e1b7390: Pull complete
7fa5b994e094: Pull complete
912bfc01b864: Pull complete
f1872ae9f82e: Pull complete
14971371c2fa: Pull complete
218ef0f3c9be: Pull complete
faa40b3dbe64: Pull complete
Digest: sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe
Status: Downloaded newer image for influxdb:latest
docker.io/library/influxdb:latest

If there is any CLI verbose logging that I can enable to show the failed pull, please let me know.

[cpuguy83]

No I mean the docker daemon logs.
journalctl -fu docker

[mvadu]

Here you go.. keep in mind, i did a service restart per my last comment.

-- Logs begin at Fri 2021-03-05 11:59:45 EST. --
Mar 05 13:03:01 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:01.851488326-05:00" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Mar 05 13:03:01 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:01.852219466-05:00" level=info msg="Loading containers: start."
Mar 05 13:03:02 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:02.636474039-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.062972196-05:00" level=info msg="Loading containers: done."
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.237784709-05:00" level=info msg="Docker daemon" commit=363e9a8 graphdriver(s)=overlay2 version=20.10.5
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.238439294-05:00" level=info msg="Daemon has completed initialization"
Mar 05 13:03:03 raspberrypi4 systemd[1]: Started Docker Application Container Engine.
Mar 05 13:03:03 raspberrypi4 dockerd[16137]: time="2021-03-05T13:03:03.616195342-05:00" level=info msg="API listen on /var/run/docker.sock"
Mar 05 13:04:36 raspberrypi4 dockerd[16137]: time="2021-03-05T13:04:36.182066506-05:00" level=info msg="Attempting next endpoint for pull after error: no matching manifest for linux/arm/v7 in the manifest list entries"
Mar 05 13:27:59 raspberrypi4 dockerd[16137]: time="2021-03-05T13:27:59.599378523-05:00" level=info msg="Attempting next endpoint for pull after error: no matching manifest for linux/arm/v7 in the manifest list entries"

[cpuguy83]

Sorry, can you add "debug": true to /etc/docker/daemon.json... from there you can either restart the daemon or sudo kill -s HUP $(pidof dockerd).

So far from what I can tell, docker is detecting your system as armv7 instead of arm64, but debug logs should help in knowing what it is doing.

[mvadu]

gotcha.. Took me some time to realize there is no config by default. I created one with just

 $ cat /etc/docker/daemon.json
{
  "debug": true
}

 $ sudo service docker stop ; sudo rm -r /var/lib/docker/; sudo service docker start

 $ docker pull influxdb
Using default tag: latest
latest: Pulling from library/influxdb
no matching manifest for linux/arm/v7 in the manifest list entries
 $ docker pull influxdb --platform=arm64
Using default tag: latest
latest: Pulling from library/influxdb
c78c297fb0d0: Pull complete
06af62193c25: Pull complete
8b846e1b7390: Pull complete
7fa5b994e094: Pull complete
912bfc01b864: Pull complete
f1872ae9f82e: Pull complete
14971371c2fa: Pull complete
218ef0f3c9be: Pull complete
faa40b3dbe64: Pull complete
Digest: sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe
Status: Downloaded newer image for influxdb:latest
docker.io/library/influxdb:latest
 $ journalctl -fu docker
-- Logs begin at Fri 2021-03-05 11:59:45 EST. --
Mar 05 15:37:48 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:48.944599273-05:00" level=debug msg="Applying tar in /var/lib/docker/overlay2/9e3ad0d2ab92d51ed7214f399d56b96ba5c13c94f750bd65ab1d37261592504a/diff" storage-driver=overlay2
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.146893428-05:00" level=debug msg="Applied tar sha256:7ef7ab563542d66d5c58c71c12d95652292d9e6e39d51a3f811a5aa53896cc2f to 9e3ad0d2ab92d51ed7214f399d56b96ba5c13c94f750bd65ab1d37261592504a, size: 0"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.342553334-05:00" level=debug msg="Using /usr/bin/unpigz to decompress"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.353787110-05:00" level=debug msg="Applying tar in /var/lib/docker/overlay2/e688297a996993147fc0cbacf9e2f096d31979f32691a0bcd527fc2c90c3373b/diff" storage-driver=overlay2
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.528390635-05:00" level=debug msg="Applied tar sha256:95da00a24c55336b4d45eb480b17eaea03b80d598f663ad4e0e9fb9c31a9000a to e688297a996993147fc0cbacf9e2f096d31979f32691a0bcd527fc2c90c3373b, size: 98"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.709004470-05:00" level=debug msg="Using /usr/bin/unpigz to decompress"
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.720995702-05:00" level=debug msg="Applying tar in /var/lib/docker/overlay2/9c4eb12b8f9f3f9c66b776f18e218e4c7a208bec602b9fc1b39d76c1a36e773a/diff" storage-driver=overlay2
Mar 05 15:37:49 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:49.896104870-05:00" level=debug msg="Applied tar sha256:8fed90b5e4a6a62e8fdb10a8f078242cea6ad6218e667500c29a9aa39f3dd3b8 to 9c4eb12b8f9f3f9c66b776f18e218e4c7a208bec602b9fc1b39d76c1a36e773a, size: 12229"
Mar 05 15:37:50 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:50.117550091-05:00" level=debug msg="Adding content digest to lease" digest="sha256:0b35e4d071ad0eef0771f7f0679276407a728b8388f98a2d3232f32179ed25fe" lease="moby-image-sha256:3186e07f4050523da24af0ff8e09695e1372807260fb00e96c9d40b04daaaa2c" remote="docker.io/library/influxdb:latest"
Mar 05 15:37:50 raspberrypi4 dockerd[16864]: time="2021-03-05T15:37:50.152887477-05:00" level=debug msg="Adding content digest to lease" digest="sha256:7418b74da890e76ef4b62152b0ce8f66b5191a067dd3d337ff5ba468acfdd9bd" lease="moby-image-sha256:3186e07f4050523da24af0ff8e09695e1372807260fb00e96c9d40b04daaaa2c" remote="docker.io/library/influxdb:latest"
^C
 $

[cpuguy83]

And is that the armv7 version instead of arm64?

[mvadu]

I am pulling this https://hub.docker.com/_/influxdb/ and it does not have a arm7 image. It has an arm64 version.

[cpuguy83]

And it works on 20.10.3?

[cpuguy83]

Also, what's the output of file $(which dockerd)

[mvadu]

that is my problem.. I did an apt update/upgrade last night, and I am not sure what version I was using before it..

file $(which dockerd)
/usr/bin/dockerd: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, Go BuildID=KWfZsgtA0v8ydK4uwzH9/W4Cc15Zw3VpW_aeXUmAA/wuC7GIfnzgPe7eXgTeJ_/aCn2OG9BBROphPzG3uIw, BuildID[sha1]=5b9deff00f2b995dcdbb844bc8724e155395238d, not stripped

[cpuguy83]

What's in /etc/apt/sources.list.d/docker.list?

[mvadu]

cat /etc/apt/sources.list.d/docker.list
deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable

[mvadu]

but if I do

docker run --rm  ubuntu uname -a
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
e0e04e4d364e: Pull complete
4c7c5587c59a: Pull complete
2e4f1be617f9: Pull complete
Digest: sha256:b4f9e18267eb98998f6130342baacaeb9553f136142d40959a1b46d6401f0f2b
Status: Downloaded newer image for ubuntu:latest
Linux c507964803a1 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 aarch64 aarch64 GNU/Linux

It pulls the aaarch64 image

[cpuguy83]

I'm not sure why armhf was detected there when the list was installed.
You can change that to arm64, assuming you have arm64 architecture enabled... dpkg --add-architecture arm64... but if its not enabled already there could be a reason, I'm not familiar with Raspbian.

It pulls the aaarch64 image

That information isn't in the image, its reading from /proc/cpuinfo, which is from the host.

[mvadu]

well, that file was generated from this command..

echo "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
     $(lsb_release -cs) stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list

if I change it to arm64 then doker-ce fails to install..

sudo apt-get install docker-ce
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package docker-ce is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'docker-ce' has no installation candidate

[mvadu]

In Raspbian with 64bit enabled, only kernel is 64bit, and all user land including docker-ce is 32 bit. But if we can pull an arm64 image, it runs fine.

[cpuguy83]

You'll need to dpkg --add-architecture arm64, most likely.... again I do not know what affect this will have on installing other packages, and you'll probably still need to tell it to install the arm64 version... e.g. apt-get install docker-ce:arm64

[mvadu]

I am thinking if this is related to moby/issues/41092. But i can swear influxdb was running until 2 days ago.. I still have its data files to prove it!!

[mvadu]

I finally found the issue. Last week Influx changed their image to target their 2.0 image, while it was 1.8x series before. If I fix the version to 1.8x, it works fine.

docker pull influxdb:1.8.4
1.8.4: Pulling from library/influxdb
5275c778c803: Pull complete
aae60b5ed65c: Pull complete
a99de29c3c44: Pull complete
266519cf240c: Pull complete
566dbce2ab06: Pull complete
ce0ef9b950cd: Pull complete
97b8013e08b5: Pull complete
206c65c8a4e2: Pull complete
Digest: sha256:a66fa6bd81bee3c70e6c408235af068422ab73d74aa5d6acfc09dc76eba0ebf7
Status: Downloaded newer image for influxdb:1.8.4
docker.io/library/influxdb:1.8.4

So it was not related to docker code issue.

[cpuguy83]

Well, you are getting the armhf version of the image because the version of Docker you have installed is compiled for armhf.
The error you were getting was because the image did not have an armhf variant and Docker assumed you want armhf since that is the target platform of the dockerd binary you have installed.