Wrong platform architecture on Docker Hub Images #735

shaderecker · 2020-12-22T00:13:55Z

Platform

OS and version: Debian Buster (I used that, but it also impacts stretch and multi-arch images)

Platform: Raspberry Pi

Expected behavior

The right platform architecture is assigned to the respective images.

Actual behavior / bug

The arch specific images are tagged with the wrong platform architecture.
Currently they all show amd64 as architecture regardless if they are specific arm or arm64 images.
Check for example the v5.2.1-armhf-buster
This armhf images wrongly displays amd64 as its architecture.

When using the image in my deployment it still works in the end, but Ansible complains with a warning that the image has the wrong architecture:

[WARNING]: Docker warning: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested

I also tried the multi-arch image (e.g. v5.2.1-buster) which shows at least all three architectures (amd64/arm/arm64) in Docker Hub, but this also gives the same warning. So even there, the arch detection is somewhat broken.

Can you please have a look into this. Thanks.

The text was updated successfully, but these errors were encountered:

dschaper · 2020-12-22T00:20:43Z

Does the issue still show with using the plain pihole/pihole:v5.2.1? There shouldn't be a need to use the buster specific variant.

shaderecker · 2020-12-22T00:28:12Z

Yes, I just tried the pihole/pihole:v5.2.1 tag. It has the same issue

dschaper · 2020-12-22T00:42:21Z

Hmm, on a Raspberry Pi I just tried:

dan@raspberrypi:~ $ docker run -it --rm --entrypoint pihole-FTL pihole/pihole:v5.2.1 -vv
Unable to find image 'pihole/pihole:v5.2.1' locally
v5.2.1: Pulling from pihole/pihole
9159b6bb9431: Pull complete
af6c343a3853: Pull complete
c26d8a26526d: Pull complete
a9605594d712: Pull complete
f45b672ae3e8: Pull complete
37b3444e8040: Pull complete
1292910fb137: Pull complete
a54bdd0520cd: Pull complete
Digest: sha256:7888e89f4d435673c75896c546fefe3177379c99c8766c89be25a905f8def533
Status: Downloaded newer image for pihole/pihole:v5.2.1
****************************** FTL **********************************
Version:         v5.3.2

****************************** dnsmasq ******************************
Version:         pi-hole-2.81
Compile options: IPv6 GNU-getopt no-DBus no-UBus no-i18n IDN DHCP DHCPv6 Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dumpfile

****************************** SQLite3 ******************************
Version:         3.31.1
Compile options: COMPILER=gcc-6.3.0 20170516 DEFAULT_FOREIGN_KEYS OMIT_DEPRECATED OMIT_LOAD_EXTENSION OMIT_MEMORYDB OMIT_PROGRESS_CALLBACK THREADSAFE=1
******************************** LUA ********************************
Lua 5.4.1  Copyright (C) 1994-2020 Lua.org, PUC-Rio

Edit:

dan@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux

shaderecker · 2020-12-22T01:01:53Z

Ok that is really weird, I ran the same command as you did and still got the error/warning:

pi@pihole-2:~ $ docker run -it --rm --entrypoint pihole-FTL pihole/pihole:v5.2.1 -vv
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
****************************** FTL **********************************
Version:         v5.3.2

****************************** dnsmasq ******************************
Version:         pi-hole-2.81
Compile options: IPv6 GNU-getopt no-DBus no-UBus no-i18n IDN DHCP DHCPv6 Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify dumpfile

****************************** SQLite3 ******************************
Version:         3.31.1
Compile options: COMPILER=gcc-6.3.0 20170516 DEFAULT_FOREIGN_KEYS OMIT_DEPRECATED OMIT_LOAD_EXTENSION OMIT_MEMORYDB OMIT_PROGRESS_CALLBACK THREADSAFE=1
******************************** LUA ********************************
Lua 5.4.1  Copyright (C) 1994-2020 Lua.org, PUC-Rio

pi@pihole-2:~ $ uname -a
Linux pihole-2 5.4.79-v7+ #1373 SMP Mon Nov 23 13:22:33 GMT 2020 armv7l GNU/Linux

Edit: I also tried removing all pihole containers and images from my raspberry and let it re-pull during the run command, but still the same warning.

dschaper · 2020-12-22T01:17:22Z

What does your docker images show?

dan@raspberrypi:~ $ docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
pihole/pihole                              v5.2.1              3803f333ac15        2 weeks ago         329MB

And try rming the image you have downloaded. The buster and stretch says the OS used to build the image, not the OS version that is required to run. You should just use the plain tag by itself.

Also, what does docker info show?

 Kernel Version: 5.4.72-v7l+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l

Edit: And for comparison:

 Server Version: 19.03.13

dschaper · 2020-12-22T01:22:53Z

And I think this came up before, but the way you install docker can cause problems. If this is Raspbian Buster then you can't apt install it:

https://docs.docker.com/engine/install/debian/#install-using-the-repository

shaderecker · 2020-12-22T01:24:19Z

Ok was too late with my previous edit :D
I also tried removing all pihole containers and images from my raspberry and let it re-pull during the run command, but still the same warning.

Other output:

pi@pihole-2:~ $ docker images
REPOSITORY      TAG       IMAGE ID       CREATED       SIZE
pihole/pihole   v5.2.1    3803f333ac15   2 weeks ago   329MB

pi@pihole-2:~ $ docker info | grep -A 3 Kernel
 Kernel Version: 5.4.79-v7+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l

Edit: Server version

Server Version: 20.10.1

shaderecker · 2020-12-22T01:30:09Z

Regarding the Docker installation on Raspbian: I am aware of the complications.
I deploy Docker with my Ansible role, which covers the official Debian Install guide: https://github.com/shaderecker/ansible-pihole/blob/master/roles/docker/tasks/main.yaml

dschaper · 2020-12-22T01:30:09Z

Looks like we have different kernel versions and docker versions.

What version of Raspberry Pi is the hardware, and what does

sudo apt-cache policy raspberrypi-kernel

show?

shaderecker · 2020-12-22T01:31:06Z

pi@pihole-2:~ $ sudo apt-cache policy raspberrypi-kernel
raspberrypi-kernel:
  Installed: 1.20201201-1
  Candidate: 1.20201201-1
  Version table:
 *** 1.20201201-1 500
        500 http://archive.raspberrypi.org/debian buster/main armhf Packages
        100 /var/lib/dpkg/status

shaderecker · 2020-12-22T01:34:16Z

The Raspberry Pi is a "Raspberry Pi 3 Model B Plus Rev 1.3"
Hardware : BCM2835
Revision : a020d3

dschaper · 2020-12-22T01:41:03Z

I see a similar report that mirrors what we are seeing. Docker 20 fails, docker 19 works:

portainer/portainer#4585

shaderecker · 2020-12-22T02:07:32Z

Hmm yes this seems to be the issue. I just downgraded to docker-ce/buster,now 5:19.03.14~3-0~raspbian-buster armhf and there it also works correctly.

shaderecker · 2020-12-22T04:02:16Z

Maybe another thing:
Regardless of the Docker version, the Image always reports back amd64 as its architecture.

pi@pihole-2:~ $ docker image inspect pihole/pihole:v5.2.1 | grep Arch
        "Architecture": "amd64",

Even the v5.2.1-armhf-buster image says "Architecture": "amd64"

I found moby/moby#36552 where the issue about wrong archs is being discussed for quite some time.

Might it be that the arch is wrongly declared during the image build process somehow?

dschaper · 2020-12-22T04:28:09Z

Might it be that the arch is wrongly declared during the image build process somehow?

If it is then its affecting multiple projects and thus I would consider that to be a problem with docker and not with the multiple projects that work fine in version 19.

dschaper · 2020-12-22T04:39:22Z

In re: that moby issue, it's rather clear that the Architecture config is from the environment that built the image. And indeed, we build the arm images on amd64 hardware. There's no real CI way to do otherwise without building our own CI runners on actual arm hardware.

shaderecker · 2020-12-22T18:33:43Z

I was trying to figure out the build process of the pihole containers but I am struggling with that.
How is it currently done?
I found this Dockerfile_build which seems to build the images in a own docker container. But not sure if this is still used? At least the building of said container fails since the link to the docker-compose binary on Github is broken.

The other thing I found were scripts for Github Actions, which I assume are used now.
Maybe some explanation/update of the build process would help here.

Another thing: I tried building an arm container on an amd64 machine with docker buildx, following this guide. And this worked successfully.
In short sth like (you can also directly build multiarch images):

docker buildx build -t "testimg:latest" --platform linux/armhf --load .

And then docker image inspect testimg:latest correctly says:

"Architecture": "arm",

And then docker correctly! complains about the image, since I tried to run it on my amd64 machine:

ubuntu@docker-build-host:~/testimg$ docker run -it --rm testimg:latest 
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
“Running on armv7l”

So maybe the image build process needs an improvement or update.
I am currently trying to build the pihole as an arm container, but this is more complicated since I am not fully understanding the current build process yet :D

dschaper · 2020-12-22T19:02:20Z

docker buildx is an experimental feature and not used in general release.

dschaper · 2020-12-22T19:06:40Z

@lightswitch05 Do you have any advice?

lightswitch05 · 2020-12-22T19:15:16Z

How the base images are built: https://github.com/pi-hole/docker-base-images/blob/master/tag-and-push.sh#L3-L6

Images are built and pushed with ./gh-actions-test.sh

gh-actions-deploy.sh then takes all the built images and builds the multi-arch manifest for it. Nothing is using buildx as far as I know.

dschaper · 2020-12-22T19:18:31Z

When it comes down to it, it works in 19, doesn't in 20. That tells me that docker broke something in 20. And again, there are multiple projects that are seeing this same issue so we're not a one-off.

If you want to improve the build process, then please do. But if it's a process that then works in 20 and doesn't work in 19 we're in the same boat.

I'd wait to see what docker says as far as why 20 doesn't work.

shaderecker · 2020-12-23T15:15:35Z

Thank you two for your input.
Discussion in docker/for-linux#1170 is ongoing. Let's see what comes out of it first.

thaJeztah · 2020-12-23T16:23:29Z

Did some further digging, and I suspect these images are cross-compiled on a amd64 (x86) machine, and have the incorrect architecture set in the image manifest (docker/for-linux#1170 (comment))

docker/for-linux#1170 (comment)

So concluding;

looks like for this case, the issue is actually with the images having the incorrect architecture set in their metadata

docker 20.10 now prints a warning ("WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested"). Older versions of docker silently tried to run the image (which in this case would work, because the image is an arm image, just "labeled" incorrectly

docker 20.10 now produces an error if the architecture of the image isn't supported by the engine (which it bases on the architecture specified in the image's manifest)

Perhaps a "I know what I'm doing; ignore what the image says" flag would be needed to allow running existing invalid images?

thaJeztah · 2020-12-23T21:31:55Z

Slightly more detail under what circumstances it fails (due to the image being invalid / having incorrect os/architecture in the metadata); docker/for-linux#1170 (comment)

thaJeztah · 2020-12-23T21:46:12Z

I think the correct fix for this would be to use buildx to do the cross-compile, instead of the

docker-pi-hole/gh-actions-test.sh

Lines 18 to 28 in a779c7f

    
           # generate and build dockerfile 
        
           docker build --tag image_pipenv --file Dockerfile_build . 
        
           docker run --rm \ 
        
               --volume /var/run/docker.sock:/var/run/docker.sock \ 
        
               --volume "$(pwd):/$(pwd)" \ 
        
               --workdir "$(pwd)" \ 
        
               --env PIPENV_CACHE_DIR="$(pwd)/.pipenv" \ 
        
               --env ARCH="${ARCH}" \ 
        
               --env ARCH_IMAGE="${ARCH_IMAGE}" \ 
        
               --env DEBIAN_VERSION="${DEBIAN_VERSION}" \ 
        
               ${enter} image_pipenv

With docker/build-push-action@v2 it should be possible to build (and push) a multi-arch image in oneusing buildx (and no need to manually set up qemu);

See the example here; https://github.com/docker/build-push-action#multi-platform-image;

Multi-platform image

name: ci

on:
  push:
    branches: master

jobs:
  multi:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Login to DockerHub
        uses: docker/login-action@v1 
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./Dockerfile
          platforms: linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x
          push: true
          tags: |
            user/app:latest
            user/app:1.0.0

but perhaps @crazy-max or @tonistiigi has additional hints

dschaper · 2020-12-23T21:51:21Z

I think the correct fix for this would be to use buildx to do the cross-compile

Happy to investigate this. As long as it's a backwards compatible fix and won't bork the installs still on 19.

thaJeztah · 2020-12-23T21:55:48Z

It should be backward compatible; The existing images are technically invalid, and the reason it worked on docker 19.03 (and before) was that it happened to fully ignore the architecture in the image-metadata once the image was pulled.

dschaper · 2020-12-23T21:59:03Z

Thanks! Will review.

thaJeztah · 2021-01-29T15:21:53Z

Basically labels will be created at build time through something like this action to reflect repo metadata.

Ah, right; in this case I'm not sure that would be the desired info, as this repository only contains the scripts to build the Dockerfile, not the source-code 🤔 (at least, I somewhat would expect the image to contain info about pihole itself)

crazy-max · 2021-01-29T16:05:18Z

@thaJeztah Yes sure you can overwrite labels too.

PromoFaux · 2021-01-29T16:27:45Z

So I started again and made a new image from scratch but the study of the basic-install.sh script is complicated to take into account.

Ahem, yes that script has grown quite organically over the years - I like to think I know it like the back of my hand - but appreciate that others get lost in it, If you have any questions....

At some point, we're going to be looking at really paring down basic-install - especially as we move closer toward things being in a a single binary (e.g upcoming 6.0 will no longer require lighttpd or php to be installed as it will ship with an embedded web server - and also RESTful API) So there will be lots that the install script is doing that can essentially go away at some point.

dschaper · 2021-01-29T17:54:03Z

Thanks for the help everyone!

Before things get too far, musl and pihole-FTL are not friendly with each other. pihole-FTL is a statically linked binary and doesn't do well on Alpine.

Edit: I guess the issues with the musl builds are just for aarch64/arm. I see there's a stable x86 musl build but I'm not sure that helps with multiple platform arch's as the goal? https://github.com/pi-hole/FTL/releases/tag/v5.6

dschaper · 2021-02-08T02:36:06Z

@DL6ER Any thoughts on musl?

DL6ER · 2021-02-14T08:17:53Z

Sorry for the late reply. I'm back in operation now.

FTL is designed to be able to be fully statically linked (including the crypto library, LUA and everything else). This is reflected by the architecture tests each CI build has to comply with to pass the tests for this commit. These are said tests for musl on x86_64:

elif [[ "${CIRCLE_JOB}" == "x86_64-musl" ]]; then

  check_machine "ELF64" "Advanced Micro Devices X86-64"
  check_static # Binary should not rely on any dynamic interpreter
  check_libs "" # No dependency on any shared library is intended
  check_file "ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped"

If there would be a single dynamic dependency, this test would fail and the binary would be rejected.

The reason for a missing pre-compiled musl-arm/aarch64 binary is rather that nobody ever asked for it :-)
I vaguely recall having been asked to add the the current x86_64-musl CI build specifically for docker somewhen in 2019.

So I'm not exactly sure what the question is here, @crazy-max do you see any issues (c)makeing FTL ?

guodajun · 2021-06-04T06:07:53Z

sudo docker run -it --platform=linux/arm64 /bin/bash

pralor-bot · 2021-06-07T18:57:09Z

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/platform-mismatch-with-docker-on-rpi4/47523/1

dschaper · 2021-06-14T19:01:10Z

If anyone at all can help with this, please help!

jorgeml · 2021-06-17T16:50:01Z

This is causing issues with Podman 3.2 as reported here: containers/podman#10682

PromoFaux · 2021-09-12T22:09:32Z

Hullo all, sorry for taking such a long time to get around to this... it should now be fixed in the latest :dev tag. Can you please check it out and report back?

shaderecker · 2021-09-13T19:44:34Z

Hi @PromoFaux, this looks really good now.
I just tested the dev tag on my Raspberry Pi and I got the correct arch from the image:

        "Architecture": "arm",
        "Variant": "v7",

Thank you for your work! 😀

PromoFaux · 2021-09-13T19:52:31Z

Good news, then my entire day yesterday was not wasted! (So even better news 😜)

Will hopefully will through with the next main tag/release currently in the process of checking which arch's build ( on branch this-will-break)

shaderecker · 2021-10-01T09:54:40Z

I tried the new image pihole/pihole:2021.10 and this works like a charm 😀
Thank you again for making that happen @PromoFaux!
I will close this now.

PromoFaux · 2021-10-01T11:35:33Z

There are probably more improvements that can come in time with this, but just want to say thanks to @crazy-max and @thaJeztah for their hints and pointers above. buildx is definitely the way forward!

thaJeztah · 2021-10-01T12:09:49Z

Thanks! Happy to see it works well for you. If you have feedback to either the buildx cli, buildkit (which does the actual build), or the github actions; feel free to open tickets there (I linked the repositories) 🤗

shaderecker mentioned this issue Dec 22, 2020

[v20.10] Platform architecture detection is broken docker/for-linux#1170

Closed

3 tasks

dschaper assigned lightswitch05 Dec 23, 2020

PromoFaux mentioned this issue Jan 31, 2021

Docker manifest for the arm64 architecture is wrong #780

Closed

billz mentioned this issue Feb 3, 2021

Feature request -- official docker container RaspAP/raspap-webgui#458

Closed

dschaper mentioned this issue Feb 16, 2021

PiHole on docker won't start anymore with the latest version #794

Closed

uhsa779 mentioned this issue Feb 25, 2021

Portainer container Pi-Hole seems to crash every 12 odd hours #805

Closed

1 task

dschaper mentioned this issue Feb 27, 2021

Set PUID and PGID as Variables #328

Open

7 tasks

dschaper added the Help Wanted label Jun 14, 2021

dschaper pinned this issue Jun 14, 2021

PromoFaux closed this as completed in 7624fc3 Sep 15, 2021

PromoFaux reopened this Sep 15, 2021

shaderecker closed this as completed Oct 1, 2021

PromoFaux unpinned this issue Oct 1, 2021

Baenwort mentioned this issue Dec 3, 2021

Docker warning - image platform doesn't match host platform datarhei/restreamer#299

Closed

ShahanaFarooqui mentioned this issue Feb 9, 2022

Wrong architecture on docker arm32v7 image Ride-The-Lightning/RTL#940

Closed

Wrong platform architecture on Docker Hub Images #735

Wrong platform architecture on Docker Hub Images #735

Comments

shaderecker commented Dec 22, 2020

Platform

Expected behavior

Actual behavior / bug

dschaper commented Dec 22, 2020

shaderecker commented Dec 22, 2020

dschaper commented Dec 22, 2020 • edited Loading

shaderecker commented Dec 22, 2020 • edited Loading

dschaper commented Dec 22, 2020 • edited Loading

dschaper commented Dec 22, 2020

shaderecker commented Dec 22, 2020 • edited Loading

shaderecker commented Dec 22, 2020

dschaper commented Dec 22, 2020

shaderecker commented Dec 22, 2020

shaderecker commented Dec 22, 2020 • edited Loading

dschaper commented Dec 22, 2020

shaderecker commented Dec 22, 2020

shaderecker commented Dec 22, 2020

dschaper commented Dec 22, 2020

dschaper commented Dec 22, 2020

shaderecker commented Dec 22, 2020

dschaper commented Dec 22, 2020

dschaper commented Dec 22, 2020

lightswitch05 commented Dec 22, 2020 • edited Loading

dschaper commented Dec 22, 2020

shaderecker commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

Multi-platform image

dschaper commented Dec 23, 2020

thaJeztah commented Dec 23, 2020

dschaper commented Dec 23, 2020

thaJeztah commented Jan 29, 2021

crazy-max commented Jan 29, 2021

PromoFaux commented Jan 29, 2021

dschaper commented Jan 29, 2021 • edited Loading

dschaper commented Feb 8, 2021

DL6ER commented Feb 14, 2021 • edited Loading

guodajun commented Jun 4, 2021

pralor-bot commented Jun 7, 2021

dschaper commented Jun 14, 2021

jorgeml commented Jun 17, 2021

PromoFaux commented Sep 12, 2021

shaderecker commented Sep 13, 2021

PromoFaux commented Sep 13, 2021

shaderecker commented Oct 1, 2021

PromoFaux commented Oct 1, 2021

thaJeztah commented Oct 1, 2021

dschaper commented Dec 22, 2020 •

edited

Loading

shaderecker commented Dec 22, 2020 •

edited

Loading

dschaper commented Dec 22, 2020 •

edited

Loading

shaderecker commented Dec 22, 2020 •

edited

Loading

shaderecker commented Dec 22, 2020 •

edited

Loading

lightswitch05 commented Dec 22, 2020 •

edited

Loading

dschaper commented Jan 29, 2021 •

edited

Loading

DL6ER commented Feb 14, 2021 •

edited

Loading