failed running [/dev/.buildkit_qemu_emulator /bin/sh -c which ls]: exit code: 127

[Rory-Z]

First of all thank you for your work, buildx is very important to me

Today I found that I kept failing when trying to build an arm64 images on x86 machine, so I tried to make a simple enough image

FROM debian:10

RUN which ls

Then I run buildx

docker buildx build -t test --platform=linux/arm64 .

I got the following error

WARN[0000] invalid non-bool value for BUILDX_NO_DEFAULT_LOAD:
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 0.8s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 31B                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/debian:10                                                                                                                                           0.6s
 => CACHED [1/2] FROM docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                              0.0s
 => => resolve docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                                     0.0s
 => ERROR [2/2] RUN which ls                                                                                                                                                                           0.2s
------
 > [2/2] RUN which ls:
#5 0.152 /bin/sh: 0: Can't open which
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c which ls]: exit code: 127

Hope your reply

[Rory-Z]

$ docker info
Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-tp-docker)

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 163
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1024-aws
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.5GiB
 Name: ip-172-31-53-38
 ID: 4WOH:WGOY:YGTK:DWP3:KRRR:U3P2:MND6:26B6:VZHG:XOUU:DYMC:L4ZM
 Docker Root Dir: /mnt/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  34.223.40.33:5000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

$ docker buildx ls
NAME/NODE   DRIVER/ENDPOINT             STATUS   PLATFORMS
mybuildx *  docker-container
  mybuildx0 unix:///var/run/docker.sock running  linux/amd64*, linux/arm64*, linux/arm/v7*, linux/386*

[tonistiigi]

docker run -it --rm --privileged tonistiigi/binfmt --install all to install qemu emulators

[Rory-Z]

Hi，@tonistiigi Thanks for your reply, I have installed qemu emulators, but the problem still exists

$ docker run -it --rm --privileged tonistiigi/binfmt --install all
2020/12/08 15:09:43 installing: arm64 qemu-aarch64 already registered
2020/12/08 15:09:43 installing: arm qemu-arm already registered
2020/12/08 15:09:43 installing: s390x qemu-s390x already registered
2020/12/08 15:09:43 installing: ppc64le qemu-ppc64le already registered
2020/12/08 15:09:43 installing: riscv64 qemu-riscv64 already registered
{
  "supported": [
    "linux/amd64",
    "linux/386"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-aarch64_be",
    "qemu-alpha",
    "qemu-arm",
    "qemu-armeb",
    "qemu-hppa",
    "qemu-m68k",
    "qemu-microblaze",
    "qemu-microblazeel",
    "qemu-mips",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-mipsel",
    "qemu-mipsn32",
    "qemu-mipsn32el",
    "qemu-or1k",
    "qemu-ppc",
    "qemu-ppc64",
    "qemu-ppc64le",
    "qemu-riscv32",
    "qemu-riscv64",
    "qemu-s390x",
    "qemu-sh4",
    "qemu-sh4eb",
    "qemu-sparc",
    "qemu-sparc32plus",
    "qemu-sparc64",
    "qemu-xtensa",
    "qemu-xtensaeb"
  ]
}

$ docker buildx build -t test --platform=linux/arm64 .
WARN[0000] invalid non-bool value for BUILDX_NO_DEFAULT_LOAD:
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 1.4s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 66B                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/debian:10                                                                                                                                           1.1s
 => CACHED [1/2] FROM docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                              0.0s
 => => resolve docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                                     0.0s
 => ERROR [2/2] RUN which ls                                                                                                                                                                           0.2s
------
 > [2/2] RUN which ls:
#5 0.167 /bin/sh: 0: Can't open which
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c which ls]: exit code: 127

[tonistiigi]

what is the output of buildx ls after installing emulators? In the output in the first post emulators are not installed.

You get this error because you manually set --platform arm64 on buildx create while the node didn't actually have arm64 support. You don't need to set it unless you want to change the priority of nodes for multi-node build. If you don't set it supported platforms are detected automatically. If you see arm64 in the list then all should be ok.

[Rory-Z]

@tonistiigi

$ docker buildx ls
NAME/NODE   DRIVER/ENDPOINT             STATUS   PLATFORMS
mybuild     docker-container
  mybuild0  unix:///var/run/docker.sock inactive
mybuildx *  docker-container
  mybuildx0 unix:///var/run/docker.sock running  linux/amd64*, linux/arm64*, linux/arm/v7*, linux/386*
test        docker-container
  test0     unix:///var/run/docker.sock running  linux/amd64, linux/386
default     docker
  default   default                     running  linux/amd64, linux/386

My buildx has been working normally on my server and Github Actions until they failed two days ago. I tried to reproduce it on the server and found that it also failed. This puzzled me. I did not upgrade docker and I did not change the code, and then I found that even if I build an image of RUN which ls, it will fail, so I think I need your team’s advice

[tonistiigi]

Sorry, I misread the above output.

$ docker run -it --rm --privileged tonistiigi/binfmt --install all
2020/12/08 15:09:43 installing: arm64 qemu-aarch64 already registered
2020/12/08 15:09:43 installing: arm qemu-arm already registered
2020/12/08 15:09:43 installing: s390x qemu-s390x already registered
2020/12/08 15:09:43 installing: ppc64le qemu-ppc64le already registered
2020/12/08 15:09:43 installing: riscv64 qemu-riscv64 already registered
{
  "supported": [
    "linux/amd64",
    "linux/386"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-aarch64_be",
    "qemu-alpha",
    "qemu-arm",
    "qemu-armeb",

This means that your host is configured with emulator support outside of tonistiigi/binfmt image and it isn't working correctly. tonistiigi/binfmt can't install emulators as the names are already used arm64 qemu-aarch64 already registered but there is actually no arm64 support in your system as supported status only shows amd64/386. This usually happens if qemu is installed in a way that it is only accessible from host and not from containers. I'd recommend you to uninstall the package that did this and use tonistiigi/binfmt image instead. It is possible that --uninstall option https://github.com/tonistiigi/binfmt#uninstalling-emulators also works.

[Rory-Z]

@tonistiigi
Great！it works！thx