balena push selects incorrect image (multiarch images, Docker manifest)

[pdcastro]

As reported in the forums (with thanks!), running balena push with a multiarch image hosted in Docker Hub, with a docker-compose file such as the following, results in "exec format error":

services:
  lora-connect:
    image: arribada/lora-connect
    ports:
    - "8070:8070"

The problem may not actually be in balena-cli, but rather the cloud builders, but this issue is being created to give more visibility to the issue and keep track of its resolution.

[pdcastro]

Link to a related issue in a private GitHub repo (access to balena users only):
https://github.com/balena-io/resin-builder/issues/626

[balena-ci]

[pdcastro] This issue has attached support thread https://jel.ly.fish/#/support-thread~764598c2-13cf-40f1-bb89-931b61ba587c

[krasi-georgiev]

@pdcastro sorry for the noise, but this is a blocker for a project that is suppose to go in production soon so wondering if there has been any development towards fixing this bug. Some update would be greatly appreciated.

[pdcastro]

@krasi-georgiev, I'm afraid there hasn't been activity on this issue as far as I am aware. If helpful, note that balena supports Dockerfile templates to allow a single Dockerfile to pull from base images of different architectures:
https://www.balena.io/docs/learn/develop/dockerfile/#dockerfile-templates

That's how balena push has "traditionally" dealt with multiple architectures. I understand that multiarch image support based on manifest files is a relatively new feature introduced by Docker. We intend to add support for it, but at the moment I don't have an ETA to share.

[krasi-georgiev]

Thanks.

That's how balena push has "traditionally" dealt with multiple architectures.

I might be wrong, but I think this feature has been in docker since one of the early versions so it is not new. Until recently though it wasn't very easy to push multi arch images and this changed with one of the recent versions which simplifies that process.

[MatthewCroughan]

#1508

[jellyfish-bot]

[shaunmulligan] This issue has attached support thread https://jel.ly.fish/#/b6953f28-5fd2-432d-9e52-1813e56652c4

[krasi-georgiev]

it would be useful to get some automated updates from the internal discussions and decisions as well

[jellyfish-bot]

[pdcastro] This issue has attached support thread https://jel.ly.fish/73473ac9-0b86-4f3c-a950-279c6ebf9753

[krasi-georgiev]

Has there been any progress with this.

I just tried it again with 12.20.0 and I still get

.
.
[Success]          Successfully uploaded images
[Error]            Some services failed to build:
[Error]              Service: lora-gps-server
[Error]                Error: no matching manifest for unknown in the manifest list entries
[Info]             Built on arm01
[Error]            Not deploying release.

The docker image does have a manifest for arm and when I specify it with the correct SHA it does work as expected.

    image: arribada/lora-gps-server@sha256:ae97f057dcbf2211554eea0dc477528cd5ad06669ab7f4c4cba0d767cb671485

[krasi-georgiev]

I just tried 12.33.0 and seems to be ok so can close it, will reopen it if still facing issues.

[pdcastro]

I think livepush / push to local device (balena push <ip-address>) still doesn't work with multiarch images, but push to cloud (balena push myApp) works.

[krasi-georgiev]

noted, I was using push to cloud

[maggie44]

I think livepush / push to local device (balena push <ip-address>) still doesn't work with multiarch images, but push to cloud (balena push myApp) works.

How about ‘deploy’ for openBalena? If I specify an image from Docker Hub, e.g. linuxserver/bookstack, will it pull the image for the arch it is intending to deploy to (arm 7) or pull for my local env (amd64)?

[compiaffe]

balena deploy using a multiarch still does not work.
i.e. docker-compose.yml as i.e.:

version: '2.1'
volumes:
  workdir:
services:
  gha-runner:
    image: myoung34/github-runner:latest
    environment:
      - RUNNER_NAME_PREFIX=nano
      - RUNNER_WORKDIR=/tmp/workdir
      - ORG_RUNNER=true
      - ORG_NAME=our_org
      - LABELS=linux-armv8-l4t-t210
    devices:
      - '/dev:/dev'
      - '/sys:/sys'
    privileged: true
    restart: "unless-stopped"
    volumes:
      - workdir:/tmp/workdir:rw

    # command: ["tail","-f", "/dev/null"] # for debugging
    labels:
      io.balena.features.balena-socket: '1'

still causes a exec error.

The target is a Jetson Nano and there are arm64 builds inside the multiarch

[pdcastro]

@Maggie0002, @compiaffe, thank you for reporting it. I can confirm that multiarch images are still not supported with balena build and balena deploy (and thus openBalena). I've had an open PR (#2145) for a while that was almost merged to solve this, but then Docker v20.10.x was released with a new "feature" (or bug, depending on one's point of view) :-) that would have broken balena deploy for all balenalib base images if that PR had been merged. We are having internal discussions in balena and discussions with Docker on the way forward. Public references:

• [v20.10.5] Inconsistent behavior of --platform flag results in build failure moby/moby#42158
• https://github.com/moby/moby/pull/42045/files
• https://github.com/moby/moby/pull/41955/files
• [v20.10] Platform architecture detection is broken docker/for-linux#1170
• Images on Docker Hub have incorrect architecture (amd64) balena-io-library/resin-rpi-raspbian#104

If I specify an image from Docker Hub, e.g. linuxserver/bookstack, will it pull the image for the arch it is intending to deploy to (arm 7) or pull for my local env (amd64)?

Currently (as you will have found by now, sorry for the delay in replying), balena build and balena deploy (but not balena push myApp) will pull an image for the CPU architecture of the machine where the Docker Engine is running (most often amd64) (assuming it is a multiarch image). This is a defect, as the correct behavior (already implemented by balena push myApp) would be for the CLI / Docker to pull the image for the architecture of the balena application (e.g. arm v7).

[pdcastro]

Until the issue is resolved for balena build and balena deploy, a workaround is described below -- it's a copy and paste of the original post in the balena forums.

The following example regards the multiarch telegraf image.

[...] the workaround is to append a sha256 digest to the FROM line of your Dockerfile, thus "manually selecting" the base image architecture. To do so, check the different sha256 digest for each available architecture on the Dockerhub page:

https://hub.docker.com/layers/telegraf/library/telegraf/1.15.3/images/sha256-655213a2041fb9eed5e8129baac75bc460005fb1f4ead6d7f3ddafd4bc884a89?context=explore

Dockerhub arch	Device	Dockerfile FROM line
arm64/v8	RPi 4	FROM telegraf:1.15.3@sha256:28fb4583d35fa39690bf84934165d41bb7f8d62686490fc87e149a5261abd6b7
arm/v7	RPi 3	FROM telegraf:1.15.3@sha256:f0665c76213e129ceba8d76443011ea8332d850ebc3749577cd498f141bed7d8
amd64	Intel NUC	FROM telegraf:1.15.3@sha256:655213a2041fb9eed5e8129baac75bc460005fb1f4ead6d7f3ddafd4bc884a89

Choose one of the Dockerfile FROM lines from the table above, depending on your device type. The sha256 digests above were extracted from the Dockerhub page linked above, by using the dropdown box to select the target architecture.

[compiaffe]

@pdcastro thanks for the clarification, looking forward to the fix, but the workaround is clear.

[peteretep]

any updates on this? It would really simplify deployment in one of my projects.

[rodmaz]

Any progress on this? This is also important for us.

[pdcastro]

@peteretep, @rodmaz, thanks for chasing this. Support for multiarch base images for balena build and balena deploy --build is still on our radar, but there hasn't been recent progress. I've raised it again with the balena team.

To reduce confusion, I am going to follow @krasi-georgiev's suggestion and close this particular issue in favour of two other existing issues:

• balena build / deploy --build selects base image of wrong architecture (multiarch image manifest) #1508 balena build / deploy --build selects base image of wrong architecture (multiarch image manifest)
• Push to local device (IP address) fails with FROM --platform #1889 Push to local device (IP address) fails with FROM --platform

... because this issue was originally reported for balena push myApp, which was resolved a while ago:

Command	Multiarch base image support
balena push <myApp>	Supported - this issue
balena push <ipAddress>	Not supported, mainly issue #1889
balena build	Not supported, issue #1508
balena deploy --build	Not supported, issue #1508