Using the "--platform" "--arch" options when running podman run (or build) causes the values you set to become the defaults when not specified next time you build/run

[jvullo]

/kind bug

My understanding from the reference is the when --platform or --arch arguments is that when not specified commands such as podman build/podman run should default to the architecture of the build host, however my experience is that when not specified, the last value used will become the default.

Note: I have not tested the --os and --variant options to see if they have the same problem.

Here is the documentation I read before making raising an issue: https://docs.podman.io/en/latest/markdown/podman-build.1.html

Excerpts:

--arch=arch
Set the architecture of the image to be built, and that of the base image to be pulled, if the build uses one, to the provided value instead of using the architecture of the build host. (Examples: arm, arm64, 386, amd64, ppc64le, s390x)

--platform=os/arch[/variant][,…]
Set the os/arch of the built image (and its base image, if your build uses one) to the provided value instead of using the current operating system and architecture of the host
....

And on this page: https://docs.podman.io/en/latest/markdown/podman-run.1.html

Excerpts:

--arch=ARCH
Override the architecture, defaults to hosts, of the image to be pulled. For example, arm.

--platform=OS/ARCH
Specify the platform for selecting the image. (Conflicts with --arch and --os) The --platform option can be used to override the current architecture and operating system.

Steps to reproduce the issue:

I have experienced this problem with both podman build & podman run, I have also tested on two seperate m1 macs with the same behaviour.

Here is a command to repeat the behaviour:

podman run -q --arch=arm64 ubuntu uname -m && \
    podman run -q ubuntu uname -m && \
    podman run -q --arch=amd64 ubuntu uname -m && \
    podman run -q ubuntu uname -m

Describe the results you received:

My output:

aarch64
aarch64
x86_64
x86_64

Describe the results you expected:

That the architecture is set back to the my host, which in my case should be aarch64.

aarch64
aarch64
x86_64
aarch64

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman --version 
podman version 4.0.3

Output of podman info:

% podman info 
host:
  arch: arm64
  buildahVersion: 1.24.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 5.15.18-200.fc35.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 5443239936
  memTotal: 6140002304
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.aarch64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 40m 6.52s
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 57
    paused: 0
    running: 0
    stopped: 57
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.aarch64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 66
  runRoot: /run/user/501/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.2
  Built: 1646319530
  BuiltTime: Thu Mar  3 14:58:50 2022
  GitCommit: ""
  GoVersion: go1.16.14
  OsArch: linux/arm64
  Version: 4.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

rpm -q podman
podman-4.0.2-1.fc35.aarch64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No & Yes

I am unable to upgrade my podman instance at the moment, not do I have anything spare to hand for more testing.

However I did confirm the release notes for 4.1 make no mention of this: https://github.com/containers/podman/releases/tag/v4.1.0

Additional environment details (AWS, VirtualBox, physical, etc.):

Using an m1 mac

[jvullo]

For a quick/rough workaround for me, I added the following into my ~/.bash_profile file

# Not a trailing space in value causes the next word to be checked for alias substitution when the alias is expanded.
alias podman='podman '
alias build="build --arch=arm64"
alias run="run --arch=arm64"

[mheon]

@flouthoc PTAL

[rhatdan]

I think this is more about libimage then it is about buildah.
@vrothberg PTAL

I think Podman is just grabbing the local image and if it is a different Arch using it.

[vrothberg]

I am on a train so I cannot check but please try with a more recent version. There were a number of changes for resolving images locally.

…

On Sat 13 Aug 2022 at 12:49, Daniel J Walsh ***@***.***> wrote: I think this is more about libimage then it is about buildah. @vrothberg <https://github.com/vrothberg> PTAL I think Podman is just grabbing the local image and if it is a different Arch using it. — Reply to this email directly, view it on GitHub <#15300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZDRAYN556YQ54PVQMYRQDVY54TFANCNFSM56M2GSVQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jvullo]

I can upgrade my podman from Wednesday and retest - will update once I've done it.

[joe-vullo-ibm]

Hi

I've upgraded and confirmed the issue persists

podman --version
podman version 4.2.0

podman run -q --arch=arm64 ubuntu uname -m && \
    podman run -q ubuntu uname -m && \
    podman run -q --arch=amd64 ubuntu uname -m && \
    podman run -q ubuntu uname -m

aarch64
aarch64
x86_64
x86_64

[flouthoc]

I think a nice solution to this is pre-populate defaults for --platform and --arch to host machine's platform and arch, this will force buildah to re-pull the image if a different platform/arch is present. I think this change can happen at buildah end. WDYT @vrothberg ?

[flouthoc]

I think a nice solution to this is pre-populate defaults for --platform and --arch to host machine's platform and arch, this will force buildah to re-pull the image if a different platform/arch is present. I think this change can happen at buildah end. WDYT @vrothberg ?

@containers/podman-maintainers WDYT ? If everyone agrees we can have a fix at buildah end for this.

[vrothberg]

Back from PTO:

Let's take a step back. I am sure there is some historical background for this behavior. Multiarch is painful and it usually boils down to exchanging one problem for another. I'll take a look at the history and report back.

[flouthoc]

Back from PTO:

Let's take a step back. I am sure there is some historical background for this behavior. Multiarch is painful and it usually boils down to exchanging one problem for another. I'll take a look at history and report back.

@vrothberg Ackd.

[vrothberg]

The actual output with Podman v4.2 looks as follows:

aarch64
WARN[0000] image platform ({arm64 linux  [] v8}) does not match the expected platform ({amd64 linux  [] })
aarch64
x86_64
x86_64

There's more background in #12682. In short, we need to keep the current behavior to remain compatible with Docker and not break existing users.

I suggest to close the issue.

[jvullo]

Hi @vrothberg

Thanks for getting back to me and I apologise for raising a duplicate issue, I promise I did have search and couldn't find the one you linked to.

There's a couple of things I want to mention:

• Missing image platform warning
• Differences in the behaviour from Docker
• Podman docs

Missing image platform warning

I've tested on three different macs, all running podman --version 4.2.0 and none produced the warning you have shown above.

If it matters, two were apple silicon macs, but one was an older intel based mac.

I'm happy to jump on call with you and evidence this if needed.

I've tried with and without the "-q" option but can't see the warning being generated.

Here is the full output:

podman run --arch=arm64 ubuntu uname -m && \ 
    podman run ubuntu uname -m && \ 
    podman run --arch=amd64 ubuntu uname -m && \ 
    podman run ubuntu uname -m 

Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob sha256:4a3049d340b7d3651a954fd25a32c42feb1086889d6287e2f15468aef865c5c4
Copying config sha256:bad148f8963fb9be6c8c260ce8a65aadd1cdfdd95e5bd16867d0f987cd7ebff3
Writing manifest to image destination
Storing signatures
aarch64
aarch64
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob sha256:d19f32bd9e4106d487f1a703fc2f09c8edadd92db4405d477978e8e466ab290d
Copying config sha256:df5de72bdb3b711aba4eca685b1f42c722cc8a1837ed3fbd548a9282af2d836d
Writing manifest to image destination
Storing signatures
x86_64

Differences in the behaviour from Docker

Podman build + print the image arch:

podman build --no-cache -q  --platform=linux/arm64 -t test-app:1 . && \
    podman image inspect test-app:1 --format '{{ .Os }}/{{ .Architecture }}' && \
    podman build --no-cache -q -t test-app:2 . && \
    podman image inspect test-app:2 --format '{{ .Os }}/{{ .Architecture }}' && \
    podman build --no-cache -q  --platform=linux/amd64 -t test-app:3 . && \
    podman image inspect test-app:3 --format '{{ .Os }}/{{ .Architecture }}' && \
    podman build --no-cache -q -t test-app:4 . && \
    podman image inspect test-app:4 --format '{{ .Os }}/{{ .Architecture }}'

Here is the output:

d19d34c52760290386c12e61a981b335da7da9d58d3e0ba2f2ad35f5c5768391
linux/arm64
d19d34c52760290386c12e61a981b335da7da9d58d3e0ba2f2ad35f5c5768391
linux/arm64
2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
linux/amd64
2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64** # Would expect this to be default to host "arm64"

Which is same as discussed previously.

Running the same command using docker has a different result, the image first pulled will become the default:

Please note docker commands I had to run this on amd64 machine, so I've reordered the command slightly

Command

 docker build --no-cache -q --platform=linux/amd64 -t test-app:1 . && \
    docker image inspect test-app:1 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker build --no-cache -q -t test-app:2 . && \
    docker image inspect test-app:2 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker build --no-cache -q  --platform=linux/arm64 -t test-app:3 . && \
    docker image inspect test-app:3 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker build --no-cache -q -t test-app:4 . && \
    docker image inspect test-app:4 --format '{{ .Os }}/{{ .Architecture }}'

Output is always "amd64"

sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**

This command is mostly same as the above but I have added a "image prune" to remove the image after each build to ensure the argument is used:

docker build --no-cache -q --platform=linux/amd64 -t test-app:1 . && \
    docker image inspect test-app:1 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker image prune -f --all > /dev/null && \
    docker build --no-cache -q -t test-app:2 . && \
    docker image inspect test-app:2 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker image prune -f  --all > /dev/null && \
    docker build --no-cache -q  --platform=linux/arm64 -t test-app:3 . && \
    docker image inspect test-app:3 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker image prune -f  --all > /dev/null && \
    docker build --no-cache -q -t test-app:4 . && \
    docker image inspect test-app:4 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker image prune -f  --all > /dev/null

Now the output is what I would have expected, defaulting to the operating system version when not specified:

sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
linux/amd64
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
linux/amd64
sha256:d19d34c52760290386c12e61a981b335da7da9d58d3e0ba2f2ad35f5c5768391
linux/arm64
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64** # <--- here it has defaulted to the os, as I would have expected. 

Note that I ran the above on a amd64 based machine with the following version of docker installed:

docker --version
Docker version 20.10.10, build b485636

So more of any FYI really, but the behaviour does match what is happening with docker either. Docker's behaviour here actually seems a worse user experience to me personally.

Podman Docs

I appreciate you don't want to make a breaking change and have added the warning as a solution but it would be nice if the documentation could be updated to reflect this behaviour.

i.e:

• https://docs.podman.io/en/latest/markdown/podman-build.1.html
• https://docs.podman.io/en/latest/markdown/podman-run.1.html

At the very least you can point issue reporters back the documentation in the future.

[vrothberg]

I've tested on three different macs, all running podman --version 4.2.0 and none produced the warning you have shown above.

That is a good point. These warnings won't show up on a remote client at the moment.

I will look into what we can do to improve the user experience. Thanks a lot for pointing that out!

[vrothberg]

That is a good point. These warnings won't show up on a remote client at the moment.

Actually they do show up on the remote client starting with Podman v4.2.0. Note that the warnings are emitted by the server. Hence, you also need to run Podman v4.2.0 in the podman machine. My memory on the issue was fading but just double checked. That should fix the user-experience parts.

I will now look into what we can do with the docs.

[tmds]

I'm running into this issue.

I have a Containerfile1.

FROM busbox

I can build and run it.

I have another Containerfile2, which pulls a specific architecture.

FROM --platform=linux/s390x busybox

I can build this image, but not run it (my system is arm64).

If I don't have the busybox image on my system, and build Containerfile2 first. Then the build of Containerfile1 will no longer work on my system because it uses the s390x image.

I understand this behavior is for compatibility with docker.
Unfortunately, the "fix" requires users to add --arch explicitly on the command.

It would be nice to be able to opt-in to this behavior with an option in containers.conf as mentioned in #12682 by @vrothberg and @rhatdan.

[rhatdan]

SGTM
Do you have an idea of the tag name?

[vrothberg]

I am not a fan (anymore) of changing that behavior. It took a long time to get it done right, so I prefer to keep things as they are. Adding an option to containers.conf adds testing and maintenance burden. It gets hairy in the details: should it only apply when looking up images in the local storage or also when pulling images?

[tmds]

I am not a fan (anymore) of changing that behavior.

I find the implicit state that comes from the local images is unexpected.

should it only apply when looking up images in the local storage or also when pulling images?

My expectation is that this option makes any command that takes an --arch, default to the host architecture.
How does that sound for implementation/testing?

[vrothberg]

My expectation is that this option makes any command that takes an --arch, default to the host architecture.
How does that sound for implementation/testing?

Isn't that the current behavior? I understood your proposal that a run foo would work after pull --arch foo.

[tmds]

Isn't that the current behavior? I understood your proposal that a run foo would work after pull --arch foo.

The current behavior is: when no arch is specified, and an image with arch xyz is present, arch is assumed to be xyz rather than the host arch.

[vrothberg]

That is correct, Podman will throw a warning in that case (as Docker) but proceed.

[rhatdan]

If both mages are availabel XYZ and the host ARCH, does Podman do the correct thing and default to host ARCH? Or is it random based on the "last" created image?

[vrothberg]

If both mages are availabel XYZ and the host ARCH, does Podman do the correct thing and default to host ARCH? Or is it random based on the "last" created image?

In case of a local manifest list, Podman will lookup the local platform unless otherwise specified by the user on the CLI.

[jvullo]

Just want to point out the docker behaviour is not the same, it's worse.

when arch is specified foo, and an image with arch bar is present, image with bar is used - effectively ignoring the --platform command after it's first used.

 docker build --no-cache -q --platform=linux/amd64 -t test-app:1 . && \
    docker image inspect test-app:1 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker build --no-cache -q -t test-app:2 . && \
    docker image inspect test-app:2 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker build --no-cache -q  --platform=linux/arm64 -t test-app:3 . && \
    docker image inspect test-app:3 --format '{{ .Os }}/{{ .Architecture }}' && \
    docker build --no-cache -q -t test-app:4 . && \
    docker image inspect test-app:4 --format '{{ .Os }}/{{ .Architecture }}'

Output from above is always "amd64"

sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**
sha256:2575b6e83cfd2cb36a913eb17dec1d66bbdb629f4001c32d191213d102b5e575
**linux/amd64**

• Docker CLI was: Docker version 20.10.10, build b485636
• Docker Engine was whatever the latest version of docker available on 22nd of august when I first did this test.

[paravoid]

I've also been bitten several times by this. The last time I reported it was at #12682 (comment). Apologies for my insistence, but I couldn't help but +1 this :)

The new warning helps and many thanks for that! But I still find the current behavior as extremely disorienting. A tangential and related issue is that the CLI output doesn't help and is in fact contributing to my confusion in addition to everything that is described above. For example, as I mentioned on the comment that I linked above, the "image list" output:

$ uname -m
x86_64
$ podman image ls
REPOSITORY                    TAG            IMAGE ID      CREATED        SIZE
docker.io/library/debian      bullseye       afbb9e728ec0  2 weeks ago    123 MB
<none>                        <none>         d8cacd17cfdc  2 weeks ago    129 MB

afbb9e728ec0 is the arm64 debian:bullseye image. d8cacd17cfdc (<none> as podman calls it) is the amd64 debian:bullseye one. This is the result of a simple podman image pull debian:bullseye; podman image pull --arch arm64 debian:bullseye. I hope it's evident how this can be confusing! Now imagine pull these images at separate times, and having cases with multiple architectures, images, etc.

Aiming for Docker compatibility is understandable and has been generally helpful for me! but I think this is a case of a feature that wasn't properly thought through on the Docker side, and necessitates deviating from it, even if behind a feature flag as proposed above.

In my ideal world, there would be a "multiarch" config flag that would a) an "ARCH" column to that output above, and would result in both containers being listed as debian:bullseye above, but with a different arch, b) same for things like podman container ls etc. c) new defaults where --arch always defaults to the native architecture, and --arch always needs to be passed for operating on non-native ones.

[rhatdan]

Let's discuss this at the next standup or container cabal, since I think @paravoid has valid ideas. @vrothberg @mtrmac WDYT?

[vrothberg]

I've shared my opinion above a number of times. Changing the current default behavior is not an option to me. Docker compatibility is the goal and it took quite some work to get there in this context.

I feel that we are all talking about slightly different things. Maybe because we're discussing multiple issues in an already closed one.

But I think we can agree that we cannot satisfy all use cases with one default. If users desire a "resolve the image by name only and ignore its platform", I am OK to add that to containers.conf. For that, please open a new issue so we can tackle things separately.