DockerStorage not running on different architecture (x86 vs M1)

[marvin-robot]

Archived from the Prefect Public Slack Community

erik.amundson: Hello, I am using Docker storage to store my flows, but I'm on an ARM mac and our production environment is x86. From the command line I can build the docker image for the correct architecture using docker buildx build --platform linux/amd64 . Is there any way to do this in the prefect.storage.docker.Docker class or can I build the image locally first then add flows to a pre-built image?

znicholasbrown: Hi <@U026XURSG3Z> - I believe you should be able to pass the platform type as an extra build_kwarg to the storage class

znicholasbrown: So something like:

Docker(
  ...
  build_kwargs=dict(platform='linux/amd64')
)

erik.amundson: That doesn't seem to change anything, I'm still getting the error

failed to get destination image "xxx": image with reference xxx was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64

I think docker buildx is separate from docker build so docker build doesn't recognize the kwarg platform

znicholasbrown: Ah in which case I don't think so; I could be wrong but I don't think the docker-py sdk supports the new docker build kit, there's an issue open for it https://github.com/docker/docker-py/issues/2230|here

znicholasbrown: Let me see if there's another way to handle this like you suggested

kevin701: I think the https://docker-py.readthedocs.io/en/stable/api.html#module-docker.api.build|platform here should support it on the build side, but the problem is you might need to specify the platform on the pull/run side per https://github.com/docker/for-linux/issues/1170|this, but py-docker doesn’t have this exposed on create_container per https://github.com/docker/docker-py/issues/2822|this .

erik.amundson: Ah I see, I tried pulling the python-3.8 (our base image) for linux/amd64 image before running the register script and that seems to be working, thank you!

kevin701: <@ULVA73B9P> archive “DockerStorage not running on different (architecture x86 vs M1)”

Original thread can be found here.