-
Notifications
You must be signed in to change notification settings - Fork 310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Base image development encounters load metadata errors (darwin-arm64) #4932
Comments
I have a suspicion that this is related to when the docker image is published to hub.docker.io. Perhaps there is some kind of issue with the k3d registry? while |
I've been trying an assortment of additional little things, and I'm not so sure anymore what the issue is... I wish I knew how to troubleshoot this more deeply than just getting |
It appears that multi-stage builds are a whole other complexity with this error at it's center |
I've opened another branch on my repro repo. This demonstrates a little more complex of a set-up with multiple-stage builds which fail with similar issues. I'm not sure if the two issues are a part of the same issue, but both prevent me from using tilt with both multi-stage and base images. For the record, independent multi-stage images work fine even pointing against latest tags. |
I tried your example (colelawrence/nodejs-express-k8s@33b9e64) on Linux/Kind and Linux/k3d, and it built OK for me. We can poke around on it more after the weekend. Given some of the symptoms you're describing, in particular:
my educated guess is you've run into some sort of multi-arch image bug (i.e., you're dealing with an image for multiple chipsets, and some part of the system is handling it incorrectly). But it's hard to say from here which components are misbehaving (whether it's tilt, or the k3d registry, or the docker build, or some interop between any two of them) 😢 |
@nicks, thanks so much! I'll be happy to help diagnose when the weekdays come. For now, I might just have to switch operating systems, or I might see if I can force building a more universal byte code with qemu. |
I've tested with I'm largely experiencing the exact same issues. It appears that this issue happens even without Tilt or k8s. So, this seems like it might be needed to be reported elsewhere. I've created a few minimal test cases at https://github.com/colelawrence/nodejs-express-k8s/tree/7923b335a4dcc3689fcd800e47122fba56c04d82 See the outputs between the following:
In this repo, I have a set of The gist so far, is that the commands work when I This may have an effect on several things, but one thing I found very interesting was this The passing version does NOT have the "Variant": "v8". I'm not sure if this is related, but it seems interesting. |
Possibly related "docker pull --platform=linux/arm64 silently falls back to linux/amd64 image" docker/for-mac#5625 |
This looks very related docker/for-mac#5873 (comment) |
One thing that I think could immediately unblock me is if I could either set docker settings via env vars for Otherwise, I am actually able to do |
Does setting DOCKER_BUILDKIT=0 fix the problem? Tilt should respect that. |
Setting I can look into re-writing images that were requiring I just deleted a following comment about having to prune, because I was using the wrong env var when experiencing that issue |
Thanks for all the debugging @colelawrence! I just opened #4934 which will allow you to either explicitly pass You could handle this automatically by putting something like the following at the beginning of your if 'DOCKER_DEFAULT_PLATFORM' not in os.environ:
arch=str(local('uname -m', command_bat='echo %PROCESSOR_ARCHITECTURE%', quiet=True, echo_off=True)).rstrip().lower()
if arch == 'arm64':
print('Setting default architecture for Docker builds to "linux/arm64"')
os.putenv('DOCKER_DEFAULT_PLATFORM', 'linux/arm64') |
Docker CLI has a `--platform` argument and `DOCKER_DEFAULT_PLATFORM` environment variable for default. This is useful when dealing with multi-architecture images in different scenarios, e.g. using an M1-based macOS machine. This adds a `platform` argument to `docker_build` which will get passed through to Docker/BuildKit. Similar to the Docker CLI, if not specified, `DOCKER_DEFAULT_PLATFORM` will be used as a default if set. Fixes #4932.
v0.22.8+ is out which supports See https://docs.tilt.dev/api.html#api.docker_build for usage |
Expected Behavior
Should be able to build base images from all images.
Using k3d, macOS M1 aarch64, running
tilt up
in the following repo:colelawrence/nodejs-express-k8s@33b9e64
Current Behavior
Steps to Reproduce
colelawrence/nodejs-express-k8s@33b9e64
k3d cluster create --config $configfile
with a create registry on latest Docker for Desktop for macOS M1.git clone https://github.com/colelawrence/nodejs-express-k8s
tilt up
Context
tilt doctor
OutputAbout Your Use Case
I cannot successfully use base images with
tilt up
.(In my real code, I actually was experiencing this
ERROR IN: [internal] load metadata for localhost:52780/tilt.dev_nodejs-express-base:tilt-858fb587e21f53c7
when I was based fromrust:buster
, so I'm suspicious that it's some kind of syntax property or build kit issue... maybe related to aarch64 (since that's usually the issue I end up hitting).I've been debugging this for about 3 hours trying to figure out what the difference was between the example and my code. I really don't know how to pry into the
ERROR IN: [internal] load metadata for
error, but I am happy to help figure it out.The text was updated successfully, but these errors were encountered: