Add ARM build

[2phost]

Describe the bug

When I run this container on an ARM system I see the following error:

standard_init_linux.go:207: exec user process caused "exec format error"

I think it may be related to the system being ARM based, and your image was not compiled in ARM based systems.
Is it possible to add an ARM build? For example using the tag arm and arm64 for each system.

Tested on Kubernetes 1.17

[amouat]

Yes, I'm pretty sure you're right. It shouldn't be a problem to add an ARM build. Previously, there was an issue with some underlying libraries that made it difficult.

I'm afraid I'm away from the office for around a week, but I'll get to this when I'm back. In the meantime I'll update the title to track this issue.

[2phost]

Ok, thanks.
If you need further information or any help (for example testing/debugging something or other) just ask and I will do it on my side.

[amouat]

Thanks, I'll probably ask you to test it if that's ok.

[amouat]

I just did a build for arm64 and it seems to have worked without changes. Is there a specific platform you wanted? arm64 or arm/v7?

Whilst it's easy to do a one-off build, I'm not sure how to set up infrastructure to do this automatically yet.

[2phost]

Thanks. My target system is ARMv7 based, as soon as you have the build ready I can test it and give you feedback.

Setup this build automatically it is a nice to have feature. I am not sure about your pipeline to create the builds, but you can build it with buildx of docker and create different tags for each architecture.

[amouat]

Please try out the image containersol/trow:armv7.

There's actually no need for different tags for each architecture thanks to manifest lists (https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#manifest-list) which allows the client to pick the correct image for the architecture.

The problem is Trow currently uses Docker Hub automated builds, and I don't think I can (easily) get the Docker Hub to do multi-platform builds - it doesn't seem to support buildx.

I've also ran into some problems with QEMU when doing builds for other platforms, which is another problem. The ideal solution is to have a build cluster with dedicated amd/armv7/arm64 machines for doing this. But I don't have the money or time to support that currently.

I'll have a think, but I suspect all I can do for the minute is manually create multi-platform builds for releases and extra documentation & scripts for creating such builds.

[2phost]

I just tried it out and I am getting the same error message as before:

standard_init_linux.go:207: exec user process caused "exec format error"

I just changed the tag, but I will try to debug the issue during this weekend.

[amouat]

You can also try amouat/trow:armv7 which I built in a slightly different way. I've added some docs to the User Guide to explain the different ways to build for different platforms.

Can I ask exactly what hw you're building for? If it's a raspberry pi, which model is it?

[amouat]

Just playing with the image on an Pi model 3 and getting the same issue. I see the architecture is wrong in docker inspect. I'll see what I can do, but I'm wondering if it's a bug in Docker.

[2phost]

I am not at home this weekend, as such I will only be able to test it during next week. About my setup, I have a lab with several Raspberry pi's 4 which I use to test several solutions before I put them in production. I am running a custom kernel just with some changes from the original raspbian image. The changes I made was just to have CRI-O running with kubernetes with cpu scheduling support, as such it shouldn't make any difference. I will give you feedback about the second image you send as soon as possible.

…

On Sat, Jan 25, 2020, 5:16 PM Adrian Mouat ***@***.***> wrote: Just playing with the image on an Pi model 3 and getting the same issue. I see the architecture is wrong in docker inspect. I'll see what I can do, but I'm wondering if it's a bug in Docker. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5CPYQG7LA7KJTGGXBANRLQ7RXXZANCNFSM4KHJ5PVQ> .

[amouat]

I just got it working on a Pi 3. I think I was hitting a "feature" in buildkit (moby/buildkit#1057) and a bug in QEMU. However containersol/trow:armv7 should now work for you.

[2phost]

I just tried it again and it seems to be working now.
Thanks for the build!

Maybe you can create some documentation about it before close this issue.

[2phost]

I just checked your documentation and it seems to have the Multiplatform Builds section ;)

[amouat]

Yeah, I added that over the weekend. It needs changed though - it seems builds with QEMU fail with an error message, so I'll recommend the cross-compilation with Rust method.

Once I've updated the docs, I'll close this issue. I might also create an arm64 build.

[munntjlx]

Just a quick question, how much memory does the whole kettle of fish use?

[kervel]

the arm64 build seems to work just fine (installed using helm). however, this document https://github.com/ContainerSolutions/trow/blob/master/docs/USER_GUIDE.md still mentions that there is only support for amd64...

[amouat]

@munntjlx sorry, I never noticed this question. Honestly, I don't know the answer. It shouldn't be a lot as it's largely stateless. I'm in the middle of doing some refactorings that may have an impact, once things are more stable, I'd like to do some profiling.

@kervel thanks, that's good to know. I'll update the guide. It might be old or it might be more just that I'll find it hard to explicitly provide support for those platforms.

[amouat]

@kervel the User Guide was just out-of-date. I've updated the text, thanks a lot for pointing it out.