Multi-arch support #324

alexkreidler · 2017-08-15T23:39:48Z

See #237 and #304

Todo:

add multi-arch build capabilities to the alpine-builder script
add multi-arch support to the configuration in some way
write multi-arch build code
generate library spec file
fix builder tagging + pushing for gliderlabs images

Last tasks

Testing?
Which versions to roll out on?

alexkreidler · 2017-08-16T15:54:17Z

Update:

I've gotten it to be able to build different architecture images from one config.
Example:

# versions/library-edge/options
export RELEASE="edge"
export MIRROR="http://dl-cdn.alpinelinux.org/alpine"
export PACKAGES="alpine-baselayout,apk-tools,alpine-keys,libc-utils"
export ARCHS=(x86_64 armhf)
export BUILD_OPTIONS=(-b -s -t UTC -r $RELEASE -m $MIRROR -p $PACKAGES)
export TAGS=(alpine:edge)

Running ./build versions/library-edge/options will generate:

    ...
    ├── library-3.6
    │   ├── Dockerfile
    │   └── options
    └── library-edge
        ├── Dockerfile
        ├── options
        ├── x86_64
        │   ├── Dockerfile
        │   └── rootfs.tar.xz
        └── armhf
            ├── Dockerfile
            └── rootfs.tar.xz

I've updated the appropriate docs, and I'd really appreciate feedback on this setup.
There are a few TODOs, listed above. It still can't tag the images properly yet, or generate the library spec
Currently, all the older images should build perfectly fine. (they do on my machine)
I've tested the generated armhf images on my Raspberry Pi 3 (armv7), and it works great!

WDYT @andyshinn @tianon

alexkreidler · 2017-08-17T21:44:00Z

Almost There!

I've got the library spec generator working with multi-arch.
If you run ./build library, you get:

# maintainer: Glider Labs <[email protected]> (@gliderlabs)
# autogenerated by https://github.com/gliderlabs/docker-alpine/blob/master/build

GitRepo: https://github.com/gliderlabs/docker-alpine.git

Tags:alpine:3.1
Architectures: amd64
GitCommit: 42d89fb9844b17e50875b97c631dd1ca1591107c
amd64-Directory: versions/library-3.1/x86_64

Tags:alpine:3.2
Architectures: amd64
GitCommit: f78fe34e562a3cd50501a44d41a4384a90d9dd42
amd64-Directory: versions/library-3.2/x86_64

Tags:alpine:3.3
Architectures: amd64
GitCommit: 003bfad270dd87ce9cba3b2782a39229c61932f5
amd64-Directory: versions/library-3.3/x86_64

Tags:alpine:3.4
Architectures: amd64
GitCommit: 9fd08ce7d9040ab265adca36633f09570176a4fa
amd64-Directory: versions/library-3.4/x86_64

Tags:alpine:3.5
Architectures: amd64
GitCommit: 731d755ebe08c13be6df34d2ea2daaacbfc974b0
amd64-Directory: versions/library-3.5/x86_64

Tags:alpine:3.6, alpine:latest
Architectures: amd64
GitCommit: 1df7cd436b81e0c8f6f84beb3bf8031de21829d4
amd64-Directory: versions/library-3.6/x86_64

Tags:alpine:edge
Architectures: amd64, arm32v7
GitCommit: c0f432b0e678c1096e041e7e63a089444ad544c9
amd64-Directory: versions/library-edge/x86_64
arm32v7-Directory: versions/library-edge/armhf

Then you just copy that into the library/alpine spec file in the docker-library/official-images repository to release a new version.

The last step is to figure out what to do with multi-arch images for gliderlabs. I'll reach out to them. They may want to just keep their images as x86_64.

Once that's figured out and I fix a few more things, this PR will be all set.

Apologies to all for clogging up this PR and the related issues with updates, etc. I am using this as a sort of documentation in place. Eventually, if requested, I could put it all in a markdown file in docs.

After this PR is good, the maintainers will decide which versions to roll out multi-arch on. Currently the only config I've changed is library/edge, which now builds x86_64 and armhf (as you can see above). I think this will require coordination with the maintainers of docker-library/official-images, as they are very careful about upgrades and don't want to break anything.

One thing I wonder is if explicitly specifying only amd64 instead of the default inferred architectures will have implications on the i386 alpine images. I hope it won't break anything.

andyshinn

In the commit function there is a glob on versions/**/rootfs.tar.xz which no longer works correctly in bash. This might be simply solved with shopt -s globstar somewhere in the script (maybe the top is fine). It requires bash 4.x but I think we can just switch the CIrcleCI builder to Ubuntu 14.04 to get bash 4.

The push function also isn't aware of the arches. But since we don't append the -$arch to the tag for x86_64 and only use push for the gliderlabs org this is probably fine.

I think we want to enable arm for all the library versions that support it. I'm not sure if @tianon had other plans for the Alpine arm variants. But if the builder and build script now support them here, we can just host them here.

So, other than the globbing issue for commiting the tarballs from CI, this looks ready for merge.

Edit: some minor linting issues as well (master branch is currently green in shellcheck so maybe give the changes a run through it to see).

andyshinn · 2017-08-19T21:44:22Z

Also, does this close #252? It looks like this supersedes #252 and maybe helps #190 as well.

andyshinn · 2017-08-19T22:00:43Z

build

+		for arch in "${ARCHS[@]}"; do
+			# Generate rootfs
+			mkdir -p "$version_dir/$arch"
+			docker run -e "TRACE=$TRACE" --rm "$BUILDER_IMAGE" -a $arch "${BUILD_OPTIONS[@]}" \


$arch should probably be double quoted here.

andyshinn · 2017-08-19T22:01:32Z

build

+	convert() {
+		# takes a space-separated list of alpine-linux architectures, and returns a space-separated list of docker architectures
+		local i=0
+		for arch in $@; do


I'm guessing $@ here should be double quoted as well.

andyshinn · 2017-08-19T22:02:33Z

build

-			echo "${tag#*:}:" \
-				"git://github.com/gliderlabs/docker-alpine@${refs:0:40}" \
-				"versions/library-${RELEASE#v}"
+		tags=${TAGS[@]}


Instead of assigning a string to an array, you can just expand the array with tags="${TAGS[*]}".

andyshinn · 2017-08-19T22:06:47Z

build

+		[[ "$ARCHS" ]] || { # compatability with older configs that were only written for x86_64 architectures
+				ARCHS=("x86_64")
+		}
+		a=$(convert ${ARCHS[@]})


Double quotes around ${ARCHS[@]}.

andyshinn · 2017-08-19T22:07:06Z

build

+		}
+		a=$(convert ${ARCHS[@]})
+
+		printf "\nTags:" && echo ${tags// /, }


Double quotes around the following 3 echoed variables as well.

alexkreidler · 2017-08-20T03:45:34Z

Fixed the linting issues, added shopt -s globstar. I'll need to test the commit function tomorrow. Hopefully that fixes it. There may be better solutions to the globbing problem, but this is easiest for now.

alexkreidler · 2017-08-20T03:50:58Z

I think this supersedes #252, because this is more fleshed out and complete. It should allow us to add aarch64 (#190) or any other currently supported architecture in both docker and alpine.

andyshinn · 2017-08-20T04:34:44Z

Hmm, it looks like the commit function will need some more cleanup than I thought. I'm still thinking about how to fix it up (or rewrite it to loop over the options files like the other functions do).

alexkreidler · 2017-08-20T14:54:31Z

I think the best option is to loop over the options files, and then instead of just adding and commiting the one rootfs.tar.xz, we add all the arch directories, and then commit them.

alexkreidler · 2017-08-20T15:57:30Z

That should fix the commit function, and still be compatible with old the old build structure.

andyshinn · 2017-08-20T19:25:43Z

This seems reasonable. I'll check it out and test tomorrow.

tianon · 2017-08-21T21:45:46Z

build

+			case "$arch" in # converts Alpine Linux arch strings to Docker arch strings
+				x86_64) echo -n "amd64";;
+				x86) echo -n "i386";;
+				armhf) echo -n "arm32v7";; # TODO: What about arm32v6?


In Alpine, armhf is arm32v6, not v7 -- Alpine doesn't officially support v7 directly (only indirectly via v6, which is fine).

tianon · 2017-08-21T21:47:03Z

build

+	}
+
+	echo
+	echo "# maintainer: Glider Labs <[email protected]> (@gliderlabs)"


To convert to the new file format, this line needs to be part of the initial "global" block, ala:

Maintainers: Glider Labs <[email protected]> (@gliderlabs) GitRepo: https://github.com/gliderlabs/docker-alpine.git

tianon · 2017-08-21T21:48:56Z

build

+		echo
+		echo "Tags: ${tags// /, }"
+		echo "Architectures: ${a// /, }"
+		echo "GitCommit: ${refs:0:40}"


If these commits are not going to be on the master branch, then each block will also need a GitFetch line:

Tags: 3.6, latest Architectures: foo, bar, baz GitFetch: refs/heads/rootfs/library-3.6 GitCommit: c6cb2bbffbc92d4c35cdaf584dd1c43a860104ea foo-Directory: versions/library-3.6/foo ...

alexkreidler · 2017-08-22T00:39:00Z

Thanks for the review. All fixed now.

andyshinn · 2017-08-22T17:16:45Z

Looks good. But CI is failing to build the armhf variant. Maybe this is just an older Docker issue (I think CircleCI is running 1.9)? I'll research this more. The same builds fine locally.

alexkreidler · 2017-08-22T19:45:11Z

Yeah, it also fails to build on play-with-docker.com, which is basically an alpine dev/demo environment. I've done some digging, and it seems that the alpine-baselayout package, and the busybox package have some install scripts that fail. I haven't looked at busybox yet. However, I've download, extracted, and compared the .apk files for both the x86_64 and the armhf variants, and the scripts are exactly the same. The main difference is the shared object libraries they require. (libc.musl-x86_64.so.1, and libc.musl-armhf.so.1 respectively). I wonder if @ncopa has any idea why the scripts fail.

What's really confusing is that they don't fail locally. I'll keep digging and update you guys on this.

andyshinn · 2017-08-22T19:48:37Z

OK, after some more research it seems this was only really working because of https://docs.docker.com/docker-for-mac/multi-arch/. The trigger scripts for arm fail in normal Linux Docker environments. This... seems really strange to me... I'm not sure why Docker for Mac is built in such a way to encourage non-deterministic behavior like this. Frustrating...

alexkreidler · 2017-08-22T19:50:19Z

I see. The unlucky issue of us both using Macs.

alexkreidler · 2017-08-22T19:51:46Z

I'll take a look at the busybox package failures and see if I see the pattern of the platform-dependent .so libs.

andyshinn · 2017-08-22T19:53:01Z

I think worst case is maybe I will just remove armhf from the ARCHS array for now. The rest of this still feels like the right direction and we can probably figure out a workaround for the triggers.

alexkreidler · 2017-08-22T19:56:47Z

Yup, busybox armhf and x86 variants have the same scripts, different .PKGINFO lib dependencies, also in this case a different binary.

@andyshinn Probably a good idea while we figure out this issue.

alexkreidler · 2017-08-23T01:33:26Z

It seems that the edge release does not provide armhf variants of the musl binaries. When downloaded and extracted, the edge/musl package actually contains the x86_64 binaries. However, the I know the 3.6 version and I'm assuming the older versions as well have the "real" armhf binaries.

I'll dig some more and try to get a fix tomorrow.

alexkreidler · 2017-08-26T02:23:57Z

The install scripts from the packages just fail. I've opened an upstream issue: https://bugs.alpinelinux.org/issues/7767

There are alternative methods to still implement multi-arch functionality, such as just automatically downloading the tar.gz releases from the server (e.g. dl-cdn.alpinelinux.org/alpine/$version/releases/$arch/alpine-minirootfs-$version-$arch.tar.gz) and committing them to git as the rootfs.
See release types here. We would probably use the mini-rootfs variant.

This method would limit customization, as we would have to extract, change stuff, then re-compress. Maybe that's a good thing for the official images, but would be harder for the gliderlabs variants.

WDYT @andyshinn about alternative methods

andyshinn · 2017-08-26T03:32:41Z

There is a --no-scripts option to apk add. I wonder if we could add this as a new builder option and just ignore the scripts if they are not doing anything needed?

The scripts that are failing (in my build at least) are:

https://git.alpinelinux.org/cgit/aports/tree/main/alpine-baselayout/alpine-baselayout.pre-install: Maybe we could just emulate this?
https://git.alpinelinux.org/cgit/aports/tree/main/alpine-baselayout/alpine-baselayout.post-install: Looks like this does a bunch of init / OpenRC stuff. Also some chowning at the end we could emulate. But I wonder if the OpenRC stuff is even needed since generally containers are not used as an init system.
https://git.alpinelinux.org/cgit/aports/tree/main/busybox/busybox.trigger: Not quite sure what all this is for.
https://git.alpinelinux.org/cgit/aports/tree/main/busybox/busybox.post-install: This seems to be part of the trigger and just has a comment that it is needed earlier. But what for?

Might be worth trying.

If we were going to go the route of just storing the upstream tarballs I'd think about just rewriting the script to pull those down and commit them. I'm not sure if extracting, testing, and modifying them makes sense in that case. I'd assume that whatever testing of how they were built is the responsibility of upstream.

alexkreidler · 2017-08-26T12:17:52Z

I suspect we'll get an unusable system, because busybox --install symlinks from each command to the busybox multi-call binary. (e.g. /bin/ls --> /bin/busybox ls)

alexkreidler · 2017-08-26T12:25:07Z

Well, just tried it out, and you can use pwd, cd, echo, printf, and shell commands, but commands like ls and ps give /bin/sh: ls: not found. I guess we'll just have to emulate the important stuff like busybox --install and/or wait till they fix apk.

alexkreidler · 2017-08-28T11:29:16Z

If we were going to go the route of just storing the upstream tarballs I'd think about just rewriting the script to pull those down and commit them. I'm not sure if extracting, testing, and modifying them makes sense in that case.

Just pulling the tarballs and committing them without any changes would limit some of the customization options that we currently have.

This would probably eliminate:

$OMIT_COMMUNITY
$REPO_EXTRA
$ADD_BASELAYOUT
$TIMEZONE
$DISABLE_ROOT_PASSWD
$ADD_APK_SCRIPT

However, this might be a good thing if the goal is to closely mirror upstream without many changes.

Of course, if these options are very important, we could just extract to a temporary working directory, make the changes as before, and then re-compress to the rootfs.

I will start working on a version that just pulls down the tarballs.

* Add -a flag for multiarch support in alpine-builder. * Add ARCHS variable for building different architectures. * Add an example of multi-arch * Update docs for multi-arch * Fix typo in library-edge options * Update library spec generator for multi-arch * Fix tagging of images for multi-arch * Fix style shellcheck issues, add comments * Fix rootfs commit glob issue by adding globstar option. * Update .gitignore for new multi-arch structure. * Rewrite commit loop based on options files, fix commit issues. * Fix library spec issues

Add -a flag for multiarch support in alpine-builder.

ea02a45

alexkreidler force-pushed the multi-arch branch from f785a6f to ea02a45 Compare August 16, 2017 13:47

alexkreidler added 3 commits August 16, 2017 11:23

Add ARCHS variable for building different architectures.

849e396

Add an example of multi-arch

bff6d50

Update docs for multi-arch

1e628d9

alexkreidler added 2 commits August 17, 2017 16:14

Fix typo in library-edge options

30c3026

Update library spec generator for multi-arch

1ff565c

Fix tagging of images for multi-arch

8d0364a

alexkreidler changed the title ~~WIP: Multi-arch support~~ Multi-arch support Aug 19, 2017

andyshinn self-requested a review August 19, 2017 21:32

andyshinn suggested changes Aug 19, 2017

View reviewed changes

andyshinn reviewed Aug 19, 2017

View reviewed changes

alexkreidler added 2 commits August 19, 2017 23:12

Fix style shellcheck issues, add comments

8bc9dc9

Fix rootfs commit glob issue by adding globstar option.

51e4e16

Update .gitignore for new multi-arch structure.

6498f39

alexkreidler force-pushed the multi-arch branch from 1bf6b51 to 44c7ab5 Compare August 20, 2017 15:50

Rewrite commit loop based on options files, fix commit issues.

c2585bf

alexkreidler force-pushed the multi-arch branch from 44c7ab5 to c2585bf Compare August 20, 2017 16:21

tianon reviewed Aug 21, 2017

View reviewed changes

Fix library spec issues

2ac88ef

andyshinn approved these changes Aug 22, 2017

View reviewed changes

andyshinn merged commit 6e74d4f into gliderlabs:master Aug 22, 2017

This was referenced Aug 22, 2017

Adds an optional ARCH flag to the build script. #252

Closed

aarch64 version #190

Closed

Migrate to CircleCI 2.0 #325

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-arch support #324

Multi-arch support #324

alexkreidler commented Aug 15, 2017 •

edited

Loading

alexkreidler commented Aug 16, 2017 •

edited

Loading

alexkreidler commented Aug 17, 2017

andyshinn left a comment •

edited

Loading

andyshinn commented Aug 19, 2017

andyshinn Aug 19, 2017

andyshinn Aug 19, 2017

andyshinn Aug 19, 2017

andyshinn Aug 19, 2017

andyshinn Aug 19, 2017

alexkreidler commented Aug 20, 2017 •

edited

Loading

alexkreidler commented Aug 20, 2017

andyshinn commented Aug 20, 2017

alexkreidler commented Aug 20, 2017

alexkreidler commented Aug 20, 2017

andyshinn commented Aug 20, 2017

tianon Aug 21, 2017

tianon Aug 21, 2017

tianon Aug 21, 2017 •

edited

Loading

alexkreidler commented Aug 22, 2017

andyshinn commented Aug 22, 2017

alexkreidler commented Aug 22, 2017

andyshinn commented Aug 22, 2017 •

edited

Loading

alexkreidler commented Aug 22, 2017

alexkreidler commented Aug 22, 2017

andyshinn commented Aug 22, 2017

alexkreidler commented Aug 22, 2017

alexkreidler commented Aug 23, 2017 •

edited

Loading

alexkreidler commented Aug 26, 2017

andyshinn commented Aug 26, 2017

alexkreidler commented Aug 26, 2017

alexkreidler commented Aug 26, 2017 •

edited

Loading

alexkreidler commented Aug 28, 2017

Multi-arch support #324

Multi-arch support #324

Conversation

alexkreidler commented Aug 15, 2017 • edited Loading

alexkreidler commented Aug 16, 2017 • edited Loading

Update:

alexkreidler commented Aug 17, 2017

Almost There!

andyshinn left a comment • edited Loading

Choose a reason for hiding this comment

andyshinn commented Aug 19, 2017

andyshinn Aug 19, 2017

Choose a reason for hiding this comment

andyshinn Aug 19, 2017

Choose a reason for hiding this comment

andyshinn Aug 19, 2017

Choose a reason for hiding this comment

andyshinn Aug 19, 2017

Choose a reason for hiding this comment

andyshinn Aug 19, 2017

Choose a reason for hiding this comment

alexkreidler commented Aug 20, 2017 • edited Loading

alexkreidler commented Aug 20, 2017

andyshinn commented Aug 20, 2017

alexkreidler commented Aug 20, 2017

alexkreidler commented Aug 20, 2017

andyshinn commented Aug 20, 2017

tianon Aug 21, 2017

Choose a reason for hiding this comment

tianon Aug 21, 2017

Choose a reason for hiding this comment

tianon Aug 21, 2017 • edited Loading

Choose a reason for hiding this comment

alexkreidler commented Aug 22, 2017

andyshinn commented Aug 22, 2017

alexkreidler commented Aug 22, 2017

andyshinn commented Aug 22, 2017 • edited Loading

alexkreidler commented Aug 22, 2017

alexkreidler commented Aug 22, 2017

andyshinn commented Aug 22, 2017

alexkreidler commented Aug 22, 2017

alexkreidler commented Aug 23, 2017 • edited Loading

alexkreidler commented Aug 26, 2017

andyshinn commented Aug 26, 2017

alexkreidler commented Aug 26, 2017

alexkreidler commented Aug 26, 2017 • edited Loading

alexkreidler commented Aug 28, 2017

alexkreidler commented Aug 15, 2017 •

edited

Loading

alexkreidler commented Aug 16, 2017 •

edited

Loading

andyshinn left a comment •

edited

Loading

alexkreidler commented Aug 20, 2017 •

edited

Loading

tianon Aug 21, 2017 •

edited

Loading

andyshinn commented Aug 22, 2017 •

edited

Loading

alexkreidler commented Aug 23, 2017 •

edited

Loading

alexkreidler commented Aug 26, 2017 •

edited

Loading