Add test_alpine to circleci config

[straight-shoota]

Fixes #6943

This is just experimental for now.

[j8r]

Great @straight-shoota ! You use my image 😮
Considering Alpine is often used for static compilation, why not enabling it too here?
Another point, I hope my image will be put in the crystal-lang organization and will become official :)

[straight-shoota]

Yes, we'll likely need to use a custom image from crystal org which gets updated on every release build and doesn't have to depend on apk being up to date.

But your image is fine for a preliminary exploration 👍

[ysbaddaden]

You can extract the static crystal binary from the official tarball. Here is a Dockerfile that will configure an Alpine Linux 3.9 image with everything needed to execute a complete workflow on CI:

FROM alpine:3.9

ARG VERSION=0.27.2
ARG REVISION=1
ARG ARCH=x86_64
ARG TARBALL=https://github.com/crystal-lang/crystal/releases/download/${VERSION}/crystal-${VERSION}-${REVISION}-linux-${ARCH}.tar.gz

RUN apk add --virtual .build-deps build-base curl git
RUN apk add --virtual .crystal-deps gc-dev libevent-dev libunwind-dev llvm5-dev pcre-dev
RUN apk add --virtual .crystal-spec-deps gmp-dev libxml2-dev openssl-dev readline-dev yaml-dev zlib-dev

RUN set -e; \
    curl -sL ${TARBALL} | tar xz -C /tmp; \
    cp /tmp/crystal-${VERSION}-${REVISION}/lib/crystal/bin/* /usr/local/bin/; \
    rm -rf /tmp/crystal-${VERSION}-${REVISION}

And disable all jobs/workflows while working on the branch, that will be quicker :)

[straight-shoota]

Unfortunately, the generic linux binary doesn't work on alpine/musl because the baked in gnu target triple (see #7196).

But if it did, that would exactly be the way I've described in #6943 (comment)

[ysbaddaden]

@straight-shoota just export FLAGS="--target $(llvm-config --host-target)" or call make FLAGS=.... We can't pass --target to crystal run but the Makefile only calls crystal build where we can pass --target and it will just work (verified).

Errata: except that compiling Crystal means compiling some ECR templates which calls crystal run 💩

[straight-shoota]

Yes, macro runs is the main issue that can't be worked around (unless with sed -i -e 's/x86_64-unknown-linux-gnu/x86_64-alpine-linux-musl/' /usr/local/bin/crystal). The default target should not be baked in but depend on ldd --version (or llvm-config --host-target), at least to distinguish between linux-musl and linux-gnu.

[ysbaddaden]

It would be great to have a minimal bootstrap compiler with just the build command. No playground tools, or docs, not even run or eval. ping @asterite

[asterite]

@ysbaddaden Yes, I always wanted to move those tools to separate binaries, even if it means having 4 or 5 "big" binaries (right now crystal is about 42MB, so maybe all of them would occupy at most 200MB, maybe less because each binary will be smaller). I'd really like that to happen. Also removing some parts of the std to separate shards (that could live in the crystal-lang organization, like XML, YAML, OAuth, etc.). But last thing I remember is that most didn't want to go into that direction (one thing was avoiding a more complex build/release that would involve multiple binaries, I don't think that's a big issue given how automated things are now).

[j8r]

That's already nice to have moved out package management. I've worked to move out Crystal playground, apparently only few wanted to. I would work again on it if there is a consensus on it.

[straight-shoota]

What does this all have to do with ci builds on alpine?

[asterite]

I think nothing 😊

[j8r]

I think because it complicates portability and porting on other environments.

[ysbaddaden]

@straight-shoota a bootstrap compiler would allow to use the released static crystal binary on alpine, and overall ease cross platform development and experimentations :)

[straight-shoota]

I don't see why that would make a difference. We just need to make the target host configurable for all compilation steps (thus including macro evaluation) and use a default depending on the environment. That's the same for both a small as well as a fat compiler.

[asterite]

Macros run in the host, why would that need to be configurable?

[straight-shoota]

Configurable as in "configures itself", instead of being baked in. So the same compiler binary can work with both musl and gnu by default (and without requiring any manual configuration).

[straight-shoota]

Now the specs are running into Failed to raise an exception: END_OF_STACK https://circleci.com/gh/crystal-lang/crystal/19693?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Maybe this is related to #6934 / #4276? Although it's not a static build.

[j8r]

@straight-shoota do you still work on this? The CI can be re-triggered, mps has pushed Crystal 0.29 to Alpine, my image use this version.
This CI will be very useful to prevent issues like this one.

[straight-shoota]

@j8r I've triggered a rebuild but it still fails for some reason I'm not going to investigate further. This is also an old branch which would need a rebase.

But anyway, this should build on crystal-lang/distribution-scripts#29 for a custom alpine-based Dockerfile so we don't need to rely on alpine package updates (which might take a few days after a Crystal release).

[j8r]

Indeed, the image will have to change. I've removed g++ on the image, because only the Crystal compiler needs it. That's offtopic: why g++ over Clang?

Edit: this is for llvm_ext, both Clang or g++ would be fine. Not sure if this file is always needed, depending of the LLVM version used.

[j8r]

Any plan to have an Alpine CI any time soon?
I experimented with Drone CI, but it looks like the tests are too long to finish (cancelled because >1hour).

[straight-shoota]

I'd like to move forward, but it is held up by crystal-lang/distribution-scripts#29 Waiting for feedback there.

[straight-shoota]

Updated to use crystallang/crystal:ci-test-dev docker image.

There's one failure in stdlib specs: #7482

[bcardiff]

It would be great to emit the junit output as other jobs. The *ci_steps does not have those instructions, it can be added there.

NB: the *ci_steps is currently not used due to the workaround of splitting the specs. But let's improveit and keep it. The DRY will come back at some point, will come back at some point.

[j8r]

Just a thought: if Alpine aarch64 CI is planned, wouldn't be better to have all Alpine CIs in GH Actions?
Since it is free, perhaps it would save some hours credits from Circle CI (I don't know the billing plan).

[straight-shoota]

There are currently no plans for Alpine aarch64. At least I'm not aware of any. Duplicating the workflow on circleci is far easier integration than setting up a new GitHub action. And AFAIK circleci credits are not a concern atm.

So we should stick with this PR which is ready to merge. The important goal is to gain spec coverage linking against musl-libc. We can always reconsider whenever we get around implementing CI for alpine on aarch64. But that's currently not a high-priority concern. Maybe at that point we might even have advanced further towards a single, integrated CI platform.

[straight-shoota]

I've removed the explicit config, now the job uses the regular alpine image. This is ready to merge!