Any scope for basing container not on debian but on "zero"?

[mohawk2]

As in this article: https://blog.kintoandar.com/2018/01/Building-healthier-containers.html - it would make for smaller containers, faster builds, etc.

I see that the current Dockerfile does all the building inside itself, and wondered whether there is scope to slim down the output image.

[zakame]

Thanks @mohawk2 ! We're basing on buildpack-deps actually, but yeah, using that results in a fairly large standard image (usually around 800+ MB unpacked.)

Per https://github.com/docker-library/official-images#repeatability we're a bit restricted on which official-image to base from, so if we're looking to build a slim version of this we could explore building a minimal deps package to import into the scratch image then build perl from there; I've been somewhat into that recently via building perls into https://github.com/test-kitchen/kitchen-docker at $WORK.

@tianon @PeterMartini what do you think?

[waterkip]

I'm looking into a multi-stage build for Perl and one can probably do this pretty easy. I don't see any build script for the Dockerfiles so I'm not sure how this all gets build.

I end up with an image that is 112MB big for 5.26.2 (compared to ~800MB).

See #53

[tianon]

Sorry for not responding sooner, I'm always buried (as you can imagine).

I think making a smaller Perl image is a great idea! Going all the way to FROM scratch is a really cool idea, but in practice I personally find that it ends up being kind of limiting (given that there aren't any additional tools like a shell in the image to help debug if things go wrong), not to mention docker-library/official-images#3383 being a hurdle for making an official image based on scratch.

I think Alpine is a really good compromise -- once you add Perl itself, the 4MB for the base becomes pretty insignificant, and then you get the benefit of a decent shell and a proper package manager for installing more dependencies (so the image can be useful without requiring multi-stage builds, too).

The current images are based on buildpack-deps:stretch, which is by definition very, very fat (on purpose). It might be worth trying to make an image based directly on debian:stretch (or even debian:stretch-slim) to see how much lighter it can be.

In many other language stack official images (Python, Ruby, etc) we include slim variants which are usually based directly on debian (instead of buildpack-deps) and then separate alpine variants for folks who want a more significant size reduction.

For example:

• python:3.7-stretch is ~922MB (aliased to python:3.7)
• python:3.7-slim-stretch is ~143MB (aliased to python:3.7-slim)
• python:3.7-alpine3.8 is ~79.4MB (aliased to python:3.7-alpine)

[zakame]

Thanks @tianon!

I think slim is doable now; it'd be similar to a test-kitchen docker build (sans doing under a Chef harness,) as you say it'd be likely just switching the base image (and perhaps adding build-essential kit.)

alpine still bugs me with the lack of locales support and failing tests last time I checked (see #23;) however, I've seen the https://github.com/perlorg folks use it (see https://github.com/perlorg/perl-docker) so it might be worth another shot at doing; we could just be very clear at expressing what the limitations are for it.

[tianon]

Yeah, totally fair! I'd suggest avoiding the catch-all "build-essential" package -- there's probably only three or four packages actually necessary ("gcc", "make", etc), and "build-essential" includes everything necessary to build Debian packages ( https://packages.debian.org/stretch/build-essential). We do have some default boilerplate Alpine-related text that gets added to any image with an Alpine variant automatically (unless overridden for that particular image): https://github.com/docker-library/docs/blob/2f7d03f9603e34227c62eb262b81e1fc85fe1fcb/.template-helpers/variant-alpine.md (which might either be enough by itself, or at least a decent start for writing something specific to Perl)

[zakame]

Did a spike for perl:5.28-slim now (basing from debian:stretch,) it seems promising:

perl    5.28-slim    c4c49d05556e     2018-08-04 03:08:00     400MB
perl    5.28         c82199ef1ab4     2018-06-25 20:49:27     885MB

I'll run a few more test builds against the other versions and build variants, and likely push a new PR this weekend:

• add a way to switch base image, not just tag
• add a new RUN step for installing build-deps for slim
• perhaps overhaul/refactor generate.pl

[tianon]

I think 400MB still leaves some room for improvement 😄

I was able to get down to 113MB with the following fairly minimal patch:

diff --git a/5.028.000-64bit/Dockerfile b/5.028.000-64bit/Dockerfile
index d009951..a537a64 100644
--- a/5.028.000-64bit/Dockerfile
+++ b/5.028.000-64bit/Dockerfile
@@ -1,10 +1,19 @@
-FROM buildpack-deps:stretch
+FROM debian:stretch-slim
 LABEL maintainer="Peter Martini <[email protected]>, Zak B. Elep <[email protected]>"
 
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+# install "netbase" for "/etc/services" and "/etc/protocols"
+        netbase \
+    && rm -rf /var/lib/apt/lists/*
+
 COPY *.patch /usr/src/perl/
 WORKDIR /usr/src/perl
 
-RUN curl -SL https://www.cpan.org/src/5.0/perl-5.28.0.tar.xz -o perl-5.28.0.tar.xz \
+RUN savedAptMark="$(apt-mark showmanual)" \
+    && apt-get update \
+    && apt-get install -y --no-install-recommends ca-certificates curl dpkg-dev gcc libc6-dev make patch xz-utils \
+    && curl -fL https://www.cpan.org/src/5.0/perl-5.28.0.tar.xz -o perl-5.28.0.tar.xz \
     && echo '059b3cb69970d8c8c5964caced0335b4af34ac990c8e61f7e3f90cd1c2d11e49 *perl-5.28.0.tar.xz' | sha256sum -c - \
     && tar --strip-components=1 -xaf perl-5.28.0.tar.xz -C /usr/src/perl \
     && rm perl-5.28.0.tar.xz \
@@ -20,7 +29,12 @@ RUN curl -SL https://www.cpan.org/src/5.0/perl-5.28.0.tar.xz -o perl-5.28.0.tar.
     && curl -LO http://www.cpan.org/authors/id/M/MI/MIYAGAWA/App-cpanminus-1.7044.tar.gz \
     && echo '9b60767fe40752ef7a9d3f13f19060a63389a5c23acc3e9827e19b75500f81f3 *App-cpanminus-1.7044.tar.gz' | sha256sum -c - \
     && tar -xzf App-cpanminus-1.7044.tar.gz && cd App-cpanminus-1.7044 && perl bin/cpanm . && cd /root \
-    && rm -fr ./cpanm /root/.cpanm /usr/src/perl /usr/src/App-cpanminus-1.7044* /tmp/*
+    && rm -fr ./cpanm /root/.cpanm /usr/src/perl /usr/src/App-cpanminus-1.7044* /tmp/* \
+    && apt-mark auto '.*' > /dev/null \
+    && apt-mark manual $savedAptMark \
+    && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
+    && rm -rf /var/lib/apt/lists/* \
+    && perl --version
 
 WORKDIR /root
 

[tianon]

I've experimented with Alpine a bit, but I think it's going to require more love. There are a number of tests that don't pass, and https://git.alpinelinux.org/cgit/aports/tree/main/perl/APKBUILD?h=3.8-stable#n11 seems to imply that it's non-trivial to get them to (as in, muslc doesn't provide the necessary locale support for the tests to work at all), so they might have to be skipped in order to generate an Alpine build.

Here's my current WIP (with the tests commented out so it generates a usable build):

diff --git a/5.028.000-64bit/Dockerfile b/5.028.000-64bit/Dockerfile
index d009951..bb0d43b 100644
--- a/5.028.000-64bit/Dockerfile
+++ b/5.028.000-64bit/Dockerfile
@@ -1,12 +1,13 @@
-FROM buildpack-deps:stretch
+FROM alpine:3.8
 LABEL maintainer="Peter Martini <[email protected]>, Zak B. Elep <[email protected]>"
 
 COPY *.patch /usr/src/perl/
 WORKDIR /usr/src/perl
 
-RUN curl -SL https://www.cpan.org/src/5.0/perl-5.28.0.tar.xz -o perl-5.28.0.tar.xz \
+RUN apk add --no-cache --virtual .build-deps dpkg dpkg-dev gcc libc-dev make \
+    && wget https://www.cpan.org/src/5.0/perl-5.28.0.tar.xz -O perl-5.28.0.tar.xz \
     && echo '059b3cb69970d8c8c5964caced0335b4af34ac990c8e61f7e3f90cd1c2d11e49 *perl-5.28.0.tar.xz' | sha256sum -c - \
-    && tar --strip-components=1 -xaf perl-5.28.0.tar.xz -C /usr/src/perl \
+    && tar --strip-components=1 -xJf perl-5.28.0.tar.xz -C /usr/src/perl \
     && rm perl-5.28.0.tar.xz \
     && cat *.patch | patch -p1 \
     && gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
@@ -14,13 +15,15 @@ RUN curl -SL https://www.cpan.org/src/5.0/perl-5.28.0.tar.xz -o perl-5.28.0.tar.
     && archFlag="$([ "$archBits" = '64' ] && echo '-Duse64bitall' || echo '-Duse64bitint')" \
     && ./Configure -Darchname="$gnuArch" "$archFlag" -Duseshrplib -Dvendorprefix=/usr/local  -des \
     && make -j$(nproc) \
-    && TEST_JOBS=$(nproc) make test_harness \
+#    && TEST_JOBS=$(nproc) make test_harness \
     && make install \
     && cd /usr/src \
-    && curl -LO http://www.cpan.org/authors/id/M/MI/MIYAGAWA/App-cpanminus-1.7044.tar.gz \
+    && wget http://www.cpan.org/authors/id/M/MI/MIYAGAWA/App-cpanminus-1.7044.tar.gz \
     && echo '9b60767fe40752ef7a9d3f13f19060a63389a5c23acc3e9827e19b75500f81f3 *App-cpanminus-1.7044.tar.gz' | sha256sum -c - \
     && tar -xzf App-cpanminus-1.7044.tar.gz && cd App-cpanminus-1.7044 && perl bin/cpanm . && cd /root \
-    && rm -fr ./cpanm /root/.cpanm /usr/src/perl /usr/src/App-cpanminus-1.7044* /tmp/*
+    && rm -fr ./cpanm /root/.cpanm /usr/src/perl /usr/src/App-cpanminus-1.7044* /tmp/* \
+    && apk del .build-deps \
+    && perl --version
 
 WORKDIR /root
 

However, the end result is 60.5MB, so perhaps it's a worthy effort? 😄

[zakame]

@tianon nice work, for that 400MB I still had some build-deps based on Debian's control plus a few extra utils to handle bzip2/xz for the other Perl versions; on stretch-slim that went down to 335MB, but this one is certainly much better!

For alpine, indeed musl locales support was my main concern, but looks like as long as we can be very clear about that, most people won't miss it; I just don't want issues here about not being able to install Locale::* or other i18n CPAN modules. My old WIP also got somewhere between 50-60MB, so definitely good work here, thanks a lot 👍

[zakame]

I've made some builds of #54 for 5.28-slim and 5.28-slim-threaded on my hub repo, please do test and let me know how it goes:

zakame/perl     5.28-slim            0111bdf0b7df     2018-08-04 22:12:05     112MB
zakame/perl     5.28-slim-threaded   e1cf2594f70d     2018-08-04 21:28:04     112MB

I've retained make and netbase in the image, so it can still install pure Perl CPAN modules like Mojolicious and Moo.

[waterkip]

As a first test with $company project with 5.26-slim. I'm just going to list what is/was needed.

In order to install HTML::Entities (uses XS) I needed to add both gcc and libc6-dev. HTML::Entities is used by libwww-perl and Catalyst-Runtime (and ~360 others).

In order to install Net::SSLeay I needed to install gcc, libc6-dev, libssl-dev, openssl and zlib1g-dev. openssl suggests ca-certificates.

In order to use cpanm with git repos (cpanm https://host.name/project/foo.git) I needed to add git and ca-certificates.

XML::Parser (dep of XML:Twig) needs libexpat1-dev
XML::LibXML needs libxml2-dev (we already install this for one of our builds, so no problems there).

[tianon]

With other language images, that's expected for the slim variants; we're trading convenience for image size, so some additional package installs are inevitable.

[waterkip]

I know, I'm just listing them so a decision can be made if a trade off can be made. I do think it is a good idea to have a build-box image for perl and a run-time image. But since multi-stage builds aren't supported (yet), well, listing what is needed.

[zakame]

@waterkip yep, I'm testing a local 5.22-slim right now with https://github.com/metacpan/metacpan-web and it also needs a similar set of packages for XS builds.

[zakame]

@waterkip how's your testing? On my side, I ran this on both MetaCPAN services (frontend web and backend api) via https://github.com/metacpan/metacpan-docker, no issues so far.

I'll merge #54 tonight and cut a release on the Hub right after if there are no further issues or todos, thanks!

[waterkip]

My testing has stalled a bit due to work and life. And I cannot test it this week as I'm heading to Glasgow for the YAPC.

[waterkip]

Our build seems to work, I'll have a tester do a full regression test to see if everything works. Expect results before Tuesday.

[zakame]

@waterkip cool, enjoy the YAPC!

I think we can release slim now, if there are any issues from your end (as well as from others) by Tuesday, feel free to raise it on a new item here.

Thanks again everyone for participating! 🎉

[waterkip]

Cool! My tester just confirmed that there are no issues as well. So it should be good to go ;)

Thanks for making a smaller base image, much appreciated!!

[zakame]

perl:slim variants are now available on the Docker Hub, thanks again everyone! 🎉 ❤️

[pplu]

Love those new slim variants! I've just packaged an app with a slim variant, and all has been very smooth 🎉