Add varnish

[teohhanhui]

Reviving #1294.

Varnish Cache is a widely used caching HTTP reverse proxy.

As we've seen in #1294 (which remained open for a long time) upstream has not shown any interest.

I believe all the best practices have been sufficiently adhered to, including the Official Images Test Suite, update.sh and generate-stackbrew-library.sh.

Docs PR: docker-library/docs#1241

Checklist for Review

NOTE: This checklist is intended for the use of the Official Images maintainers both to track the status of your PR and to help inform you and others of where we're at. As such, please leave the "checking" of items to the repository maintainers. If there is a point below for which you would like to provide additional information or note completion, please do so by commenting on the PR. Thanks! (and thanks for staying patient with us ❤️)

• associated with or contacted upstream?
• does it fit into one of the common categories? ("service", "language stack", "base distribution")
• is it reasonably popular, or does it solve a particular use case well?
• does a documentation PR exist? (should be reviewed and merged at roughly the same time so that we don't have an empty image page on the Hub for very long)
• dockerization review for best practices and cache gotchas/improvements (ala the official review guidelines)?
• 2+ dockerization review?
• existing official images have been considered as a base? (ie, if foobar needs Node.js, has FROM node:... instead of grabbing node via other means been considered?)
• if FROM scratch, tarballs only exist in a single commit within the associated history?
• passes current tests? any simple new tests that might be appropriate to add? (https://github.com/docker-library/official-images/tree/master/test)

[teohhanhui]

ping @tianon @yosifkit

[teohhanhui]

Tagging @kevin1024 @avoinea @andrerom from the previous PR.

[teohhanhui]

My new attempt to contact upstream:
https://twitter.com/gquintard_/status/1011691328353292288

/cc @gquintard who wrote this blog post: https://info.varnish-software.com/blog/varnish-docker

[gquintard]

Hi, thanks for tagging me.

I had a look and only have a couple of questions/objections:

• why compile from source when we provide https://packagecloud.io/varnishcache/varnish60 and https://packagecloud.io/varnishcache/varnish41 ? To cover more platforms if people chose to tinker?
• I'd really like to see the docker run command sport the --tmpfs /var/lib/varnish:exec from the blog to avoid killing performance
• you really don't need the whole source for 99.9999999999% of the vmods anymore, just using the installed headers is enough.

[teohhanhui]

why compile from source

It seems to be the Docker way. And yes, it allows anyone to easily customize the build.

I'll look into the tmpfs.

99.9999999999%

Point taken. But seeing that the source .tgz is only ~3.0 MB, I don't think that's a major issue.

[gquintard]

My point about the vmods using the source is really more about maintainability and being correct; people will read your instructions and think that vmods still need the source, even though your instructions were already outdated when you wrote them. I mean, vmods today don't even know about VARNISHSRC anymore.

Plus, your argument about the source being small doesn't stand since you take care of removing the vmod's tgz in the Dockerfile, and that file is super small.

Other than than, no issues on the purely varnish sides of things

[teohhanhui]

Okay, I'll remove the Varnish source after compilation. 😄

[andrerom]

Biased, but would also like to see https://github.com/varnish/varnish-modules part of the image, some of the modules are becoming quite broadly used (xkey, variable, sessions, ..).

It's quite easy to install it on top of the varnish packages @gquintard recommends, example of that:
https://github.com/ezsystems/ezplatform/blob/master/doc/docker/Dockerfile-varnish

[teohhanhui]

If it's easy to install, we don't need to include it. For example, the official PHP image only includes non-default extensions that are impossible or too hard to install.

But if there's a unified way to install most Varnish modules (I doubt so), we could certainly add a helper script.

[gquintard]

Most vmods are built from the same template (vmod_example) so, with a bunch of wildcards you can get/untar/configure/build/install in fairly unified manner. Just add a variable to install dependencies, and off you go.

…

On Fri, Jun 29, 2018, 11:55 Teoh Han Hui ***@***.***> wrote: If it's easy to install, we don't need to include it. For example, the official PHP image only includes non-default extensions that are impossible or too hard to install. But if there's a unified way to install most Varnish modules (I doubt so), we could certainly add a helper script. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4404 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADmgKcw-7zTegHIne3Zh1EAtKZ7oq_83ks5uBgeegaJpZM4UR5U7> .

[teohhanhui]

@gquintard

Building requires the Varnish header files and uses pkg-config to find the necessary paths.

from https://github.com/varnishcache/libvmod-example

So we still need to keep the source around...

EDIT: Hmm, I guess I was mistaken... The header files are in /usr/local/include/varnish

[teohhanhui]

All issues raised by @gquintard have been resolved. 🎉

[teohhanhui]

@andrerom Alpine variant added. 🎉

Also, the debian variant is now 33% smaller.

[docker-library-bot]

Hello! ✨

Thanks for your interest in contributing to the official images program. 💭

As you may have noticed, we've usually got a pretty decently sized queue of new images (not to mention image updates and maintenance of images under @docker-library which are maintained by the core official images team). As such, it may be some time before we get to reviewing this image (image updates get priority both because users expect them and because reviewing new images is a more involved process than reviewing updates), so we apologize in advance! Please be patient with us (and avoid poking us about your image via other communication means -- rest assured, we've seen your PR and it's in the queue). ❤️

We do try to proactively add and update the "new image checklist" on each PR, so if you haven't looked at it yet, that's a good use of time while you wait. ☔

Thanks! 💖 💙 💚 ❤️

[teohhanhui]

and avoid poking us about your image via other communication means

Oops! Sorry...

[tianon]

Thanks @docker-library-bot ❤️

why compile from source

It seems to be the Docker way.

Just to be clear, compiling from source isn't "the Docker way" -- for all official images, we explicitly recommend following upstream's recommendations. For example, postgres installs the PostgreSQL packages built/provided by upstream (and only builds them from their source for architectures that upstream doesn't publish packages for). We do tend to have a lot of upstreams that don't publish their own recommended packages, so compiling from source is often the "recommended" path from the upstream.

A few other initial notes after taking a first pass over the Dockerization:

1. are gcc, libc-dev, and libgcc needed at runtime for Varnish, or would they be more appropriate in VMOD_BUILD_DEPS? (are they used in VCC-compiler? I'm not terribly familiar with how Varnish works under the hood)

2. is there a default example .vcl file that makes sense to ship? (it's OK if the answer is no, for example haproxy doesn't ship one because there isn't really any default that makes sense for even a minority of users and I understand it has a somewhat similar use to Varnish)

3. are the patches in /varnish-alpine-patches/ temporary (as in, something upstream is considering including and/or handling), or are those expected to stay something specific to the Docker image?

4. --without-jemalloc strikes me as odd -- if upstream defaults to using jemalloc, shouldn't the Docker image also?

5. setting the default args via entrypoint only if there aren't any args is kind of odd -- that means that users who do docker run ... varnish will get a very different set of arguments from users who do docker run ... varnish -f ...; I'd recommend either:

   • if varnishd can accept a later argument that overwrites the value of any of -F, -a, -f, or -s (or any of those should just always be set for Docker use anyhow), just always add them before what the user specifies
   • put them in CMD [] instead so that users can use docker inspect to determine the default set of arguments (using environment variables to change arguments to the application is a bit odd, we don't have many official images currently which do so)

[teohhanhui]

1. Yes, they're runtime dependencies. At first I had the same thought too...

2. I don't think so, as we'll have nothing to use as the default "backend"?

3. All the patches are taken directly from Alpine. I think upstream has been working on improving musl libc support, but obviously it's still not perfect.

4. Because compilation fails with jemalloc, and the Alpine guys haven't cracked that one yet.

[gquintard]

3. in master, all patches are now superfluous

[teohhanhui]

put them in CMD [] instead so that users can use docker inspect to determine the default set of arguments (using environment variables to change arguments to the application is a bit odd, we don't have many official images currently which do so)

But wouldn't that make for very bad DX?

EDIT: I've made the CMD as minimal as possible.

[teohhanhui]

@tianon @gquintard Is it important to use upstream built packages in this case? I looked at the postgres Dockerfile, and it seems to add a lot of complexity...

[gquintard]

it's not that it important, it's just that's it way faster, for rom, the Dockerfile is literally 5 lines long:

FROM centos:7

RUN yum install -y epel-release && \
    curl -s https://packagecloud.io/install/repositories/varnishcache/varnish60/script.rpm.sh | bash && \
    yum install -y varnish

You get the init scripts, libs are packaged in a devel package if you want to build vmods, I honestly don't see the complexity here

[gquintard]

oh, and we have weekly packages too!

[teohhanhui]

@gquintard Init scripts are not useful in a Docker container. Well, the complexity is due to the fact that official Docker images are built for multiarch. And as we can see in the postgres Dockerfile, it's not easy / obvious to see what's going on with all the detection logic.

[teohhanhui]

@gquintard @tianon A point against using the upstream packages:

https://packagecloud.io/varnishcache/varnish62 is still empty at time of writing, 2 weeks after Varnish 6.2 was released.

[teohhanhui]

Also, it seems like Varnish 6.2.0 still requires patches on Alpine:

https://git.alpinelinux.org/aports/tree/main/varnish?id=d1ee09bdd4d8b028c154d428eabebaa10d9f9686

[gquintard]

not sure those are still necessary, but I'll check

[teohhanhui]

@tianon Do you see any outstanding issues? What do I need to do to get the Dockerization review moving?

[gquintard]

https://packagecloud.io/varnishcache/varnish62 is still empty at time of writing, 2 weeks after Varnish 6.2 was released.

fixed

[teohhanhui]

@gquintard Yeah, I've noticed. Is it automated now?

[gquintard]

@teohhanhui, it was already automated, but something broke the pipeline and we didn't notice, this shouldn't happen anymore.

[teohhanhui]

Do you think it's important to switch to the official package? I'll look into it if you think it's necessary.

[gquintard]

It's really just that the official repo is:

• official
• dead simple to install

If there's any issue with it on Docker, I'd rather have the official Docker images use them so we know fast when something goes wrong.

Past that, it's your choice

[teohhanhui]

@gquintard @tianon I've done it at coopTilleuls/docker-varnish#42

Would appreciate your review.

I have some concerns about the inconsistency this has caused between both variants of the image though... For example, /var/lib/varnish vs /usr/local/var/varnish

[gquintard]

I'm going to declare myself incompetent on the matter, somehow coopTilleuls/docker-varnish#42 adds more code, and I have no intention of digging through it. The mere fact that the Dockerfile is creating a local APT repo is terrifying.

I have some concerns about the inconsistency this has caused between both variants of the image though... For example, /var/lib/varnish vs /usr/local/var/varnish

Fix the custom built packages to do the same thing as the prepackaged versions? Or just don't build from source?

[teohhanhui]

The mere fact that the Dockerfile is creating a local APT repo is terrifying.

That's only for architectures other than amd64, where there is no prebuilt binary.

Or just don't build from source?

Sure, if you provide prebuilt binaries that work on Alpine. I'm not sure if we should install from the official Alpine repository, as chances are they'd be a bit behind.

[gquintard]

Sure, if you provide prebuilt binaries that work on Alpine. I'm not sure if we should install from the official Alpine repository, as chances are they'd be a bit behind.

best way I found is to checkout the aports and use the APKBUILD. alpine won't hold onto old packages, but we can reuse their knowledge

please have a look at this: coopTilleuls/docker-varnish/pull/43 , it's certainly not perfect, but it should make things more manageable (I'm monitoring travis to see if I missed anything)

That's only for architectures other than amd64, where there is no prebuilt binary.

While I appreciate the will to cover all the architectures, Varnish was built (no pun intended) from the ground up for 64bits architectures, so the others architectures you can target are arm64, which is possibly a bit out of the ordinary. In any case, I'd rather go for good enough and try to add support for that one later, if there's demand for it.

[yosifkit]

Closing in favor of #5855.