thredds-docker and base infrastructure security updates

[lesserwhirls]

Occasionally we post things to the thredds user email list regarding tomcat (end of life announcements, for example), but we strongly encourage those who run a TDS to subscribe directly to the security list associated with the servlet container and front-end web server (if applicable) they have chosen to run on their systems, at a minimum, as those are the only true authoritative source regarding security information. With the docker container, however, we at Unidata have made one choice up front -- thou shalt run tomcat -- and the containers we rely upon have made some choices as well -- thou shalt run with AdoptOpenJDK binaries -- so it's a little different.

So that brings up the question of how users of our docker containers would know that they should pull a new image to pick up base infrastructure security updates, and how they could verify that the new image contains the latest and greatest patched version of the base infrastructure components (Tomcat, java, etc).

After speaking with @julienchastang yesterday, we came up with something along these lines.

First, we re-institute the auto-builds for any of the Unidata managed containers when an upstream container updates. To break down what that means, let's consider the linage of the thredds-docker container.

• Our thredds-docker Dockerfile uses unidata/tomcat-docker:8.5, which builds upon:
  • tomcat:8.5-jdk8-openjdk, which uses:
    • openjdk-8 (which uses binaries from the AdoptOpenJDK project - see OpenJDK 8u212 and 11.0.3 image quality concerns docker-library/openjdk#320 (comment) about what that means).

It goes a little deeper from there, but basically what we want is for new docker images for Unidata/tomcat-docker, Unidata/thredds-docker, etc. to get pushed out to Dockerhub if and when any of these upstream containers update so that upstream security fixes are available to our users as soon as possible.

So, when would someone know to update their docker containers?

1. AdoptOpenJDK releases on a quarterly cycle (the same established quarterly cycle used by Oracle i.e. "the Tuesday closest to the 17th day of January, April, July and October."), and so at a minimum users should plan to pull or rebuild and redeploy their images/systems/etc. on that cycle.
2. Next, we should strongly recommend that they subscribe directly to tomcat-announce, at a minimum, to keep up-to-date on security issues related to tomcat. Security issues would not be announced on that list without Tomcat's own containers being updated first, and with auto-builds enabled, ours would be updated as well.

So...when? Again, at a minimum, "the Tuesday closest to the 17th day of January, April, July and October" plus any time a security message regarding tomcat goes out over the tomcat-announce email list. At. A. Minimum.

How do I know things are really up-to-date?

1. For those wishing to verify that Unidata containers performed an auto-build cycle and that they will be getting the latest and greatest when they pull, they may inspect the build information on Dockerhub:
   1. Visit the thredds-docker tags page
   2. Select a tag of interest (for example, the latest) tag.
   3. Inspect the build information (as seen in the link above, line 23 shows ENV TOMCAT_VERSION=8.5.53)

Thoughts?

[tc33133]

Sean, Thanks for starting this discussion. I like the idea of you pushing updates to dockerhub, but also want to say I depend on notifications especially for security related stuff. Personally, I think the best way for a user like me to be notified to update is from the thredds listserv or a thredds-docker listserv. Even if I'm aware of a tomcat issue from the tomcat listserv, it will be some delay before the container is updated. Obviously, I will need to keep up with my webserver security on my own, but since tomcat is packaged with thredds I would appreciate a notification from thredds(-docker) listserv when the container is updated. Thank you and @julienchastang <https://github.com/julienchastang> for your work on this Tom

…

On Wed, Mar 25, 2020 at 8:50 AM Sean Arms ***@***.***> wrote: Occasionally we post things to the thredds user email list regarding tomcat (end of life announcements, for example), but we strongly encourage those who run a TDS to subscribe directly to the security list associated with the servlet container and front-end web server (if applicable) they have chosen to run on their systems, at a minimum, as those are the only true authoritative source regarding security information. With the docker container, however, we at Unidata have made one choice up front -- *thou shalt run tomcat* -- and the containers we rely upon have made some choices as well -- *thou shalt run with AdoptOpenJDK binaries* -- so it's a little different. So that brings up the question of how users of our docker containers would know that they should pull a new image to pick up base infrastructure security updates, and how they could verify that the new image contains the latest and greatest patched version of the base infrastructure components (Tomcat, java, etc). After speaking with @julienchastang <https://github.com/julienchastang> yesterday, we came up with something along these lines. First, we re-institute the auto-builds for any of the Unidata managed containers when an upstream container updates. To break down what that means, let's consider the linage of the thredds-docker container. - Our thredds-docker Dockerfile <https://github.com/Unidata/thredds-docker/blob/master/Dockerfile> uses unidata/tomcat-docker:8.5 <https://github.com/Unidata/tomcat-docker/blob/master/Dockerfile>, which builds upon: - tomcat:8.5-jdk8-openjdk <https://github.com/docker-library/tomcat/blob/master/8.5/jdk8/openjdk/Dockerfile>, which uses: - openjdk-8 <https://github.com/docker-library/openjdk/blob/master/8/jdk/Dockerfile> (which uses binaries from the AdoptOpenJDK project - see docker-library/openjdk#320 (comment) <docker-library/openjdk#320 (comment)> about what that means). It goes a little deeper from there, but basically what we want is for new docker images for Unidata/tomcat-docker, Unidata/thredds-docker, etc. to get pushed out to Dockerhub if and when *any* of these upstream containers update so that upstream security fixes are available to our users as soon as possible. So, when would someone know to update their docker containers? 1. AdoptOpenJDK releases on a quarterly cycle <https://adoptopenjdk.net/support.html#roadmap> (the same established quarterly cycle used by Oracle i.e. "the Tuesday closest to the 17th day of January, April, July and October."), and so *at a minimum* users should plan to pull or rebuild and redeploy their images/systems/etc. on that cycle. 2. Next, we should strongly recommend that they subscribe directly to tomcat-announce <http://tomcat.apache.org/lists.html#tomcat-announce>, at a minimum, to keep up-to-date on security issues related to tomcat. Security issues would not be announced on that list without Tomcat's own containers being updated first, and with auto-builds enabled, ours would be updated as well. So...when? Again, at a minimum, "the Tuesday closest to the 17th day of January, April, July and October" plus any time a security message regarding tomcat goes out over the tomcat-announce email list. At. A. Minimum. How do I know things are really up-to-date? 1. For those wishing to verify that Unidata containers performed an auto-build cycle and that they will be getting the latest and greatest when they pull, they may inspect the build information on Dockerhub: 1. Visit the thredds-docker tags <https://hub.docker.com/r/unidata/thredds-docker/tags> page 2. Select a tag of interest (for example, the latest <https://hub.docker.com/layers/unidata/thredds-docker/latest/images/sha256-61bad19a48752fbf283bd1271c639dcb0fb3e6c672e3ac5fca949bae7dfd4d6b?context=explore>) tag. 3. Inspect the build information (as seen in the link above, line 23 shows ENV TOMCAT_VERSION=8.5.53) Thoughts? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#242>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGINLK7CNDDLCIQJEYFU6W3RJIR3PANCNFSM4LTRUFDQ> .

[lesserwhirls]

Thank you for your feedback @tc33133!

Even if I'm aware of a tomcat issue from the tomcat listserv, it will be some delay before the container is updated.

Based on a comparison of the history of the official Tomcat Docker image we build from and the timing of the announcements on tomcat-announce list, the new tomcat docker images are made available before any announcements are made. If we have auto builds up and running, the new thredds-tomcat docker images would be ready before the announcement from tomcat-announce hit your inbox, so there shouldn't be any delay.

Obviously, I will need to keep up with my webserver security on my own, but since tomcat is packaged with thredds I would appreciate a notification from thredds(-docker) listserv when the container is updated.

I can appreciate that, and certainly if changes are made within our Dockerfiles that result in new containers, we should document and announce those (since I'm not directly involved in the management of the containers, I'm unsure of how things currently work).

If our containers get updated due to upstream patches and releases (without any changes to our Dockerfile or base configuration files), I'm not sure an announcement from us could happen more timely or be more informative than what the official tomcat list sends out. Again, once auto builds are in place, we would have updated containers within minutes of the official tomcat containers being available.

[julienchastang]

Sadly, I came across this though I have not given up on this idea yet: docker/hub-feedback#1717. As I understand, base image changes were triggering a huge number of builds so this feature was removed, unfortunately. Still, we may be able to do something from cron as similar to here: chialab/docker-php#41

Edit 5/21/2020: Also related https://stackoverflow.com/questions/26423515/how-to-automatically-update-your-docker-containers-if-base-images-are-updated

Edit 7/24/2020: DockerHub may have reinstated triggered autobuilds based on upstream changes:

Though it appears this will only work for "non-official" images and Tomcat is an official image :-(