Presto development docker images

Docker Image Names

The docker images in this repository are expected to be given names of the form prestodev/hdp2.6-hive. The Dockerfile and other files needed to build the prestodev/hdp2.6-hive image are located in the directory prestodev/hdp2.6-hive.

Generally speaking, the images should not be built manually with docker build.

Building docker images

The docker images should be built using make. To build the docker image named prestodev/hdp2.6-hive, run make prestodev/hdp2.6-hive. Make will build the image and its dependencies in the correct order.

If you want to build a base image and all the images depending on it, you can use the *.dependants targets. E.g.

make prestodev/hdp2.6-base.dependants

will build the hdp2.6-base and all the images depending on it (transitively).

Releasing (pushing) docker image

All of the docker images in the repository share the same version number. This is because most of the images depend on a parent image that is also in the repository (e.g. prestodev/hdp2.6-hive is FROM prestodev/hdp2.6-base), or are meant to be used together in testing (prestodev/hdp2.6-hive and prestodev/hdp2.6-hive-kerberized).

Having all of the images on the same version number make troubleshooting easy: If all of the docker images you are using have the same version number then they are in a consistent state.

This means that we treat the repository as a single codebase that creates multiple artifacts (Docker images) that all need to be released together.

Best practice for publishing a snapshot or release version is to use the Jenkins job. Login to Jenkins and search for docker-images. If you must publish a new version manually, follow these steps:

To release a snapshot version of the repository do the following

1. docker login
2. Verify in the Makefile that VERSION is set to something ending in -SNAPSHOT.
3. make snapshot

To release a release (final) version of the repository do the following

1. docker login
2. Verify in the Makefile that VERSION is set to something not ending in -SNAPSHOT.
3. make release

To release a snapshot or final version, you must log in to docker using the docker login command.

Typical workflow

Normally developers are working on a snapshot version of the next release, and the VERSION macro in the Makefile should be set to a snapshot version such as 35-SNAPSHOT. A typical workflow is as follows:

1. Develop changes
2. Commit changes
3. make snapshot to push snapshot releases to dockerhub as needed
4. Repeat as needed

Eventually, version 35-SNAPSHOT is ready for release. To release version 35, do the following:

1. Change VERSION to the release version: 35-SNAPSHOT -> 35
2. Commit the repository
3. make release to push the images to dockerhub
4. Change VERSION to the next snapshot version: 35 -> 36-SNAPSHOT
5. Commit the repository
6. Continue developing as described above

make snapshot does the following:

• Creates a tag for the image with the git hash of the git repository on dockerhub

make release does the following:

• Updates the 'latest' tag for the image on dockerhub
• Creates a tag for the image on dockerhub with the git hash of the git repository
• Creates a tag for the image on dockerhub with the $(VERSION) specified in the Makefile
• Creates a tag in the git repository with the name release-$(VERSION)

Several rules are enforced about the state of the repository when pushing to dockerhub:

• For a snapshot or a release, the repository must be in a clean state (no uncommitted files)
• For a release, the branch must be master

Upgrading Docker images for consumers

For a project that uses Travis for continuous integration, you can upgrade the docker images used by the project using the following process.

1. Develop locally, testing your changes
2. When you are satisfied with your changes, run make snapshot to release a snapshot build to dockerhub.
3. Create a branch of the dependent project
4. Set the tag for the images on the project to the tag of the snapshot build
5. Push the branch to github.
6. Merge your changes into docker-images/master
7. Update the release version as described above, and run make release
8. Create a PR against upstream/master that uses the new release of the docker images.
9. After Travis passes, merge the PR

Docker build arguments (ARG, --build-arg)

Docker build arguments are documented in the Dockerfile reference

Args are used by specifying the ARG directive in a Dockerfile:

ARG FOO
RUN echo $FOO >/etc/foo

The value of FOO then needs to be set in the Makefile:

FOO := Docker images build on $(shell uname -s) are superior to all others.

Note that docker build does not allow the variable reference $FOO to be written ${FOO} or $(FOO). Further note that it won't warn you about this; instead, you'll likely end up with an error later in the build or a broken image.

docker build won't let you pass --build-args that don't have a corresponding key in the Dockerfile. This means that the build system can't just pass the union of all of the --build-args needed by every Dockerfile in the repository. The build system handles this largely the same way it handles figuring out what the correct dependency order is for building the images, described below.

Build args with a default value are not handled at present. Feel free to add that functionality in bin/flag.sh if needed.

How the build system works.

At a high level, a docker image depends on two things:

1. Its Dockerfile
2. Its parent image, specified by the from FROM line in the Dockerfile.

Using the relative directory from the root of the repo as the image name, we could, in principle, write a rule of the form

prestodev/foo: prestodev/foo/Dockerfile $(extract_parent prestodev/foo/Dockerfile)
	cd prestodev/foo && docker build -t prestodev/foo .

Using automatic variables we could shorten that to the following:

prestodev/foo: $@/Dockerfile $(extract_parent $@/Dockerfile)
	cd $@ && docker build -t $@ .

This is conceptually valid, but it doesn't work: Automatic variables aren't available in the prerequisites. The solution to solve that is to use a pattern rule:

$(images): %: %/Dockerfile $(extract_parent %/Dockerfile)
	...

That almost works. Almost because you can't use the stem (%) in a function call.

Instead, we can use three features of make together to accomplish the same thing.

1. You can specify the same target multiple times with different dependencies. Make will build all of the dependencies before running the commands to build the target.
2. you can use the include directive to tell make to include another file.
3. If a file specified by an include directive doesn't exist, make will look for a rule to create that file.

prestodev/foo: prestodev/foo_parent
prestodev/foo: prestodev/foo/Dockerfile
	...

The strategy is to include a separate file that specifies the dependency on the parent image. This file isn't in the repo, so the Makefile has a rule to make it from the image's Dockerfile. The second rule specifies the dependency on the Dockerfile and builds the image using docker build.

Recursive Make Considered Harmful explains this technique in section 5.4 and applies it to C source files and the .h files they include. I've adapted it here.

The bin/depend.sh script generates a .d file in $(DEPDIR) from the Dockerfile for the image:

$(DEPDIR)/prestodev/foo.d: prestodev/foo/Dockerfile
	...

The corresponding .d file will take one of two forms:

1. if foo's parent is built from this repository

   prestodev/foo: prestodev/foo_parent
   

2. if foo's parent should be pulled from dockerhub

   prestodev/foo:
   

In the first case, make now knows that foo_parent is a dependency of foo, and builds it first.

In the second case, we don't add a dependency for make, and docker itself is responsible for pulling foo's parent from dockerhub as part of the docker build process.

A major difference between the approach explained in Recursive Make Considered Harmful is that bin/depend.sh needs to know what images the repo knows how to build so it can output the second form for parent images we don't know how to build. We do this by passing in the names of all of the images we know how to build.