How the Magic Works

[edwarnicke]

How the magic works

This repo utilizes aggressively a number of multi-stage Docker tricks that are worth explaining, because they can be used broadly

We are doing three things in the Dockerfile, denoted by their named stage:

1. Building small vpp and vpp-dbg images, including the ability to patch
2. generating the govpp 'binapi' for the build vpp image, using the standard 'go:generate' idiom
3. Extracting the VPP_VERSION being used so we can utilize it to tag published docker images.

Building small vpp images

Ultra small vpp images are built via three Docker stages:

1. vppbuild
2. vppinstall
3. vpp

The reason for these stages is to cauterize the bloat from each activity.

Building vpp means bloating an image
up with a bunch of build dependencies, build artifacts, etc. Building vpp installation on top of that would
lead to a multi-GB image, which is undesirable. So we isolate that work in the 'vppbuild' stage.

Installing vpp means copying the resulting *.deb packages from the 'vppbuild' stage and installing them. Unfortunately,
because you can't combine a COPY and RUN step, that results in image bloat from the *.deb files themselves, so we isolate
that in the 'vppinstall' image.

Finally, we utilize a trick to trim out the bloat when building the 'vpp' stage.

The 'vppbuild' stage

govpp/Dockerfile

Lines 10 to 20 in f9c1af6

	FROM ubuntu:${UBUNTU_VERSION} as vppbuild
	ARG VPP_VERSION
	RUN apt-get update
	RUN DEBIAN_FRONTEND=noninteractive TZ=US/Central apt-get install -y git make python3 sudo asciidoc
	RUN git clone -b ${VPP_VERSION} https://github.com/FDio/vpp.git
	WORKDIR /vpp
	COPY patch/ patch/
	RUN git apply patch/*.patch
	RUN DEBIAN_FRONTEND=noninteractive TZ=US/Central UNATTENDED=y make install-dep
	RUN make pkg-deb
	RUN ./src/scripts/version > /vpp/VPP_VERSION

is a fairly standard Ubuntu oriented build. It results in a bunch of *.deb files.

The 'vppinstall' stage

govpp/Dockerfile

Lines 22 to 30 in f9c1af6

	FROM ubuntu:${UBUNTU_VERSION} as vppinstall
	ARG VPP_VERSION
	COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/
	COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"]
	ARG VPP_VERSION
	RUN VPP_INSTALL_SKIP_SYSCTL=false apt install -f -y --no-install-recommends /pkg/*.deb ca-certificates iputils-ping iproute2 tcpdump; \
	rm -rf /var/lib/apt/lists/*; \
	rm -rf /pkg

uses

govpp/Dockerfile

Line 24 in f9c1af6

COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/

to copy in the indexes in /var/lib/apt/lists/ that result from having run apt-get update in the 'vppbuild' stage,
thus avoiding the cost of redownloading them.

govpp/Dockerfile

Line 25 in f9c1af6

COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"]

copies the *.deb files we wish to install from where they were built in 'vppbuild'

govpp/Dockerfile

Line 27 in f9c1af6

RUN VPP_INSTALL_SKIP_SYSCTL=false apt install -f -y --no-install-recommends /pkg/*.deb ca-certificates iputils-ping iproute2 tcpdump; \

installs the *.deb files:

1. -f causes apt-get to install any missing dependencies.
2. -y causes apt-get to run in an unattended mode where the answer to the questions are y
3. --no-install-recommends causes apt-get to only install required (rather than recommended) dependencies to keep the image size down

govpp/Dockerfile

Line 28 in f9c1af6

rm -rf /var/lib/apt/lists/*; \

removes the apt indexes from apt-get update

and

govpp/Dockerfile

Line 29 in f9c1af6

rm -rf /pkg

removes the *.deb files

There is one problem with this. Because the image still has the layers from

govpp/Dockerfile

Lines 24 to 25 in f9c1af6

	COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/
	COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"]

the 'vppinstall' stage is still going to be bloated by that amount. We solve this in the 'vpp' stage

The 'vpp' stage

The 'vpp' stage is our final lean runnable. It uses a very simple but slick trick, which has a small caveat to it.

govpp/Dockerfile

Lines 31 to 32 in f9c1af6

	FROM ubuntu:${UBUNTU_VERSION} as vpp
	COPY --from=vppinstall / /

Simply starts from 'ubuntu:${UBUNTU_VERSION}' (thus reusing the layers form that standard image) and then
copies the entire '/' directory in from 'vppinstall' (which has removed the apt-get indexes and *.deb files).

Because docker COPY is generally smart enough to only copy in the changed files... the layer resulting from

govpp/Dockerfile

Line 32 in f9c1af6

COPY --from=vppinstall / /

should only contain the deltas between the final result of 'vppinstall' and the starting point of 'ubuntu:${UBUNTU_VERSION}'

Resulting in something that looks like:

docker history ghcr.io/edwarnicke/govpp/vpp:v20.09                                                                                                                                                                                                                           ──(Sun,Jan31)─┘
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
8b4ea0febd25   42 minutes ago   /bin/sh -c #(nop) COPY dir:b8f7abee062c48863…   96.2MB    
<missing>      10 days ago      /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B        
<missing>      10 days ago      /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B        
<missing>      10 days ago      /bin/sh -c [ -z "$(apt-get indextargets)" ]     0B        
<missing>      10 days ago      /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   811B      
<missing>      10 days ago      /bin/sh -c #(nop) ADD file:2a90223d9f00d31e3…   72.9MB    

The caveat is, there is a bug in docker that will, depending on the storage driver
you are using, copy over all the files, resulting in a much larger image. Fortunately, building in GitHub Actions does not
seem to hit this issue. Unfortunately, building in Docker for Mac does.
To fix the issue in Docker for Mac, follow the instructions for setting Docker Engine options
and set your storage driver to 'overlay'

{
  "storage-driver": "overlay"
}

Building idiomatic go:generate for 'binapi'

In go, we generally generate code using a //go:generate directive.

In the case of govpp, you need to use the 'binapi-generator' run against the json api files installed by the vpp debs in /usr/share/vpp/api/.
We utilize Docker to make this easy by:

1. Building 'binapi-generator' as a Docker stage
2. Creating a 'gen' Docker stage that copies in /usr/share/vpp/api/ from our 'vpp' stage and run's binapi, outputting to /gen
3. Add a //go:generate line to gen.go to run the 'gen' stage to generate the code.

The 'bin-apigenerator' stage

govpp/Dockerfile

Lines 41 to 46 in f9c1af6

	FROM golang:1.15.3-alpine3.12 as binapi-generator
	ENV GO111MODULE=on
	ENV CGO_ENABLED=0
	ENV GOBIN=/bin
	ARG GOVPP_VERSION
	RUN go get git.fd.io/govpp.git/cmd/binapi-generator@${GOVPP_VERSION}

Is a pretty standard 'go get to build' stage for 'binapi-generator'

The 'gen' stage

govpp/Dockerfile

Lines 48 to 53 in f9c1af6

	FROM alpine:3.12 as gen
	COPY --from=vpp /usr/share/vpp/api/ /usr/share/vpp/api/
	COPY --from=binapi-generator /bin/binapi-generator /bin/binapi-generator
	COPY --from=vppbuild /vpp/VPP_VERSION /VPP_VERSION
	WORKDIR /gen
	CMD VPP_VERSION=$(cat /VPP_VERSION) binapi-generator ${PKGPREFIX+-import-prefix ${PKGPREFIX}}

Actually performs the generation if 'docker run'

govpp/Dockerfile

Line 49 in f9c1af6

COPY --from=vpp /usr/share/vpp/api/ /usr/share/vpp/api/

copies in the '/usr/share/vpp/api/*' json api files from the 'vpp' stage

govpp/Dockerfile

Line 50 in f9c1af6

COPY --from=binapi-generator /bin/binapi-generator /bin/binapi-generator

copies in the 'binapi-generator' from the 'binapi-generator' stage

govpp/Dockerfile

Line 51 in f9c1af6

COPY --from=vppbuild /vpp/VPP_VERSION /VPP_VERSION

copies in the VPP_VERSION we had stashed in the 'vppbuild' stage.

govpp/Dockerfile

Lines 52 to 53 in f9c1af6

	WORKDIR /gen
	CMD VPP_VERSION=$(cat /VPP_VERSION) binapi-generator ${PKGPREFIX+-import-prefix ${PKGPREFIX}}

sets the Workdir to /gen and runs the binapi-generator.

VPP_VERSION=$(cat /VPP_VERSION) - sets the VPP_VERSION env binapi-generator needs from the stashed VPP_VERSION value

${PKGPREFIX+-import-prefix ${PKGPREFIX}} - will output -import-prefix ${PKGPREFIX} if the PKGPREFIX env variable is set, and nothing otherwise.

The //go:generate directive

govpp/gen.go

Line 19 in f9c1af6

//go:generate bash -c "docker run -e PKGPREFIX=$(go list)/binapi -v $(go list -f '{{ .Dir }}'):/gen $(docker build . -q --build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git))"

uses 'docker run' to run the 'gen' stage and generate the code

1. bash bash -c "docker run ... " is used to give us a shell to work with (because we are doing a lot of magic here)
2. -e PKGPREFIX=$(go list)/binapi - sets the env variable PKGPREFIX in the docker container to the value of
   $(go list)/binapi. $(go list) is the value of the package in which the gen.go file resides.
3. -v $(go list -f '{{ .Dir }}'):/gen mounts $(go list -f '{{ .Dir }}') from the host into /gen/ in the container.
   1. $(go list -f '{{ .Dir }}') outputs the directory the module containing gen.go is in.
4. $(docker build . -q --build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git))
   1. -q - 'quiet' - outputs the id of the resulting image built
   2. --build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git)) - sets the build-arg GOVPP_VERSION
   3. $(go list -m -f '{{ .Version }}' git.fd.io/govpp.git) - outputs the version of git.fd.io/govpp.git in the go.mod file.

Extracting the VPP_VERSION being used in the Dockerfile

In .github/workflows/ci.yaml we want to be able to tag and push images based
on the VPP_VERSION. Because we only wish to specify the VPP_VERSION once in the Dockerfile, we have a job:

govpp/Dockerfile

Lines 5 to 8 in f9c1af6

	FROM ubuntu:${UBUNTU_VERSION} as version
	ARG VPP_VERSION
	ENV VPP_VERSION ${VPP_VERSION}
	CMD echo ${VPP_VERSION}

that can be used to extract that version:

govpp/.github/workflows/ci.yaml

Line 92 in f9c1af6

VPP_VERSION=$(docker run $(docker build -q . --target version))