Reworked docker docs

docs/developer-guides/Scripting_Docker.md

@@ -11,10 +11,13 @@
     - [Versioning](#versioning)
     - [Variables](#variables)
     - [Platform architecture](#platform-architecture)
+    - [Dockerignore file](#dockerignore-file)
   - [FAQ](#faq)
 
 ## Overview
 
+This document provides instructions on how to build Docker images using our automated build process. You'll learn how to specify version tags, commit changes, and understand the build output.
+
 Docker is a tool for developing, shipping and running applications inside containers for Serverless and Kubernetes-based workloads. It has grown in popularity due to its ability to address several challenges faced by engineers, like:
 
 - **Consistency across environments**: One of the common challenges in software development is the "it works on my machine" problem. Docker containers ensure that applications run the same regardless of where the container is run, be it a developer's local machine, a test environment or a production server.
@@ -43,7 +46,6 @@ Here are some key features built into this repository's Docker module:
 - Incorporates metadata through `Dockerfile` labels for enhanced documentation and to conform to standards
 - Integrates a linting routine to ensure `Dockerfile` code quality
 - Includes an automated test suite to validate Docker scripts
-- Provides a ready-to-run example to demonstrate the module's functionality
 - Incorporates a best practice guide
 
 ## Key files
@@ -61,50 +63,108 @@ Here are some key features built into this repository's Docker module:
   - [`docker.test.sh`](../../scripts/docker/tests/docker.test.sh): Main file containing all the tests
   - [`Dockerfile`](../../scripts/docker/tests/Dockerfile): Image definition for the test suite
   - [`VERSION`](../../scripts/docker/tests/VERSION): Version patterns for the test suite
-- Usage example
-  - Python-based example [`hello_world`](../../scripts/docker/examples/python) app showing a multi-staged build
-  - A set of [make targets](https://github.com/nhs-england-tools/repository-template/blob/main/scripts/docker/docker.mk#L18) to run the example
 
 ## Usage
 
 ### Quick start
 
-Run the test suite:
+The Repository Template assumes that you will want to build more than one docker image as part of your project.  As such, we do not use a `Dockerfile` at the root of the project.  Instead, each docker image that you create should go in its own folder under `infrastructure/images`.  So, if your application has a docker image called `my-shiny-app`, you should create the file `infrastructure/images/my-shiny-app/Dockerfile`.  Let's do that.
+
+First, we need an application to package.  Let's do the simplest possible thing, and create a file called `main.py` in the root of the template with a familiar command in it:
+
+```python
+print("hello world")
+```
+
+Run this command to make the directory:
+
+```shell
+mkdir -p infrastructure/images/my-shiny-app
+```
+
+Now, edit `infrastructure/images/my-shiny-app/Dockerfile` and put this into it:
+
+```dockerfile
+FROM python
+
+COPY ./main.py .
+
+CMD ["python", "main.py"]
+```
+
+Note the paths in the `COPY` command.  The `Dockerfile` is stored in a subdirectory, but when `docker` runs it is executed in the root of the repository so that's where all paths are relative to.  This is because you can't `COPY` from parent directories. `COPY ../../main.py .` wouldn't work.
+
+The name of the folder is also significant. It should match the name of the docker image that you want to create.  With that name, you can run the following `make` task to run `hadolint` over your `Dockerfile` to check for common anti-patterns:
+
+```shell
+ $ DOCKER_IMAGE=my-shiny-app make docker-lint
+/workdir/./infrastructure/images/my-shiny-app/Dockerfile.effective:1 DL3006 warning: Always tag the version of an image explicitly
+make[1]: *** [scripts/docker/docker.mk:34: _docker] Error 1
+make: *** [scripts/docker/docker.mk:20: docker-lint] Error 2
+```
+
+All the provided docker `make` tasks take the `DOCKER_IMAGE` parameter.
+
+`hadolint` found a problem, so let's fix that.  It's complaining that we've not specified which version of the `python` docker container we want. Change the first line of the `Dockerfile` to:
+
+```dockerfile
+FROM python:3.12-slim-bookworm
+```
+
+Run `DOCKER_IMAGE=my-shiny-app make docker-lint` again, and you will see that it is silent.
+
+Now let's actually build the image.  Run the following:
+
+```shell
+DOCKER_IMAGE=my-shiny-app make docker-build
+```
+
+And now we can run it:
+
+```shell
+ $ DOCKER_IMAGE=my-shiny-app make docker-run
+hello world
+```
+
+If you list your images, you'll see that the image name matches the directory name under `infrastructure/images`:
 
 ```shell
-$ make docker-test-suite-run
+ $ docker image ls
+REPOSITORY                   TAG                 IMAGE ID      CREATED        SIZE
+localhost/my-shiny-app       latest              6a0adeb5348c  2 hours ago    135 MB
+docker.io/library/python     3.12-slim-bookworm  d9f1825e4d49  5 weeks ago    135 MB
+localhost/hadolint/hadolint  2.12.0-alpine       19b38dcec411  16 months ago  8.3 MB
+```
 
-test-docker-build PASS
-test-docker-test PASS
-test-docker-run PASS
-test-docker-clean PASS
+Your process might want to add specific tag formats so you can identify docker images by date-stamps, or git hashes.  The Repository Template supports that with a `VERSION` file.  Create a new file called `infrastructure/images/my-shiny-app/VERSION`, and put the following into it:
+
+```text
+${yyyy}${mm}${dd}-${hash}
 ```
 
-Run the example:
+Now, run the `docker-build` command again, and towards the end of the output you will see something that looks like this:
 
 ```shell
-$ make docker-example-build
+Successfully tagged localhost/my-shiny-app:20240314-07ee679
+```
 
-#0 building with "desktop-linux" instance using docker driver
-...
-#12 DONE 0.0s
+Obviously the specific values will be different for you.  See the Versioning section below for more on this.
 
-$ make docker-example-run
+It is usually the case that there is a specific image that you will most often want to build, run, and deploy.  You should edit the root-level `Makefile` to document this and to provide shortcuts.  Edit `Makefile`, and change the `build` task to look like this:
 
- * Serving Flask app 'app'
- * Debug mode: off
-WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
- * Running on all addresses (0.0.0.0)
- * Running on http://127.0.0.1:8000
- * Running on http://172.17.0.2:8000
-Press CTRL+C to quit
+```make
+build: # Build the project artefact @Pipeline
+	DOCKER_IMAGE=my-shiny-app
+	make docker-build
 ```
 
+Now when you run `make build`, it will do the right thing.  Keeping this convention consistent across projects means that new starters can be on-boarded quickly, without needing to learn a new set of conventions each time.
+
 ### Your image implementation
 
-Always follow [Docker best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) while developing images. Start with creating your container definition for the service and store it in the `infrastructure/images` directory.
+Always follow [Docker best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) while developing images.
 
-Here is a step-by-step guide:
+Here is a step-by-step guide for an image which packages a third-party tool.  It is mostly similar to the example above, but demonstrates the `.tool-versions` mechanism.
 
 1. Create `infrastructure/images/cypress/Dockerfile`
 
@@ -212,6 +272,10 @@ Set the `docker_image` or `DOCKER_IMAGE` variable for your image. Alternatively,
 
 For cross-platform image support, the `--platform linux/amd64` flag is used to build Docker images, enabling containers to run without any changes on both `amd64` and `arm64` architectures (via emulation).
 
+### Dockerignore file
+
+If you need to exclude files from a `COPY` command, put a [`Dockerfile.dockerignore`](https://docs.docker.com/build/building/context/#filename-and-location) file next to the relevant `Dockerfile`.  They do not live in the root directory.  Any paths within `Dockerfile.dockerignore` must be relative to the repository root.
+
 ## FAQ
 
 1. _We built our serverless workloads based on AWS Lambda and package them as `.zip` archives. Why do we need Docker?_

scripts/docker/docker.lib.sh

@@ -27,10 +27,9 @@ function docker-build() {
 
   version-create-effective-file
   _create-effective-dockerfile
-  # The current directory must be changed for the image build script to access
-  # assets that need to be copied
-  current_dir=$(pwd)
-  cd "$dir"
+
+  tag=$(_get-effective-tag)
+
   docker build \
     --progress=plain \
     --platform linux/amd64 \
@@ -43,16 +42,36 @@ function docker-build() {
     --build-arg GIT_COMMIT_HASH="$(git rev-parse --short HEAD)" \
     --build-arg BUILD_DATE="$(date -u +"%Y-%m-%dT%H:%M:%S%z")" \
     --build-arg BUILD_VERSION="$(_get-effective-version)" \
-    --tag "${DOCKER_IMAGE}:$(_get-effective-version)" \
+    --tag "${tag}" \
     --rm \
     --file "${dir}/Dockerfile.effective" \
     .
-  cd "$current_dir"
+
   # Tag the image with all the stated versions, see the documentation for more details
   for version in $(_get-all-effective-versions) latest; do
-    docker tag "${DOCKER_IMAGE}:$(_get-effective-version)" "${DOCKER_IMAGE}:${version}"
+    if [ ! -z "$version" ]; then
+      docker tag "${tag}" "${DOCKER_IMAGE}:${version}"
+    fi
   done
-  docker rmi --force "$(docker images | grep "<none>" | awk '{print $3}')" 2> /dev/null ||:
+}
+
+# Create the Dockerfile.effective file to bake in version info
+# Arguments (provided as environment variables):
+#   dir=[path to the Dockerfile to use, default is '.']
+function docker-bake-dockerfile() {
+
+  local dir=${dir:-$PWD}
+
+  version-create-effective-file
+  _create-effective-dockerfile
+}
+
+# Run hadolint over the generated Dockerfile.
+# Arguments (provided as environment variables):
+#  dir=[path to the image directory where the Dockerfile.effective is located, default is '.']
+function docker-lint() {
+  local dir=${dir:-$PWD}
+  file=${dir}/Dockerfile.effective ./scripts/docker/dockerfile-linter.sh
 }
 
 # Check test Docker image.
@@ -81,12 +100,13 @@ function docker-check-test() {
 function docker-run() {
 
   local dir=${dir:-$PWD}
+  local tag=$(dir="$dir" _get-effective-tag)
 
   # shellcheck disable=SC2086
   docker run --rm --platform linux/amd64 \
     ${args:-} \
-    "${DOCKER_IMAGE}:$(dir="$dir" _get-effective-version)" \
-    ${cmd:-}
+    "${tag}" \
+    ${DOCKER_CMD:-}
 }
 
 # Push Docker image.
@@ -114,7 +134,8 @@ function docker-clean() {
   done
   rm -f \
     .version \
-    Dockerfile.effective
+    Dockerfile.effective \
+    Dockerfile.effective.dockerignore
 }
 
 # Create effective version from the VERSION file.
@@ -207,6 +228,13 @@ function _create-effective-dockerfile() {
 
   local dir=${dir:-$PWD}
 
+  # If it exists, we need to copy the .dockerignore file to match the prefix of the
+  # Dockerfile.effective file, otherwise docker won't use it.
+  # See https://docs.docker.com/build/building/context/#filename-and-location
+  # If using podman, this requires v5.0.0 or later.
+  if [ -f "${dir}/Dockerfile.dockerignore" ]; then
+    cp "${dir}/Dockerfile.dockerignore" "${dir}/Dockerfile.effective.dockerignore"
+  fi
   cp "${dir}/Dockerfile" "${dir}/Dockerfile.effective"
   _replace-image-latest-by-specific-version
   _append-metadata
@@ -276,6 +304,20 @@ function _get-effective-version() {
   head -n 1 "${dir}/.version" 2> /dev/null ||:
 }
 
+# Print the effective tag for the image with the version. If you don't have a VERSION file
+# then the tag will be just the image name.  Otherwise it will be the image name with the version.
+# Arguments (provided as environment variables):
+#   dir=[path to the image directory where the Dockerfile is located, default is '.']
+function _get-effective-tag() {
+
+  local tag=$DOCKER_IMAGE
+  version=$(_get-effective-version)
+  if [ ! -z "$version" ]; then
+    tag="${tag}:${version}"
+  fi
+  echo "$tag"
+}
+
 # Print all Docker image versions.
 # Arguments (provided as environment variables):
 #   dir=[path to the image directory where the Dockerfile is located, default is '.']

scripts/docker/docker.mk

@@ -4,26 +4,39 @@
 # Custom implementation - implementation of a make target should not exceed 5 lines of effective code.
 # In most cases there should be no need to modify the existing make targets.
 
-docker-build: # Build Docker image - optional: docker_dir|dir=[path to the Dockerfile to use, default is '.'] @Development
-	make _docker cmd="build" \
+DOCKER_IMAGE ?= $(or ${docker_image}, $(or ${IMAGE}, $(or ${image}, ghcr.io/org/repo)))
+DOCKER_TITLE ?= $(or "${docker_title}", $(or "${TITLE}", $(or "${title}", "Service Docker image")))
+
+docker-bake-dockerfile: # Create Dockerfile.effective - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
+	make _docker cmd="bake-dockerfile" \
 		dir=$(or ${docker_dir}, ${dir})
-	file=$(or ${docker_dir}, ${dir})/Dockerfile.effective
-	scripts/docker/dockerfile-linter.sh
+
+docker-build: # Build Docker image - optional: docker_dir|dir=[path to the Dockerfile to use, default is '.'] @Development
+	dir=$(or ${docker_dir}, ${dir})
+	make _docker cmd="build"
+docker-build: docker-lint
+
+docker-lint: # Run hadolint over the Dockerfile - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
+	dir=$(or ${docker_dir}, ${dir})
+	make _docker cmd="lint"
+docker-lint: docker-bake-dockerfile
 
 docker-push: # Push Docker image - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
 	make _docker cmd="push" \
 		dir=$(or ${docker_dir}, ${dir})
 
+docker-run: # Run Docker image - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Development
+	make _docker cmd="run" \
+		dir=$(or ${docker_dir}, ${dir})
+
 clean:: # Remove Docker resources (docker) - optional: docker_dir|dir=[path to the image directory where the Dockerfile is located, default is '.'] @Operations
 	make _docker cmd="clean" \
 		dir=$(or ${docker_dir}, ${dir})
 
 _docker: # Docker command wrapper - mandatory: cmd=[command to execute]; optional: dir=[path to the image directory where the Dockerfile is located, relative to the project's top-level directory, default is '.']
 	# 'DOCKER_IMAGE' and 'DOCKER_TITLE' are passed to the functions as environment variables
-	DOCKER_IMAGE=$(or ${DOCKER_IMAGE}, $(or ${docker_image}, $(or ${IMAGE}, $(or ${image}, ghcr.io/org/repo))))
-	DOCKER_TITLE=$(or "${DOCKER_TITLE}", $(or "${docker_title}", $(or "${TITLE}", $(or "${title}", "Service Docker image"))))
+	dir=$(realpath $(or ${dir}, infrastructure/images/${DOCKER_IMAGE}))
 	source scripts/docker/docker.lib.sh
-	dir=$(realpath ${dir})
 	docker-${cmd} # 'dir' is accessible by the function as environment variable
 
 # ==============================================================================
@@ -40,44 +53,15 @@ docker-shellscript-lint: # Lint all Docker module shell scripts @Quality
 docker-test-suite-run: # Run Docker test suite @ExamplesAndTests
 	scripts/docker/tests/docker.test.sh
 
-docker-example-build: # Build Docker example @ExamplesAndTests
-	source scripts/docker/docker.lib.sh
-	cd scripts/docker/examples/python
-	DOCKER_IMAGE=repository-template/docker-example-python
-	DOCKER_TITLE="Repository Template Docker Python Example"
-	TOOL_VERSIONS="$(shell git rev-parse --show-toplevel)/scripts/docker/examples/python/.tool-versions.example"
-	docker-build
-
-docker-example-lint: # Lint Docker example @ExamplesAndTests
-	dockerfile=scripts/docker/examples/python/Dockerfile
-	file=$${dockerfile} scripts/docker/dockerfile-linter.sh
-
-docker-example-run: # Run Docker example @ExamplesAndTests
-	source scripts/docker/docker.lib.sh
-	cd scripts/docker/examples/python
-	DOCKER_IMAGE=repository-template/docker-example-python
-	args=" \
-		-it \
-		--publish 8000:8000 \
-	"
-	docker-run
-
-docker-example-clean: # Remove Docker example resources @ExamplesAndTests
-	source scripts/docker/docker.lib.sh
-	cd scripts/docker/examples/python
-	DOCKER_IMAGE=repository-template/docker-example-python
-	docker-clean
-
 # ==============================================================================
 
 ${VERBOSE}.SILENT: \
 	_docker \
 	clean \
+	docker-bake-dockerfile \
 	docker-build \
-	docker-example-build \
-	docker-example-clean \
-	docker-example-lint \
-	docker-example-run \
+	docker-lint \
 	docker-push \
+	docker-run \
 	docker-shellscript-lint \
 	docker-test-suite-run \