Building stages
Updated: 2022-09-06
-
Building stages
- Introduction
- Docker Hub stage
- Local stage
-
Environment variables
-
Non-secret variables
- Variable
DOCKER_BUILDKIT - Variable
GIST_ID - Variable
DEPLOY_GIST_ID - Variable
POST_PUSH_WORKFLOW_REPO - Variable
POST_PUSH_WORKFLOW_BRANCH - Variable
FORCE_BUILDING - Variable
FORCE_README_PUBLISHING - Variable
PROHIBIT_BUILDING - Variable
PROHIBIT_README_PUBLISHING - Variable
VERSION_STICKER_PREFIX - Variable
VERSION_STICKER_SUFFIX
- Variable
- Secret environment variables
- GitHub Actions secrets
-
Non-secret variables
There are two kinds of building environments - the Docker Hub and the local computer. These environments will be referenced as stages.
Both stages have different purposes.
The Docker Hub stage is intended for building and publishing the final Docker images.
The local stage is primarily intended for developing the new images.
However, the building pipeline is implemented that way, that it can partially or completely duplicate the Docker Hub stage. In other words, the local stage can be also used for building and publishing the final images.
The intention has been, that the differences between both stages are as small as possible.
The Docker Hub stage itself is actually the Builds page of the builder repository after entering the edit mode by clicking the button Configure Automated Builds.
It should be understood, that the following configuration relates only to the builder repository on the Docker Hub. The deployment repositories require no configuration.
This section contains the link to the GitHub repository with all necessary resources.
The values of the settings AUTOTEST and REPOSITORY LINKS can be left at their default Off values.
This sections contains the auto-builder rules.
The fields Source Type and Source should be generally set to Branch and master.
The fields Source and Docker Tag provide the arguments branch and blend to the hook scripts, as it is described on the page How building works.
The value of the field Dockerfile location can be left at its default value Dockerfile, because the actual value will be set by the hook source file env.rc, as described on the page How building works.
The field Build Context should be set to /docker.
The field Autobuild should be set to Off, because the building will be generally initiated using a webhook.
This section allows setting the environment variables that are described below.
The following variables are required:
- DOCKER_BUILDKIT=1
- GIST_ID=(secret)
- DEPLOY_GIST_ID=(secret)
- GIST_TOKEN=(secret)
- POST_PUSH_WORKFLOW_TOKEN=(secret)
The following variables are optional:
- FORCE_BUILDING=(0|1)
- FORCE_README_PUBLISHING=(0|1)
- PROHIBIT_BUILDING=(0|1)
- PROHIBIT_README_PUBLISHING=(0|1)
This sections contains the definition of the webhook, which can be used for triggering the execution of the auto-builder rules described above.
The name of the webhook is not important, but its value from the field Trigger Url should be defined as a GitHub Actions secret as it is described below.
It should be understood, that the webhook will trigger all the auto-builder rules defined in the section Build rules.
There is a limit of max. 25 auto-builder rules on the Docker Hub (free plan), even if it's not documented.
If you need more rules in the builder repository, you can use multiple builder repositories, that are all bound to the same source GitHub repository.
All these sibling builder repositories should be configured exactly the same with the exception of the webhook, which must be unique for each builder repository.
Then there are two other small adjustments to do:
-
In the source GitHub repository you have to add the webhook secrets for the new builder repositories (as described bellow).
-
In the auto-building workflow you have to trigger all the builder repositories (as it is described on the page How CI works).
The local stage is a computer with Docker installed. Both Linux and Windows operating systems are supported.
It is recommended to use the Visual Studio Code on the local stage, but it is not a requirement.
Building the images on the local stage is no different from doing it on the Docker Hub stage. However, the local stage offers more flexibility.
Some examples:
-
It is possible to execute the complete building pipeline or only part of it.
-
It is possible to build the final images and push them into their deployment repositories on the Docker Hub.
-
It is possible to build images for private local use, skipping the updating the metadata, the GitHub Gists and the pushing to the Docker Hub.
-
It is possible to modify and debug the hook scripts, Dockerfiles, the startup scripts and all that stuff the developers want to do.
There is no special configuration required on the local stage. The scripts and the environment variables are the same as on the Docker Hub stage.
To make it more convenient, there is the file example-secrets.rc, which can be modified and sourced in the terminal when beginning the building session session.
The local building and also the publishing to the Docker Hub can be done two ways:
- by following the local building example and executing the hook scripts
- by using the utility script builder.sh
There is also the utility util-readme.sh, which allows previewing the README files and publishing them to the Docker Hub from the local stage.
Regardless if the building takes place locally or on the Docker Hub, some environment variables must be set because they are used by the building process.
Not all environment variables are required in each scenario.
Some of the environment variables contain secrets and they should be used carefully. They should not be written into logs, for example.
To make it more convenient, the project includes the file example-secrets.rc, which contains all the environment variables used by the building pipeline. This file can be modified and sourced in the terminal window (at shell level zero) in the beginning of each building session.
For example, if the example file has been modified, renamed to my-secrets.rc and stored into the current directory, then it can be source with the following statement:
source ./docker/hooks/my-secrets.rcIf building on the Docker Hub, the same environment variables should be set in the section BUILD ENVIRONMENT VARIABLES (see the screenshot above).
This variable should be set to 1 to ensure, that the BuildKit is used for building.
This is the ID of the Github Gist, which belongs to the builder repository. It stores the metadata used by building process.
This is the ID of the GitHub Gist, which belongs to the deployment repositories. It contains the metadata used by the published process, e.g. JSON endpoints for the badges.
The name of the GitHub repository, which contains the GitHub Actions workflow, which publishes the README files.
The name of the branch in the POST_PUSH_WORKFLOW_REPO repository, which contains the GitHub Actions workflow, which publishes the README files.
If set to 1, then the image will be built and pushed into its deployment repository even if its verbose version sticker value has not changed since its last release.
If set to 1, then the README file will be re-published even if the image has not been re-built and pushed into its deployment repository.
If set to 1, then the image will not be built and pushed into its deployment repository even if its verbose version sticker value has changed since its last release.
If set to 1, then the post_push hook script will not trigger the GitHub Actions workflow dockerhub-post-push.yml, which publishes the readme file to the Docker Hub. This can be useful by testing on the Docker Hub or by building from the non-default branches.
Defines the string, which will be used as the prefix of the short version sticker value.
Defines the string, which will be used as the suffix of the short version sticker value.
This secret allows access and manipulation of the GitHub Gists.
Its value is the GitHub PAT (personal access token) secret value, which is created in the GitHub repository (Settings/Developer settings/Personal access tokens).
The name of the PAT is not important, but it must have the gist scope permission, allowing to access the GitHub Gists.
It should be understood that this PAT could have access to all the GitHub Gists of the same GitHub owner (account).
This secret is created on the GitHub and it is used on the Docker Hub or locally for accessing the GitHub Gists.
This secret allows access and manipulation of the GitHub Actions workflows and the GitHub repository itself.
Its value is the GitHub PAT (personal access token) secret value, which is created in the GitHub repository (Settings/Developer settings/Personal access tokens).
The name of the PAT is not important, but it must have both the repo and workflow scope permissions.
It should be understood that this PAT could have access to all the GitHub repositories and the GitHub Actions workflows of the same GitHub owner (account).
This secret is created on the GitHub and it is used on the Docker Hub or locally for accessing the workflows on the GitHub.
These secrets are required for publishing the README files because the current (unofficial) Docker Hub API does not support personal access tokens (PAT).
Their values are the Docker Hub account's name and password.
These secrets are created on the Docker Hub and used on the GitHub or locally for accessing the repositories on the Docker Hub.
These are the same secrets as described above. They should be defined as the GitHub Actions secrets DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD, because they are used under these names by the workflow dockerhub-post-push.yml, which publishes the README files to the Docker Hub.
To be able to trigger the build rules on the Docker Hub, there should be a webhook defined in the section Build triggers (see the screenshot above).
The value of the webhook (Trigger Url) should be defined as the GitHub Actions secret DockerHubWebhookBuildRelease, because it is used under this name by the workflow dockerhub-autobuild.yml, which triggers the auto-building on the Docker Hub.
If you want to use multiple builder repositories, then name the additional secrets like DockerHubWebhookBuildRelease_1, DockerHubWebhookBuildRelease_2 and so on.