Building stages
[ DRAFT ]
Updated: 2020-12-27
- Building stages
There are two kinds of building environments - Docker Hub and local computer. These environments will be referenced as stages.
Both stages have different purposes.
The Docker Hub stage is intended for building and publishing the final Docker images.
The local stage is primarily intended for developing new images.
However, the building pipeline is implemented that way, that it can partially or completely duplicate the Docker Hub stage. In other words, the local stage can be also used for building and publishing the final images.
The intention has been, that differences between both stages are as small as possible.
The Docker Hub stage is actually the Builds page of the builder repository after entering the edit mode by clicking the button Configure Automated Builds.
It should be understood, that the following configuration relates only to the builder repository on Docker Hub. The deployment repositories require no configuration.
This section contains the link to the GitHub repository with all necessary resources.
The values of the settings AUTOTEST and REPOSITORY LINKS can be both left at theirs default Off values.
This sections contains the auto-builder rules.
The fields Source Type and Source should be generally set to Branch and master.
The fields Source and Docker Tag provide the arguments branch and blend to the hook scripts, as it is described on the page How building works.
The value of the field Dockerfile location can be left at its default value Dockerfile, because the actual value will be set by the hook source file env.rc, as described on the page How building works.
The field Build Context should be set to /docker.
The field Autobuild should be set to Off, because the building will be generally initiated using the webhook.
This section allows setting the environment variables that are described below.
The following variables are required:
- DOCKER_BUILDKIT=1
- GIST_ID=(secret)
- DEPLOY_GIST_ID=(secret)
- GIST_TOKEN=(secret)
- POST_PUSH_WORKFLOW_TOKEN=(secret)
The following variables are optional:
- FORCE_BUILDING=(0|1)
- FORCE_README_PUBLISHING=(0|1)
This sections contains a definition of the webhook, which can be used for triggering the execution of the auto-builder rules described above.
The name of the webhook is not important, but its value from the field Trigger Url should be defined as a GitHub Actions secret as it is described below.
It should be understood, that the webhook will trigger all the auto-builder rules defined in the section Build rules.
The local stage is a computer with Docker installed. Both Linux and Windows operating systems are supported.
It is recommended to use Visual Studio Code on local stage, but it is not a requirement.
Building the images on local stage is no different from doing it on the Docker Hub stage. However, the local stage offers much more flexibility.
Some examples:
-
It is possible to execute the complete building pipeline or only part of it.
-
It is possible to build the final images and push them into their deployment repositories on Docker Hub.
-
It is possible to build images for private local use, skipping updating the metadata, gists and pushing to Docker Hub.
-
It is possible to modify and debug the hook scripts, Dockerfiles, the startup script and do all the stuff developers need to do.
There is no special configuration required on local stage. The scripts and the environment variables are the same on both stages.
To make it more convenient, there is the file example-secrets.rc, which can be modified and sourced in the beginning of each terminal session.
There is also the utility util-readme.sh, which allows previewing and publishing the README files to Docker Hub from the local computer.
Regardless if building locally or on Docker Hub, some environment variables also play a role in building process.
Not all environment variables are required in each scenario.
Some of the environment variables contain secrets and they should be used carefully. They should not be written into logs, for example.
To make it more convenient, the project includes the file example-secrets.rc, which contains all the environment variables used by the building pipeline. This file can be modified and sourced in the terminal window (at shell level zero) in the beginning of each terminal session.
For example, if the example file has been already modified, renamed to my-secrets.rc and stored into the current directory, then it can be source with the following statement:
source ./docker/hooks/my-secrets.rcIf building on Docker Hub, the same environment variables should be set in the section BUILD ENVIRONMENT VARIABLES (see the screenshot above).
This variable should be set to 1 to ensure, that the BuildKit is used for building.
This is the ID of the Github gist, which belongs to the builder repository. It stores metadata used by building the images.
This is the ID of the GitHub gist, which belongs to the deployment repositories. It contains metadata for the published images, e.g. JSON endpoints for the badges.
The name of the GitHub repository, which contains the GitHub Actions workflow, which publishes the README files.
The name of the branch in the POST_PUSH_WORKFLOW_REPO repository, which contains the GitHub Actions workflow, which publishes the README files.
If set to 1, then the image will be built and pushed into its deployment repository even if its verbose version sticker value has not changed since its last release.
If set to 1, then the README file will be re-published even if the image has not been re-built and pushed into its deployment repository.
Defines the string, which will be used as the prefix of the short version sticker value.
Defines the string, which will be used as the suffix of the short version sticker value.
This secret allows access and manipulation of the GitHub gists.
Its value is the GitHub PAT (personal access token) secret value, which is created in the GitHub repository (Settings/Developer settings/Personal access tokens).
The name of the PAT is not important, but it must have the gist scope permission, allowing to access gists.
It should be understood that this PAT could have access to all gists of the same GitHub owner (account).
This secret is created on GitHub and used on Docker Hub or locally for accessing the gists on GitHub.
This secret allows access and manipulation of the GitHub Actions workflows and the GitHub repository itself.
Its value is the GitHub PAT (personal access token) secret value, which is created in the GitHub repository (Settings/Developer settings/Personal access tokens).
The name of the PAT is not important, but it must have both the repo and workflow scope permissions.
It should be understood that this PAT could have access to all GitHub repositories and GitHub Actions workflows of the same GitHub owner (account).
This secret is created on GitHub and used on Docker Hub or locally for accessing the workflows on GitHub.
This secrets are required for publishing the README files because the current (unofficial) DockerHub API does not support personal access tokens (PAT).
Their values are the Docker Hub account name and password.
These secrets are created on Docker Hub and used on GitHub or locally for accessing repositories on Docker Hub.
These are the same secrets as described above. They should be defined as the GitHub Actions secrets DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD, because they are used under these names by the workflow dockerhub-post-push.yml, which publishes the README files to Docker Hub.
To be able to trigger the build rules on Docker Hub, there should be a webhook defined in the section Build triggers (see the screenshot above).
The value of the webhook (Trigger Url) should be defined as the GitHub Actions secret DockerHubWebhookBuildRelease, because it is used under this name by the workflow dockerhub-autobuild.yml, which triggers the auto-building on Docker Hub.