-
Notifications
You must be signed in to change notification settings - Fork 189
GitLab CI in bwCloud
The University of Freiburg operates an OpenStack cloud, which we would like to use for additional runners. First, go to https://bwservices.uni-freiburg.de/, log in with your University of Stuttgart account (acXXXXXX), and sign up for the bwCloud SCOPE service. You'll get an email after a day or two saying that you were granted access. Then you can go to the same page again and set a password.
Next, ask go to https://bw-support.scc.kit.edu and open a support ticket to request creation of a new group project. @mkuron already did that and you can ask him to request that your account be added to the stuttgart_espresso project.
Once you have access, log into https://portal.bw-cloud.org/, which is the OpenStack management portal. Switch to the stuttgart_espresso project.
- Under Compute, Key Pairs, I created an SSH key pair named gitlab, whose private key is stored on elk in /etc/gitlab-runner/bwcloud-acXXXXXX.pem.
- Under Network, Security Groups, I created a firewall ruleset named gitlab-runner, which allows access to TCP port 2376 for 129.69.120.0/24.
- Go to Identity, Projects and copy the Project ID (cceb6ed21ffc4b32a26169873a531ba8) for the group project.
On elk, run sudo gitlab-runner register --executor docker+machine
. Tell it to register with https://gitlab.icp.uni-stuttgart.de
and add the tags docker,linux,nofirewall,ptrace
. Then edit /etc/gitlab-runner/config.toml until it looks like this:
concurrent = 100
check_interval = 0
[session_server]
session_timeout = 1800
listen_address = "0.0.0.0:8093"
advertise_address = "elk.icp.uni-stuttgart.de:8093"
[[runners]]
name = "bwCloud"
url = "https://gitlab.icp.uni-stuttgart.de/"
token = "AAAAAAAAAAAAAAAAAA"
executor = "docker+machine"
limit = 48
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "ubuntu"
privileged = false
cap_add = ["SYS_PTRACE"]
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache:/cache"]
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.machine]
IdleCount = 1
MaxBuilds = 4
MachineDriver = "openstack"
MachineName = "gitlab-ci-%s"
MachineOptions = [
"[email protected]",
"openstack-domain-name=Default",
"openstack-password=YYYYYYYYYYYYYY",
"openstack-tenant-id=cceb6ed21ffc4b32a26169873a531ba8",
"openstack-region=Mannheim",
"openstack-auth-url=https://idm01.bw-cloud.org:5000/v3",
"openstack-flavor-name=m1.medium",
"openstack-image-name=Debian 10",
"openstack-ssh-user=debian",
"openstack-keypair-name=gitlab",
"openstack-user-data-file=/etc/gitlab-runner/cloud-runner.sh",
"openstack-private-key-file=/etc/gitlab-runner/bwcloud-acXXXXXX.pem",
"openstack-sec-groups=default,gitlab-runner"
]
OffPeakTimezone = "Europe/Berlin"
OffPeakIdleCount = 0
OffPeakIdleTime = 0
OffPeakPeriods = [
"* * 0-8,19-23 * * mon-fri *",
"* * * * * sat,sun *"
]
Now put the following into /etc/gitlab-runner/cloud-runner.sh:
#!/bin/bash
which cloud-init >/dev/null
if [ "$?" != "0" ]; then
echo "Only run this script via cloud-init"
exit 1
fi
# create the cache directory and make it writable
mkdir /cache
chmod 777 /cache
# install the binfmt handlers for QEMU
apt-get update
apt-get install -y qemu-user-static
On elk, you can directly use docker-machine
.
You need to use sudo -H
to get access to the instances created by gitlab-runner.
If you want to create your own instances for testing, you can do that without sudo
.
To view running instances created by gitlab-runner:
sudo -H docker-machine ls
To log into one of them:
sudo -H docker-machine ssh runner-ZZZZZZZZ-gitlab-ci-NNNNNNNNNN-HHHHHHHH
To manually create an instance (the arguments are the same as those given in the MachineOptions
section above, i.e. openstack-username
becomes --openstack-username
.):
docker create --driver openstack --openstack-keypair-name=gitlab --openstack-private-key-file=./bwcloud.pem \
--openstack-domain-name=Default --openstack-tenant-id=cceb6ed21ffc4b32a26169873a531ba8 \
[email protected] --openstack-password=YYYYYYYYYY \
--openstack-auth-url https://idm01.bw-cloud.org:5000/v3/ --openstack-flavor-name=m1.medium \
--openstack-image-name="Debian 10" --openstack-region=Mannheim --openstack-ssh-user=debian \
--openstack-sec-groups=default,gitlab-runner my_test_vm
To log into that instance:
docker-machine ssh my_test_vm
To delete your instance:
docker-machine rm -f my_test_vm
docker-machine's OpenStack support is quite incompletes. This makes the configuration file less pretty in two ways, slows down builds by requiring unnecessary reboots and prevents the use of ccache.
The provided Linux images are also not perfect. A bug in the current Debian image delays the boot process unnecessarily, while a missing kernel module in the Ubuntu image breaks QEMU support for us.
It is also possible to specify openstack-tenant-name
, but that uses a deprecated API that bwCloud no longer supports. Ticket: https://github.com/docker/machine/issues/3774.
An acXXXXXX account name is used in the configuration file, which might become a problem once that person leaves the university. One could create a project token under Identity, Application Credentials, but docker-machine cannot currently use that. Ticket: https://github.com/docker/machine/issues/4684.
All instances have a 12 GB root disk, which only a few builds and Docker images fit onto. One could attach 40 GB temporary disks to the instance via
openstack server create --image "Debian 10" --flavor m1.nano --block-device vdb=empty:snapshot:40:true test1
after creating a blank snapshot via
openstack volume create --size 1 empty
openstack volume snapshot create --volume empty empty
but docker-machine doesn't support block device mappings. Once it does, one can modify cloud-runner.sh to format and mount that disk.
Current workaround: set MaxBuilds = 4
so machines are discarded after a few builds.
Possible workaround: install an NFS server and a Docker registry proxy in a permanently-running instance. Beware that the NFS server would be writable to all bwCloud users because we can't simultaneously use a private and a public network. Ticket: https://github.com/docker/machine/issues/1444.
Sometimes the GitLab runner doesn't properly destroy instances. They disappear from docker-machine, but remain running. This seems to sometimes happen when GitLab runner is restarted, but also when instance creation fails.