Skip to content

Commit

Permalink
Merge pull request #1776 from tormath1/tormath1/sysext
Browse files Browse the repository at this point in the history
✨ Add `flatcar-sysext` template to use regular Flatcar images
  • Loading branch information
k8s-ci-robot authored Jan 2, 2024
2 parents e590271 + 5a39a20 commit 7edf6a1
Show file tree
Hide file tree
Showing 10 changed files with 553 additions and 7 deletions.
6 changes: 4 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,8 @@ e2e-templates: $(addprefix $(E2E_NO_ARTIFACT_TEMPLATES_DIR)/, \
cluster-template-without-lb.yaml \
cluster-template.yaml \
cluster-template-flatcar.yaml \
cluster-template-k8s-upgrade.yaml)
cluster-template-k8s-upgrade.yaml \
cluster-template-flatcar-sysext.yaml)
# Currently no templates that require CI artifacts
# $(addprefix $(E2E_TEMPLATES_DIR)/, add-templates-here.yaml) \
Expand Down Expand Up @@ -406,7 +407,8 @@ release-notes: $(RELEASE_NOTES) ## Generate release notes
templates: ## Generate cluster templates
templates: templates/cluster-template.yaml \
templates/cluster-template-without-lb.yaml \
templates/cluster-template-flatcar.yaml
templates/cluster-template-flatcar.yaml \
templates/cluster-template-flatcar-sysext.yaml

templates/cluster-template.yaml: kustomize/v1alpha8/default $(KUSTOMIZE) FORCE
$(KUSTOMIZE) build "$<" > "$@"
Expand Down
38 changes: 34 additions & 4 deletions docs/book/src/clusteropenstack/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,33 @@ The image can be referenced by exposing it as an environment variable `OPENSTACK

Some OS like [Fedora CoreOS](https://getfedora.org/en/coreos) or [Flatcar](https://www.flatcar.org/) do not use cloud-init but [Ignition](https://coreos.github.io/ignition/) to provision the instance. You need to enable the [Ignition experimental feature](https://cluster-api.sigs.k8s.io/tasks/experimental-features/ignition.html): `export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true`

To use Flatcar image:
* Build the image with the [image-builder](https://image-builder.sigs.k8s.io/capi/providers/openstack.html): `make OEM_ID=openstack build-qemu-flatcar`
* Export the name of the uploaded image: `export OPENSTACK_FLATCAR_IMAGE_NAME=flatcar-stable-3374.2.5-kube-v1.25.6`
* When generating the cluster configuration, use the following Cluster API [flavor](https://cluster-api.sigs.k8s.io/clusterctl/commands/generate-cluster.html?#flavors): `--flavor flatcar` (_NOTE_: Don't forget to refer to the [external-cloud-provider](https://cluster-api-openstack.sigs.k8s.io/topics/external-cloud-provider.html) section)
Flatcar comes in two [flavor][flavor] variants:
* `flatcar`

This variant relies on a Flatcar image built using the image-builder project: the Kubernetes version is bound to the Flatcar version and a rebuild of the image is required for each Kubernetes or Flatcar upgrade.

To build and use Flatcar image:
* Build the image with the [image-builder][image-builder]: `make OEM_ID=openstack build-qemu-flatcar`
* Upload the image
* Export the name of the uploaded image: `export OPENSTACK_FLATCAR_IMAGE_NAME=flatcar-stable-3374.2.5-kube-v1.25.6`
* When generating the cluster configuration, use the following Cluster API [flavor][flavor]: `--flavor flatcar` (_NOTE_: Don't forget to refer to the [external-cloud-provider][external-cloud-provider] section)

* `flatcar-sysext`

This variant relies on a plain Flatcar image and it leverages [systemd-sysext][systemd-sysext] feature to install and update Kubernetes components: the Kubernetes version is not bound to the Flatcar version (i.e Flatcar can be independently upgraded from Kubernetes and vice versa).

The template comes with a [systemd-sysupdate][systemd-sysupdate] configuration file that will download each new patch version of Kubernetes (i.e if you start with Kubernetes 1.x.y, systemd-sysupdate will automatically pull 1.x.y+1 but not 1.x+1.y), please note that this behavior is disabled by default. To enable the Kubernetes auto-update you can:
* Update the template to enable the `systemd-sysupdate.timer`
* Or run the following command on the nodes: `sudo systemctl enable --now systemd-sysupdate.timer`

When the Kubernetes release reaches end-of-life it will not receive updates anymore. To switch to a new major version, do a `sudo rm /etc/sysupdate.kubernetes.d/kubernetes-*.conf` and download the new update config into the folder with `cd /etc/sysupdate.kubernetes.d && sudo wget https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf`.

To coordinate the node reboot, we recommend to use [Kured][kured]. Note that running `kubeadm upgrade apply` on the first controller and `kubeadm upgrade node` on all other nodes is not automated (yet), see the [docs](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/).

To use Flatcar image:
* Upload an image on OpenStack from the Flatcar release servers (e.g for Stable, you might use this image: https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img)
* Export the name of the uploaded image: `export FLATCAR_IMAGE_NAME=flatcar_production_openstack_image`
* When generating the cluster configuration, use the following Cluster API [flavor][flavor]: `--flavor flatcar-sysext` (_NOTE_: Don't forget to refer to the [external-cloud-provider][external-cloud-provider] section)

## SSH key pair

Expand Down Expand Up @@ -640,3 +663,10 @@ $ kubectl get openstackcluster
NAME CLUSTER READY NETWORK SUBNET BASTION
nonha nonha true 2e2a2fad-28c0-4159-8898-c0a2241a86a7 53cb77ab-86a6-4f2c-8d87-24f8411f15de 10.0.0.213
```

[external-cloud-provider]: https://cluster-api-openstack.sigs.k8s.io/topics/external-cloud-provider.html
[flavor]: https://cluster-api.sigs.k8s.io/clusterctl/commands/generate-cluster.html?#flavors
[image-builder]: https://image-builder.sigs.k8s.io/capi/providers/openstack.html
[kured]: https://github.com/kubereboot/kured
[systemd-sysext]: https://www.flatcar.org/docs/latest/provisioning/sysext/
[systemd-sysupdate]: https://www.freedesktop.org/software/systemd/man/latest/sysupdate.d.html
3 changes: 2 additions & 1 deletion hack/ci/cloud-init/controller.yaml.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/cirros/2022-12-05/cirros-0.6.1-x86_64-disk.img,"
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/ubuntu/2023-09-29/ubuntu-2204-kube-v1.27.2.img,"
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/ubuntu/2023-09-29/ubuntu-2204-kube-v1.28.2.img,"
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/flatcar/flatcar-stable-3602.2.0-kube-v1.28.2.img"
IMAGE_URLS+="https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/flatcar/flatcar-stable-3602.2.0-kube-v1.28.2.img,"
IMAGE_URLS+="https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_openstack_image.img"

[[post-config|$NOVA_CONF]]
[DEFAULT]
Expand Down
7 changes: 7 additions & 0 deletions kustomize/v1alpha7/flatcar-sysext/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../default

patches:
- path: patch-flatcar.yaml
185 changes: 185 additions & 0 deletions kustomize/v1alpha7/flatcar-sysext/patch-flatcar.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha7
kind: OpenStackCluster
metadata:
name: ${CLUSTER_NAME}
spec:
apiServerLoadBalancer:
$patch: delete
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: "${CLUSTER_NAME}-control-plane"
spec:
replicas: ${CONTROL_PLANE_MACHINE_COUNT}
kubeadmConfigSpec:
joinConfiguration:
nodeRegistration:
name: $${COREOS_OPENSTACK_HOSTNAME}
kubeletExtraArgs:
provider-id: null
initConfiguration:
nodeRegistration:
name: $${COREOS_OPENSTACK_HOSTNAME}
kubeletExtraArgs:
# Fixme(lentzi90): This is here just to override the value set in the default
# kustomization. It will be replaced with a value that works for flatcar in
# https://github.com/kubernetes-sigs/cluster-api-provider-openstack/pull/1564
provider-id: null
format: ignition
ignition:
containerLinuxConfig:
additionalConfig: |
storage:
links:
- path: /etc/extensions/kubernetes.raw
hard: false
target: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
files:
- path: /etc/sysupdate.kubernetes.d/kubernetes-${KUBERNETES_VERSION%.*}.conf
mode: 0644
contents:
remote:
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf
- path: /etc/sysupdate.d/noop.conf
mode: 0644
contents:
remote:
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
- path: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
contents:
remote:
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
systemd:
units:
- name: systemd-sysupdate.service
dropins:
- name: kubernetes.conf
contents: |
[Service]
ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi"
- name: update-engine.service
# Set this to 'false' if you want to enable Flatcar auto-update
mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true}
- name: locksmithd.service
# NOTE: To coordinate the node reboot in this context, we recommend to use Kured.
mask: true
- name: systemd-sysupdate.timer
# Set this to 'true' if you want to enable the Kubernetes auto-update.
# NOTE: Only patches version will be pulled.
enabled: false
- name: [email protected]
enabled: true
- name: kubeadm.service
enabled: true
dropins:
- name: 10-flatcar.conf
contents: |
[Unit]
Requires=containerd.service coreos-metadata.service
After=containerd.service coreos-metadata.service
[Service]
EnvironmentFile=/run/metadata/flatcar
preKubeadmCommands:
- export COREOS_OPENSTACK_HOSTNAME=$${COREOS_OPENSTACK_HOSTNAME%.*}
- envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp
- mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: ${CLUSTER_NAME}-md-0
spec:
template:
spec:
joinConfiguration:
nodeRegistration:
name: $${COREOS_OPENSTACK_HOSTNAME}
kubeletExtraArgs:
# Fixme(lentzi90): This is here just to override the value set in the default
# kustomization. It will be replaced with a value that works for flatcar in
# https://github.com/kubernetes-sigs/cluster-api-provider-openstack/pull/1564
provider-id: null
preKubeadmCommands:
- export COREOS_OPENSTACK_HOSTNAME=$${COREOS_OPENSTACK_HOSTNAME%.*}
- envsubst < /etc/kubeadm.yml > /etc/kubeadm.yml.tmp
- mv /etc/kubeadm.yml.tmp /etc/kubeadm.yml
format: ignition
ignition:
containerLinuxConfig:
additionalConfig: |
storage:
links:
- path: /etc/extensions/kubernetes.raw
hard: false
target: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
files:
- path: /etc/sysupdate.kubernetes.d/kubernetes-${KUBERNETES_VERSION%.*}.conf
mode: 0644
contents:
remote:
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION%.*}.conf
- path: /etc/sysupdate.d/noop.conf
mode: 0644
contents:
remote:
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/noop.conf
- path: /opt/extensions/kubernetes/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
contents:
remote:
url: https://github.com/flatcar/sysext-bakery/releases/download/latest/kubernetes-${KUBERNETES_VERSION}-x86-64.raw
systemd:
units:
- name: systemd-sysupdate.service
dropins:
- name: kubernetes.conf
contents: |
[Service]
ExecStartPre=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes"
ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C kubernetes update
ExecStartPost=/usr/bin/sh -c "readlink --canonicalize /etc/extensions/kubernetes.raw > /tmp/kubernetes-new"
ExecStartPost=/usr/bin/sh -c "if ! cmp --silent /tmp/kubernetes /tmp/kubernetes-new; then touch /run/reboot-required; fi"
- name: update-engine.service
# Set this to 'false' if you want to enable Flatcar auto-update
mask: ${FLATCAR_DISABLE_AUTO_UPDATE:=true}
- name: locksmithd.service
# NOTE: To coordinate the node reboot in this context, we recommend to use Kured.
mask: true
- name: systemd-sysupdate.timer
# Set this to 'true' if you want to enable the Kubernetes auto-update.
# NOTE: Only patches version will be pulled.
enabled: false
- name: [email protected]
enabled: true
- name: kubeadm.service
enabled: true
dropins:
- name: 10-flatcar.conf
contents: |
[Unit]
Requires=containerd.service coreos-metadata.service
After=containerd.service coreos-metadata.service
[Service]
EnvironmentFile=/run/metadata/flatcar
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha7
kind: OpenStackMachineTemplate
metadata:
name: ${CLUSTER_NAME}-md-0
spec:
template:
spec:
image: ${FLATCAR_IMAGE_NAME}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha7
kind: OpenStackMachineTemplate
metadata:
name: ${CLUSTER_NAME}-control-plane
spec:
template:
spec:
image: ${FLATCAR_IMAGE_NAME}
Loading

0 comments on commit 7edf6a1

Please sign in to comment.