capi: support Flatcar Container Linux #248

dongsupark · 2020-06-05T15:16:55Z

This is a cluster-API image builder for Flatcar Container Linux.

On the Packer side it includes packer/qemu/qemu-flatcar.json and packer/qemu/packer.json. In packer/config/ it also includes some other JSON files with configuration for ansible and packages installed with Ansible.

On the Ansible side there's some preliminary playbook in ansible/.

notes

images/capi/ansible/roles/setup/tasks/bootstrap-flatcar.yml installs python under /opt.
images/capi/ansible/roles/kubernetes/tasks/url.yml installs artificts of Kubernetes and CNI under /opt and /etc, mainly because /usr is read-only in Flatcar.
It unpacks only parts of containerd binaries, since Flatcar already has its own built-in containerd.
To make containerd work correctly, we install Flatcar-specific config files for containerd, so it listens on its containerd socket /run/docker/libcontainerd/docker-containerd.sock.
Since Flatcar has its own containerd socket, we need to also specify the socket name when running commands like kubeadm config images pull, /opt/bin/ctr images import as well as in /etc/crictl.yaml.
We do not delete /etc/kubeadm.yml for Flatcar, because later steps running kubeadm init require the config to be in place.

TODOs

Make sure it works for other platforms than qemu & AWS

/cc @vbatts @t-lo @dims @detiber @codenrhoden

k8s-ci-robot · 2020-06-05T15:16:59Z

@dongsupark: GitHub didn't allow me to request PR reviews from the following users: vbatts, t-lo.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

This is a cluster-API image builder for Flatcar Container Linux.

On the Packer side it includes packer/qemu/flatcar.json and packer/qemu/packer.json. In packer/config/ it also includes some other JSON files with configuration for ansible and packages installed with Ansible.

On the Ansible side there's some preliminary playbook in ansible/.

notes

It needs to disable gather_facts in images/capi/ansible/node.yml, because it is not possible to run python before running setup tasks. Note that python is not installed by default in Flatcar.

To make ansible_env.SUDO_USER work with gather_facts disabled, setup: filter=ansible_env is needed in images/capi/ansible/roles/sysprep/tasks/main.yml.

images/capi/ansible/roles/setup/tasks/bootstrap-flatcar.yml installs python under /opt.

images/capi/ansible/roles/kubernetes/tasks/url-flatcar.yml installs artificts of Kubernetes and CNI under /opt and /etc, mainly because /usr is read-only in Flatcar.

images/capi/ansible/roles/sysprep/tasks/main.yml sets hostname with use: systemd in case of Flatcar. It is a workaround to avoid hostname detection errors when running ansible in Flatcar. We fixed the issue in upstream Ansible, but it is not included in any Ansible release yet.

It unpacks only parts of containerd binaries, since Flatcar already has its own built-in containerd.

To make containerd work correctly, we install Flatcar-specific config files for containerd, so it listens on its containerd socket /run/docker/libcontainerd/docker-containerd.sock.

Since Flatcar has its own containerd socket, we need to also specify the socket name when running commands like kubeadm config images pull, /opt/bin/ctr images import as well as in /etc/crictl.yaml.

We do not delete /etc/kubeadm.yml for Flatcar, because later steps running kubeadm init require the config to be in place.

TODOs

Make sure it works for other platforms than qemu

Get Makefile support multiple Flatcar channels and versions

/cc @vbatts @t-lo @dims @detiber @codenrhoden

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2020-06-05T15:17:03Z

Welcome @dongsupark!

It looks like this is your first PR to kubernetes-sigs/image-builder 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/image-builder has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

dongsupark · 2020-06-05T15:17:52Z

/assign @vincepri

k8s-ci-robot · 2020-06-08T10:33:49Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

t-lo · 2020-06-08T11:26:07Z

Signed. Can we re-evaluate please?

dongsupark · 2020-06-16T08:26:07Z

Rebased, and added AWS AMI support.

CecileRobertMichon · 2020-06-17T23:16:50Z

images/capi/id_rsa

@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEogIBAAKCAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzI


@dongsupark was this checked in by mistake?

No, the key is exactly the same key as the vagrant insecure ssh key.
It is needed for vagrant tests.

Now I am thinking, it might be better to rename the key to a descriptive name, e.g. vagrant-insecure-key.

CecileRobertMichon · 2020-06-17T23:21:19Z

images/capi/hack/image-build-flatcar.sh

+    make FLATCAR_CHANNEL="$channel" FLATCAR_VERSION="$release" \
+        build-${CAPI_PROVIDER}-flatcar
+    run_vagrant
+elif [[ ${CAPI_PROVIDER} = "aws" ]] || [[ ${CAPI_PROVIDER} = "ami" ]]; then


any plans to add the same support for azure?

Sure, we can add Azure support in the future.
Before that, I would like to get some feedback from maintainers about the current PR, to make sure we are doing in the right direction.

Renamed the ssh key file to vagrant-insecure-key.

codenrhoden

Okay, a lot to go over here, so I'm sure it will take a few rounds and some experimentation...

This is exciting to see Flatcar come in, and I'm really happy to see it making progress. I think the main stumbling blocks i see is the stuff around setting gather_facts to False for Ansible, and trying to set some default facts directly. As this PR currently stands, it's going to break the majority of existing builds (it will break all OVAs, All AMIs, and basically everything but Flatcar as far as I can tell). So that's the big thing we'll have to sort out.

On the templating side, there are several that are copied with a flatcar specific version that I think can remain as a single template and just use conditional to avoid having to change things int wo places.

Seeing this work is making me raise the priority of some other changes that are planned, that will make adding additional OS's to the AMI and QEMU builders easier. Thanks for this!

images/capi/README-flatcar.md

images/capi/ansible/node.yml

images/capi/ansible/roles/containerd/templates/etc/containerd/config-flatcar.toml

codenrhoden · 2020-06-22T20:25:44Z

images/capi/ansible/roles/containerd/templates/etc/systemd/system/containerd-flatcar.service

+LimitNOFILE=1048576
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity


I think you mentioned that Flatcar ships with containerd. Is there not already a unit file present for it? I'm concerned about there being confusion or conflicts between this and the additional drop-ins that are put in place today by

image-builder/images/capi/ansible/roles/containerd/tasks/main.yml

Lines 42 to 60 in 9cf00c9

- name: Creates unit file directory

file:

path: /etc/systemd/system/containerd.service.d

state: directory

- name: Create containerd boot order drop in file

template:

dest: /etc/systemd/system/containerd.service.d/boot-order.conf

src: etc/systemd/system/containerd.service.d/boot-order.conf

- name: Create containerd memory pressure drop in file

template:

dest: /etc/systemd/system/containerd.service.d/memory-pressure.conf

src: etc/systemd/system/containerd.service.d/memory-pressure.conf

- name: Create containerd max tasks drop in file

template:

dest: /etc/systemd/system/containerd.service.d/max-tasks.conf

src: etc/systemd/system/containerd.service.d/max-tasks.conf

I already see one example where a specific setting is set twice (TasksMax).

I think you mentioned that Flatcar ships with containerd. Is there not already a unit file present for it? I'm concerned about there being confusion or conflicts between this and the additional drop-ins that are put in place today...

Right. Flatcar already has built-in containerd and docker provided by torcx. It basically means that after boots it is not possible to modify any torcx artifacts, including the containerd's systemd unit file as well as the containerd config. However, we need to download containerd binary files, modify its config, so the new containerd points to the built-in gRPC socket provided by Docker. To do that, we need to install a new containerd unit file that points to a new containerd config that has path to the new containerd socket.

I admit it could look ugly, but I was not able to find a better solution.

images/capi/ansible/roles/kubernetes/files/usr/lib/systemd/system/kubelet-flatcar.service

images/capi/packer/qemu/packer.json

dongsupark · 2020-06-23T15:21:03Z

@codenrhoden Thanks a lot for your thorough review.
Will try to address the issues.

We should be able to configure path to containerd CRI socket file, `containerd_cri_socket`, for each distro. Its main use case is to support distros with different CRI socket path, for example, `/run/docker/libcontainerd/container.sock` for Flatcar. By default it is `/var/run/containerd/containerd.sock` as it has been in all other distros. This PR is split out from kubernetes-sigs#248, as suggested by codenrhoden.

dongsupark · 2020-12-03T16:30:35Z

@codenrhoden Rebased the PR. Please have a look.

This is a cluster-API image builder for [Flatcar Container Linux](https://www.flatcar-linux.org). On the Packer side it includes `packer/qemu/flatcar.json` and `packer/qemu/packer.json`. In `packer/config/` it also includes some other JSON files with configuration for ansible and packages installed with Ansible. On the Ansible side there's some preliminary playbook in `ansible/`.

This change adds support for building all channels of Flatcar Container Linux. It also fixes an issue that made the build fail when make was called directly instead of using the wrapper script hack/image-build-flatcar.sh. Define an env variable `CAPI_PRIVIDER`, which is `aws`, `ami` or `qemu`. Default is `qemu`. Run vagrant commands only when the provider is qemu. If it is `aws` or `ami`, run `make build-ami-default`. Otherwise fail.

We do not need to necessarily install and unpack cri-containerd tarball, if we have containerd 1.3. Simply create symlinks to existing binaries from `/usr/bin` to `/opt/bin`, and set correct paths to them. Note, it only works for containerd 1.3, current Beta (2605.3.0) and Alpha (2605.1.0). OTOH stable (2512.4.0) with containerd 1.1 does not work with this approach yet.

In case of Flatcar, if the given containerd version does not match with the running containerd version, the image-builder should fail hard. A version mismatch with the current version. So it means we need to always set a correct version from Packer side, to make the containerd installation work for Flatcar.

Also unpack `containerd-shim-runc-v[12]` binaries from the cri-containerd-cni tarball, because Flatcar by default does not provide the runc shim binaries. Without the shims, containerd does not work correctly. Also We need to create {{sysusr_prefix}}/bin before unpacking containerd tarballs, to avoid issues happening when unpacking containerd tarballs into non-existent directories.

We should be able to set Flatcar channels and versions via `build_name`, `FLATCAR_CHANNEL`, or `FLATCAR_VERSION`, to use also non-default channels like alpha.

We should use `build_name` as-is, to avoid issues that can happen in builds for other distros.

Since we started simply using the `build_name` variable, we do not need to have `only` directives for checking for Flatcar. Simply remove the `only` directives.

We should run bootstrap-flatcar.sh only in case of Flatcar. We need to check the if clauses in `exec_command`. It is not possible to check for the `BUILD_NAME` in `inline` or `script`.

We should not embed vagrant insecure key in the PR. Instead we simply download the keypair from the official vagrant github repo, to use them for ignition config for Flatcar. Ignition template file `ignition-builder.json` will include a pre-defined string `BUILDERSSHAUTHKEY`, which will be then replaced with the ssh public key given by Packer.

codenrhoden · 2020-12-07T21:14:46Z

/lgtm

I think this is ready to go! I'll leave this for about 24 hours or so and see if anyone has anything else to say, but I'm happy to merge and iterate. Huge effort @dongsupark!

codenrhoden · 2020-12-09T20:43:23Z

/approve

🎉 🚀

k8s-ci-robot · 2020-12-09T20:43:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: codenrhoden, dongsupark

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~images/capi/OWNERS~~ [codenrhoden]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dongsupark · 2020-12-10T07:34:02Z

/retest

k8s-ci-robot · 2020-12-10T07:34:16Z

@dongsupark: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jsturtevant · 2020-12-10T19:35:13Z

/ok-to-test

k8s-ci-robot requested review from dims and detiber June 5, 2020 15:16

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 5, 2020

k8s-ci-robot requested a review from codenrhoden June 5, 2020 15:16

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 5, 2020

k8s-ci-robot assigned vincepri Jun 5, 2020

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 8, 2020

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 8, 2020

vincepri removed their assignment Jun 10, 2020

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 16, 2020

dongsupark force-pushed the dongsu/capi-flatcar branch from 9e35bd9 to 15c3800 Compare June 16, 2020 08:23

k8s-ci-robot added do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 16, 2020

dongsupark force-pushed the dongsu/capi-flatcar branch from 15c3800 to 9737ee3 Compare June 16, 2020 08:25

k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Jun 16, 2020

CecileRobertMichon reviewed Jun 17, 2020

View reviewed changes

CecileRobertMichon mentioned this pull request Jun 17, 2020

Flatcar support for CAPZ kubernetes-sigs/cluster-api-provider-azure#629

Closed

CecileRobertMichon reviewed Jun 17, 2020

View reviewed changes

codenrhoden suggested changes Jun 22, 2020

View reviewed changes

dongsupark mentioned this pull request Jun 24, 2020

capi: rewrite capi image-builder for Flatcar kinvolk/image-builder#3

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 1, 2020

dongsupark force-pushed the dongsu/capi-flatcar branch 2 times, most recently from 037b956 to d56f11b Compare December 3, 2020 16:27

Dongsu Park and others added 10 commits December 7, 2020 13:15

capi: use variables build_name instead of hard-coding

d7e3648

We should be able to set Flatcar channels and versions via `build_name`, `FLATCAR_CHANNEL`, or `FLATCAR_VERSION`, to use also non-default channels like alpha.

capi: use build_name in general packer configs

8298419

We should use `build_name` as-is, to avoid issues that can happen in builds for other distros.

capi: remove unnecessary only directives

6ab0a4f

Since we started simply using the `build_name` variable, we do not need to have `only` directives for checking for Flatcar. Simply remove the `only` directives.

capi: run bootstrap-flatcar.sh only if Flatcar

4d3e6a5

We should run bootstrap-flatcar.sh only in case of Flatcar. We need to check the if clauses in `exec_command`. It is not possible to check for the `BUILD_NAME` in `inline` or `script`.

dongsupark force-pushed the dongsu/capi-flatcar branch from d56f11b to 3d00a5d Compare December 7, 2020 16:47

k8s-ci-robot assigned codenrhoden Dec 7, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 7, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 9, 2020

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Dec 10, 2020

k8s-ci-robot merged commit fda3464 into kubernetes-sigs:master Dec 10, 2020

dongsupark deleted the dongsu/capi-flatcar branch December 11, 2020 08:29

carstenkoester mentioned this pull request Dec 11, 2020

capi packer qemu build no longer builds qemu images; unconditionally builds vagrant boxes #454

Closed

t-lo mentioned this pull request Jul 28, 2021

Cluster-API: upstream support for most common providers flatcar/Flatcar#445

Closed

15 tasks

dongsupark mentioned this pull request Sep 24, 2021

Improve CAPI support in Flatcar images #701

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

capi: support Flatcar Container Linux #248

capi: support Flatcar Container Linux #248

dongsupark commented Jun 5, 2020 •

edited

Loading

k8s-ci-robot commented Jun 5, 2020

notes

TODOs

k8s-ci-robot commented Jun 5, 2020

dongsupark commented Jun 5, 2020

k8s-ci-robot commented Jun 8, 2020

t-lo commented Jun 8, 2020

dongsupark commented Jun 16, 2020

CecileRobertMichon Jun 17, 2020

dongsupark Jun 18, 2020

CecileRobertMichon Jun 17, 2020

dongsupark Jun 18, 2020

dongsupark Jul 6, 2020

codenrhoden left a comment

codenrhoden Jun 22, 2020

dongsupark Jul 6, 2020

dongsupark commented Jun 23, 2020

dongsupark commented Dec 3, 2020

codenrhoden commented Dec 7, 2020

codenrhoden commented Dec 9, 2020

k8s-ci-robot commented Dec 9, 2020

dongsupark commented Dec 10, 2020

k8s-ci-robot commented Dec 10, 2020

jsturtevant commented Dec 10, 2020

		@@ -0,0 +1,27 @@
		-----BEGIN RSA PRIVATE KEY-----
		MIIEogIBAAKCAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzI

	- name: Creates unit file directory
	file:
	path: /etc/systemd/system/containerd.service.d
	state: directory

	- name: Create containerd boot order drop in file
	template:
	dest: /etc/systemd/system/containerd.service.d/boot-order.conf
	src: etc/systemd/system/containerd.service.d/boot-order.conf

	- name: Create containerd memory pressure drop in file
	template:
	dest: /etc/systemd/system/containerd.service.d/memory-pressure.conf
	src: etc/systemd/system/containerd.service.d/memory-pressure.conf

	- name: Create containerd max tasks drop in file
	template:
	dest: /etc/systemd/system/containerd.service.d/max-tasks.conf
	src: etc/systemd/system/containerd.service.d/max-tasks.conf

capi: support Flatcar Container Linux #248

capi: support Flatcar Container Linux #248

Conversation

dongsupark commented Jun 5, 2020 • edited Loading

notes

TODOs

k8s-ci-robot commented Jun 5, 2020

notes

TODOs

k8s-ci-robot commented Jun 5, 2020

dongsupark commented Jun 5, 2020

k8s-ci-robot commented Jun 8, 2020

t-lo commented Jun 8, 2020

dongsupark commented Jun 16, 2020

CecileRobertMichon Jun 17, 2020

Choose a reason for hiding this comment

dongsupark Jun 18, 2020

Choose a reason for hiding this comment

CecileRobertMichon Jun 17, 2020

Choose a reason for hiding this comment

dongsupark Jun 18, 2020

Choose a reason for hiding this comment

dongsupark Jul 6, 2020

Choose a reason for hiding this comment

codenrhoden left a comment

Choose a reason for hiding this comment

codenrhoden Jun 22, 2020

Choose a reason for hiding this comment

dongsupark Jul 6, 2020

Choose a reason for hiding this comment

dongsupark commented Jun 23, 2020

dongsupark commented Dec 3, 2020

codenrhoden commented Dec 7, 2020

codenrhoden commented Dec 9, 2020

k8s-ci-robot commented Dec 9, 2020

dongsupark commented Dec 10, 2020

k8s-ci-robot commented Dec 10, 2020

jsturtevant commented Dec 10, 2020

dongsupark commented Jun 5, 2020 •

edited

Loading