Add helm support for custom_cni deployment

[kukacz]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add option to deploy custom_cni type of network backend using helm chart.

There is already manifest based method of custom network backend deployment implemented. Some backend providers (eg. Cilium) offer helm charts of their applications and prefer those for deployment and following upgrades. This change adds such exclusive option in addition to the existing manifest one.

Which issue(s) this PR fixes:

N/A

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: kukacz / name: Lukáš Kubín (f392b36, b4147c0, 77d11d7, 2e313e4, b6ca652, 500644a, 1a94f6e, 8b70c85, 5cf2581, 6b65695)

[k8s-ci-robot]

Welcome @kukacz!

It looks like this is your first PR to kubernetes-sigs/kubespray 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kubespray has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @kukacz. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MrFreezeex]

Hi @kukacz, thanks for the PR! Could you add a test using this?
You would need to add it in .gitlab-ci/packet.yml and the variables in tests/files/packet_${TEST_NAME}.yml with your preferred CNI/helm chart config.

I think we could have a file inventory/sample/group_vars/k8s_cluster/k8s-net-custom-cni.yml as a small documentation for users.

/ok-to-test

[MrFreezeex]

Thanks for the great docs! 🙏 It looks good to me but I think you need to rebase your branch to make the CI happy 🤔. I should be able to trigger the test you added afterwards.

[kukacz]

Thanks for the great docs! 🙏 It looks good to me but I think you need to rebase your branch to make the CI happy 🤔. I should be able to trigger the test you added afterwards.

Thanks @MrFreezeex! Of course, rebased now, basic CI tests passed now. Would you kindly trigger the packet_debian12-custom-cni-helm test for me? I guess I can't start the manual test myself, right?

[MrFreezeex]

Hmmm it seems that the pipeline for stuck 🤔, could you rebase again or amend last commit/push force to trigger it again. And yes correct I will be able to trigger your test manually.

[kukacz]

@MrFreezeex Sure, rebased from upstream master again (no changes) and force-pushed amended commit. Thanks!

[MrFreezeex]

Ok so the pipeline are not reported in github but actually launched, I triggered your new job but it seems to fail unfortunately :(: https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/5529779199

[MrFreezeex]

So now Cilium is crashing unfortunately: https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/5533156698

It shows this in the logs:

==== START logs for container config of pod kube-system/cilium-fhrdq ====
Running
level=info msg=Invoked duration="596.4µs" function="cmd.glob..func36 (build-config.go:32)" subsys=hive
level=info msg=Starting subsys=hive
level=info msg="Establishing connection to apiserver" host="https://10.233.0.1:443" subsys=k8s-client
level=info msg="Connected to apiserver" subsys=k8s-client
level=info msg="Start hook executed" duration=7.484664ms function="client.(*compositeClientset).onStart" subsys=hive
level=info msg="Reading configuration from config-map:kube-system/cilium-config" configSource="config-map:kube-system/cilium-config" subsys=option-resolver
level=info msg="Got 104 config pairs from source" configSource="config-map:kube-system/cilium-config" subsys=option-resolver
level=info msg="Reading configuration from cilium-node-config:kube-system/" configSource="cilium-node-config:kube-system/" subsys=option-resolver
level=info msg="Got 0 config pairs from source" configSource="cilium-node-config:kube-system/" subsys=option-resolver
level=info msg="Start hook executed" duration=7.432645ms function="cmd.(*buildConfig).onStart" subsys=hive
level=info msg=Stopping subsys=hive
level=info msg="Stop hook executed" duration="10.981µs" function="client.(*compositeClientset).onStop" subsys=hive
==== END logs for container config of pod kube-system/cilium-fhrdq ====
==== START logs for container mount-cgroup of pod kube-system/cilium-fhrdq ====
cp: cannot create regular file '/hostbin/cilium-mount': Permission denied
==== END logs for container mount-cgroup of pod kube-system/cilium-fhrdq ====
==== START logs for container apply-sysctl-overwrites of pod kube-system/cilium-fhrdq ====
Request log error: the server rejected our request for an unknown reason (get pods cilium-fhrdq)
==== END logs for container apply-sysctl-overwrites of pod kube-system/cilium-fhrdq ====
==== START logs for container mount-bpf-fs of pod kube-system/cilium-fhrdq ====
Request log error: the server rejected our request for an unknown reason (get pods cilium-fhrdq)
==== END logs for container mount-bpf-fs of pod kube-system/cilium-fhrdq ====
==== START logs for container clean-cilium-state of pod kube-system/cilium-fhrdq ====
Request log error: the server rejected our request for an unknown reason (get pods cilium-fhrdq)
==== END logs for container clean-cilium-state of pod kube-system/cilium-fhrdq ====
==== START logs for container install-cni-binaries of pod kube-system/cilium-fhrdq ====
Request log error: the server rejected our request for an unknown reason (get pods cilium-fhrdq)
==== END logs for container install-cni-binaries of pod kube-system/cilium-fhrdq ====
==== START logs for container cilium-agent of pod kube-system/cilium-fhrdq ====
Request log error: the server rejected our request for an unknown reason (get pods cilium-fhrdq)
==== END logs for container cilium-agent of pod kube-system/cilium-fhrdq ====

BTW adding this: https://github.com/kubernetes-sigs/kubespray/blob/master/tests/files/custom_cni/values.yaml#L11 is probably a good idea but it don't seems to be the reason of cilium failing though...

[kukacz]

Hi @MrFreezeex, it seems that there might be a mount directory ownership issue. Hard to confirm it without access to the tested system. Also, it might be specific to the Debian image or even something else I'm not experiencing in my deployments which are actually using Ubuntu 22.04. And I had no issues deploying there with default chart values.

So now I can try changing the test packet to use Ubuntu image instead Debian, or perhaps apply some force with non-default securityContext.priviledged: true. Do I have an option how to test the change on an environment similar to the CI test one, including the Kubespray configuration used? I mean doing it in my virtual machine or similar, without stressing the CI infra and having to request someone to trigger the manual test for me?

Possibly related: #10499 and cilium/cilium#23838.

[MrFreezeex]

Thanks for the investigation 🙏. So something have probably changed between Cilium 1.13 and 1.14 as the other version of custom_cni Cilium install 1.13.0 fine. The second issue suggested that /opt/cni/bin should be owned by root as well. I think we could change that, it's in roles/network_plugin/cni/tasks/main.yml. Also please add the CIDR in your value https://github.com/kubernetes-sigs/kubespray/blob/master/tests/files/custom_cni/values.yaml#L11 IIRC you will have issues with the test that check if all the pods are in the correct CIDR if you don't do it.

[MrFreezeex]

Thanks for the great work! Here is working gitlab CI job: https://gitlab.com/kargo-ci/kubernetes-sigs-kubespray/-/jobs/5542894180 :D
/lgtm

[kukacz]

/assign @chadswen

[chadswen]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chadswen, kukacz, MrFreezeex

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [chadswen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment