Add Configuration to cilium agent via ConfigMap on ControllerDeployment of Gardener Extension

[veith4f]

How to categorize this issue?
/area control-plane
/area usability
/area quality
/area robustness

/kind enhancement
/kind bug

What happened:
Cilium extension auto-configures itself using internal configmap. Apparently, the idea is to hide the complexity of cilium from users of this extension. However, in some cases the auto-configuration is somewhat lacking, see: #492.

What you expected to happen:
Even if problem indicated by above PR is fixed, real issue persists. Cilium can be used in variety of ways, (kube-proxy replacement, masquerading, xdp, and so on) // users need proper way to configure desired mode of operation. In terms of auto-configuration, there may be few profiles describing probable modes of operation but ultimately, individual cilium parameters should be configurable by an admin using the extension.

How to reproduce it (as minimally and precisely as possible):
does not apply

Anything else we need to know?:

Environment:
latest gardener, latest extension

[veith4f]

Assume I want to configure the extension such that it uses kubeproxyReplacement, XDP and so on. I would like this to be configurable as follows.

apiVersion: v1
kind: ConfigMap
metadata:
  name: myconfigmap
data:
  key1: "foo"
  key2: "bar"
---
apiVersion: core.gardener.cloud/v1beta1
kind: ControllerDeployment
metadata:
  name: networking-cilium
type: helm
providerConfig:
  chart: H4sIAAAAAAAAA+0ca2/b...
  config: myconfigmap

How do you like this idea? Should I implement it and open a PR?

[veith4f]

More over, under given circumstances cilium has trouble automatically figuring out MTU sizes and packet fragmentation options.
See

• Cilium sets wrong MTU to veth-interfaces cilium/cilium#23711 (comment)
• Cilium is using wrong interface for MTU detection cilium/cilium#14339 (comment)
• possible extra encapsulation when accessing nodeport service via external wireguard interface cilium/cilium#26128 (comment)

There is a need for configuration.

[axel7born]

Hi @veith4f,
I get that there is a need for additional configuration options.
Can't we expose those options the same way, as we already do with other options.
What would be the advantage of the ConfigMap?

[veith4f]

Hi @axel7born,

right now, the extension is usually installed via ControllerRegistration and ControllerDeployment. The latter offers a few parameters as defined by https://github.com/gardener/gardener-extension-networking-cilium/blob/master/charts/gardener-extension-networking-cilium/values.yaml. Parameters supplied there need to be juggled along until they ultimately are mounted in https://github.com/gardener/gardener-extension-networking-cilium/blob/master/charts/internal/cilium/charts/agent/templates/daemonset.yaml.

My thought was that there should be sort of a default ConfigMap (as there is now) but additionally another optional one that is watched by the Controller (if it exists) and which is used to override values of the default ConfigMap such that the result of this operation ultimately gets mounted by the cilium agent DaemonSet.

That just seemed like the simplest way to me but any way of exposing as many parameters as possible is fine.

[axel7born]

Configuring a ConfigMap in the ControllerDeployment would set values for all cilium shoots in a landscape. That seems to be a bit odd. Is there really a need for such a change?
Cilium configuration is done via the NetworkConfig . In case there are additional config options needed, they should be exposed that way.

[veith4f]

If I understand correctly, using NetworkConfig allows configuration of individual shoot clusters. That is a worthwhile feature and allows even more control. With that being said, I don't think that landscape-wide configuration is odd. When you design a Kubernetes product and offer it as a digital service, you want to offer a specific feature set that includes the way you set up networking. Of course you can make each cluster a "customer project" where you have a completely individual network setup. But I think this approach does not scale well and gets very difficult to manage very quickly. It is cool to have this option for special needs customers but it seems to me that defining a landscape-wide network standard is an exercise that most gardener users would be very well advised to run.

[axel7born]

The current approach for landscape-wide configuration in gardener extensions would be via helm values.
Or via a webhook that mutates the network config accordingly.

[veith4f]

If I understand correctly, per shoot NetworkConfig is defined in
https://github.com/gardener/gardener-extension-networking-cilium/blob/master/pkg/apis/cilium/types_network.go#L169
and offers just a few options.

Processing happens in https://github.com/gardener/gardener-extension-networking-cilium/blob/master/pkg/controller/actuator_reconcile.go#L96 .

The actuator gathers the different parameter sources and renders the configmap that gets mounted by cilium agent DaemonSet.

In my mind, an ideal solution would allow full configuration both on landscape level and shoot cluster level.

Let's assume that both are implemented and that shoot cluster level can be configured via fully implemented NetworkConfig.
How would you define the parameters at landscape level?

[veith4f]

I am going to try and answer my own question.

All existing parameters are defined in https://github.com/cilium/cilium/blob/main/pkg/option/config.go#L55 FF.
They can be supplied to the command in --param-style or via --config-dir parameter in which case a file may be placed under the relevant path indicating the parameter value.

Either way we have a flat list of parameters that should be exposed both at shoot-cluster level and landscape level.

I propose to make this happen the simplest way possible which is:

1. the gardener extension at landscape level uses a default configmap that contains a flat list of all the parameters initialized with sensible values such as to allow an easy getting started routine for new users
2. all those values can be overridden with values.yaml or kubectl edit
3. the per-shoot networkconfig is replaced with a per-shoot configmap that may or may not exist and that offers the exact same flat list of parameters that the global configmap offers while each individual value may or may not exist.
4. the actuator code get the landscape level configmap, applys the shoot level configmap on top and writes out a new configmap that gets mounted by cilium-agent and cilium-operator.

[veith4f]

@axel7born Any thoughts?

[axel7born]

@veith4f I think there are more aspects to discuss. Should we maybe have a call?
Can you contact me via my email so we could set something up?