Add helm-based installation for hubble integration

[aanm]

Besides switched to a helm-based installation. This PR also enables TLS for hubble. Previously, cilium-cli had the TLS disabled for hubble-ui

[aanm]

putting in draft since the CI is failing a lot

[michi-covalent]

tested with gke. enabling hubble is working, but i'm seeing issues with cilium hubble port-forward and hubble-ui 🤔

✅ install cilium

% ./cilium install
🔮 Auto-detected Kubernetes kind: GKE
ℹ️  using Cilium version "v1.11.3"
🔮 Auto-detected cluster name: gke-cilium-dev-us-west2-a-michi-23762
🔮 Auto-detected IPAM mode: kubernetes
🔮 Auto-detected datapath mode: gke
✅ Detected GKE native routing CIDR: 10.32.0.0/14
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=gke-cilium-dev-us-west2-a-michi-23762,cni.binPath=/home/kubernetes/bin,encryption.nodeEncryption=false,gke.disableDefaultSnat=true,gke.enabled=true,ipam.mode=kubernetes,ipv4NativeRoutingCIDR=10.32.0.0/14,kubeProxyReplacement=disabled,nodeinit.enabled=true,nodeinit.reconfigureKubelet=true,nodeinit.removeCbrBridge=true,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator
🚀 Creating Resource quotas...
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.11.3...
🚀 Creating GKE Node Init DaemonSet...
🚀 Creating Agent DaemonSet...
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
♻️  Restarting unmanaged pods...
♻️  Restarted unmanaged pod kube-system/event-exporter-gke-5479fd58c8-vnmw4
♻️  Restarted unmanaged pod kube-system/konnectivity-agent-6db7c55769-hn7f5
♻️  Restarted unmanaged pod kube-system/konnectivity-agent-autoscaler-6849fff748-hstzp
♻️  Restarted unmanaged pod kube-system/kube-dns-697dc8fc8b-rbvhg
♻️  Restarted unmanaged pod kube-system/kube-dns-autoscaler-844c9d9448-cbqbr
♻️  Restarted unmanaged pod kube-system/l7-default-backend-69fb9fd9f9-bwrq6
♻️  Restarted unmanaged pod kube-system/metrics-server-v0.4.5-56fcb96597-l5ghr
✅ Cilium was successfully installed! Run 'cilium status' to view installation health

✅ enable hubble with ui

 % ./cilium hubble enable --ui
🔑 Found CA in secret cilium-ca
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=gke-cilium-dev-us-west2-a-michi-23762,cni.binPath=/home/kubernetes/bin,encryption.nodeEncryption=false,gke.disableDefaultSnat=true,gke.enabled=true,hubble.enabled=true,hubble.listenAddress=:4244,hubble.relay.enabled=true,hubble.relay.tls.server.enabled=true,hubble.socketPath=/var/run/cilium/hubble.sock,hubble.tls.enabled=true,hubble.ui.enabled=true,hubble.ui.securityContext.enabled=false,ipam.mode=kubernetes,ipv4NativeRoutingCIDR=10.32.0.0/14,kubeProxyReplacement=disabled,nodeinit.enabled=true,nodeinit.reconfigureKubelet=true,nodeinit.removeCbrBridge=true,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator
✨ Patching ConfigMap cilium-config to enable Hubble...
🚀 Creating ConfigMap for Cilium version 1.11.3...
♻️  Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
✨ Generating certificates...
🔑 Generating certificates for Relay...
✨ Deploying Relay...
✨ Generating certificates...
🔑 Generating certificates for UI...
✨ Deploying Hubble UI and Hubble UI Backend...
⌛ Waiting for Hubble to be installed...
✅ Hubble was successfully enabled!

⁉️ port-forward command seems to get stuck:

% ./cilium hubble port-forward
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=gke-cilium-dev-us-west2-a-michi-23762,cni.binPath=/home/kubernetes/bin,encryption.nodeEncryption=false,gke.disableDefaultSnat=true,gke.enabled=true,hubble.enabled=true,hubble.listenAddress=:4244,hubble.relay.enabled=true,hubble.relay.tls.server.enabled=true,hubble.socketPath=/var/run/cilium/hubble.sock,hubble.tls.enabled=true,hubble.ui.enabled=true,hubble.ui.securityContext.enabled=false,ipam.mode=kubernetes,ipv4NativeRoutingCIDR=10.32.0.0/14,kubeProxyReplacement=disabled,nodeinit.enabled=true,nodeinit.reconfigureKubelet=true,nodeinit.removeCbrBridge=true,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator
...

⁉️ hubble-ui is not loading

i see errors like these in hubble-ui backend container log:

level=error msg="flow error: EOF\n" subsys=ui-backend
level=error msg="fetching hubble flows: connecting to hubble-relay (attempt #1) failed: rpc error: code = Canceled desc = context canceled\n" subsys=ui-backend
...

[michi-covalent]

tested again with the updated branch, and everything worked.

✅ cilium install
✅ cilium hubble enable
✅ cilium hubble port-forward and then hubble observe.
✅ cilium hubble ui and checked kube-system namespace shows flows.

[rolinh]

I tested the changes and it is functional. However, I have some concerns, a low one, a medium one and a high one :)

1. (low) Why is helm template ... printed out when running cilium hubble port-forward? I wouldn't expect this command to be printed out for commands that I don't expect to change anytning in the cluster.
2. (medium) We now print the b64 encoded TLS keypair when installing. Could we somehow hide or truncate it before printing?

% cilium hubble enable --ui
🔑 Found CA in secret cilium-ca
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.11.3 --set cluster.id=0,cluster.name=kind-kind,encryption.nodeEncryption=false,hubble.enabled=true,hubble.listenAddress=:4244,hubble.relay.enabled=true,hubble.relay.tls.server.enabled=true,hubble.socketPath=/var/run/cilium/hubble.sock,hubble.tls.ca.cert=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNGRENDQWJxZ0F3SUJBZ0lVZkM0MldQRzZOdFBmdXJWMS9IZGljUEJ6dnRvd0NnWUlLb1pJemowRUF3SXcKYURFTE1Ba0dBMVVFQmhNQ1ZWTXhGakFVQmdOVkJBZ1REVk5oYmlCR2NtRnVZMmx6WTI4eEN6QUpCZ05WQkFjVApBa05CTVE4d0RRWURWUVFLRXdaRGFXeHBkVzB4RHpBTkJnTlZCQXNUQmtOcGJHbDFiVEVTTUJBR0ExVUVBeE1KClEybHNhWFZ0SUVOQk1CNFhEVEl5TURRd056RTRNRE13TUZvWERUSTNNRFF3TmpFNE1ETXdNRm93YURFTE1Ba0cKQTFVRUJoTUNWVk14RmpBVUJnTlZCQWdURFZOaGJpQkdjbUZ1WTJselkyOHhDekFKQmdOVkJBY1RBa05CTVE4dwpEUVlEVlFRS0V3WkRhV3hwZFcweER6QU5CZ05WQkFzVEJrTnBiR2wxYlRFU01CQUdBMVVFQXhNSlEybHNhWFZ0CklFTkJNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBEQVFjRFFnQUVPOFMyUGxNNHEwcVhSUGJrN01wOURuVk8KTnhVcWx4L3M3VVdUVERlTzBQQnNqdjZDa25JUmNkK2RMZW1oUjhsSzZkZGhDQUd6eHYvMXRJSm5YRXNmQjZOQwpNRUF3RGdZRFZSMFBBUUgvQkFRREFnRUdNQThHQTFVZEV3RUIvd1FGTUFNQkFmOHdIUVlEVlIwT0JCWUVGTGtuCiszTnlKdDJVRHN4Z3NDY01RR0ZjaXdoL01Bb0dDQ3FHU000OUJBTUNBMGdBTUVVQ0lRQ1JDcjZHcC9heThBS1IKNXJXWklabm54dXBmalZ0ZktuVTcrVGxQVE5vVFp3SWdUT3JIcno4UmE1dUwzbXBGNk9DdFd0MkIrVVI1TjVlSApJeDJjTGJOcU1NMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=,hubble.tls.ca.key=LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUxVRmFreStMNmttcW14S3ZVc25jYzhtdXRxRmVaNmlrZ1ExQWwxdlZXbEFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTzhTMlBsTTRxMHFYUlBiazdNcDlEblZPTnhVcWx4L3M3VVdUVERlTzBQQnNqdjZDa25JUgpjZCtkTGVtaFI4bEs2ZGRoQ0FHenh2LzF0SUpuWEVzZkJ3PT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=,hubble.tls.enabled=true,hubble.ui.enabled=true,hubble.ui.securityContext.enabled=false,ipam.mode=kubernetes,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
✨ Patching ConfigMap cilium-config to enable Hubble...
🚀 Creating ConfigMap for Cilium version 1.11.3...
♻️  Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
✨ Generating certificates...
🔑 Generating certificates for Relay...
✨ Deploying Relay...
✨ Generating certificates...
🔑 Generating certificates for UI...
✨ Deploying Hubble UI and Hubble UI Backend...
⌛ Waiting for Hubble to be installed...
✅ Hubble was successfully enabled!

3. (high) I think that enabling TLS for the Relay service should be done as part of another PR as it affects the user experience. I want to make it clear that overall I think we should enable TLS (or even mTLS) by default for the Relay service. However, doing so without further consideration breaks current user's workflows and would require us to also update Cilium documentation. Indeed, when a user now runs cilium hubble port-forward and then run a Hubble CLI command such as hubble status or hubble observe, they are greeted with the following message (after some time elapsed):

% hubble status
failed to connect to 'localhost:4245': context deadline exceeded: connection closed before server preface received

This is because the Hubble CLI needs to be told to use TLS, e.g. by providing the --tls flag. However, in itself, this is insufficient as users also need to either pass the scary --tls-allow-insecure OR import the CA and validate via server name (which is rather painful to do.
I think that ultimately, we should update the Hubble CLI so that it reads the kubeconfig and does the right thing for things to work out of the box (without even requiring cilium hubble port-forward). Could we then create issues for relevant work to enable TLS by default and drop this change from this PR?

[michi-covalent]

andre claims that robin gave a +1 offline. merging.

[qmonnet]

@aanm Have you tested this change with Kind, by any chance? Trying with the latest release, I'm getting the following error:

$ cilium hubble enable 
🔑 Found CA in secret cilium-ca

Error: Unable to enable Hubble:  unable to retrieve helm values secret kube-system/cilium-cli-helm-values: secrets "cilium-cli-helm-values" not found

I bisected to commit 269a984, but can't tell for sure whether this is a bug or if I'm doing something wrong.

[EDIT] Sorted this out offline. This is because I installed Cilium with Helm, instead of using cilium install as I should have. Follow-up: We'll have to update the docs that explain how to install Cilium on Kind (currently recommending to use Helm).