cilium hubble enable|disable are asymmetric

[ensonic]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

When I have cilium installed via helm with all defaults, I can use cilium hubble enable --ui to enable the hubble ui + relay. The confusing part is that I can't disable them using cilium hubble disable --ui.

As a workaround I've tried:

> cilium hubble enable --helm-set hubble.ui.enabled=false,hubble.relay.enabled=false
// Error: Unable to enable Hubble: services "hubble-peer" already exists

Cilium Version

cilium-cli: v0.12.12 compiled with go1.19.4 on linux/amd64
cilium image (default): v1.12.5
cilium image (stable): v1.12.7
cilium image (running): v1.12.7

Kernel Version

5.19.11-1rodete1-amd64 cilium/cilium#1 SMP PREEMPT_DYNAMIC Debian 5.19.11-1rodete1 (2022-10-31) x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:43:11Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[nbusseneau]

Hello, thanks for the report! This is probably related to #751. In a nutshell, the Cilium CLI does not interact with Helm to apply its changes, instead it does its own thing, so there are quirks around mixing both. We are aware of the issue and would very much love to fix it, but haven't been able to complete that work as of yet.

FWIW the command cilium hubble disable is supposed to disable both Hubble Relay and Hubble UI, no matter if --ui was specified or not at cilium hubble enable time. I haven't tried if that would work in your case (again, Helm quirks are possible), but it's worth a try!

[ensonic]

Yes cilium hubble disable works, but it also disabled hubble, right? Quick test:

cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 1, Ready: 1/1, Available: 1/1
Containers:       cilium             Running: 1
                  cilium-operator    Running: 1
Cluster Pods:     13/13 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.7@sha256:8cb6b4742cc27b39e4f789d282a1fc2041decb6f5698bfe09112085a07b1fd61: 1
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.7@sha256:80f24810bf8484974c757382eb2c7408c9c024e5cb0719f4a56fba3f47695c72: 1

So hubble is off right now. Lets enable

cilium hubble enable
⚠️  Error parsing helm cli secret: unable to retrieve helm values secret kube-system/cilium-cli-helm-values: secrets "cilium-cli-helm-values" not found
⚠️  Proceeding in unknown installation state
🔮 Auto-detected cilium version v1.12.7
🔑 Found CA in secret cilium-ca
ℹ️  helm template --namespace kube-system cilium cilium/cilium --version 1.12.7 --set hubble.enabled=true,hubble.relay.enabled=true,hubble.tls.ca.cert=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURGRENDQWZ5Z0F3SUJBZ0lSQUpVYnFER29CaWE2WVNZdWo2eXJGNGt3RFFZSktvWklodmNOQVFFTEJRQXcKRkRFU01CQUdBMVVFQXhNSlEybHNhWFZ0SUVOQk1CNFhEVEl6TURJeE5URTBNRGN5T0ZvWERUSTJNREl4TkRFMApNRGN5T0Zvd0ZERVNNQkFHQTFVRUF4TUpRMmxzYVhWdElFTkJNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DCkFROEFNSUlCQ2dLQ0FRRUF3WHk5S0lzWU14ZzAyQ013MG9zalJHdG5mZ1A1TjRlQlpVKzdnU0lkUTcvT2U2VkIKVDB3Z2pySmU5UWlSaGtjeVp2RjQyV0EvK3pnNHphTnlHUE9pZkh5dEtuZ0oxQnBRZytmcHFlRWN5ZTJZVzcyLwpBUWlLWW9qamhvZ3lmbllQOW5TSXFGbXY2ZXZyaGhZL3grYTY3QmI1dWpHNERaNmh0UDBjWitJcjk2V0lWcW1mCnN6dG9NUVV1eWN5TmVCbkxxanNOWVUwWmtNOGNLOFNSd0NYaDh5dVJqd1RSN09FWGEwek1PWG5UYzdOeFJmdmoKa2VSRG0ydjlOUGpUQkZyenYzc1FtVXN6MTRUYXRGNkZibXFrTmEyRURIcWNNR2NDbENVeFZrdEkrNVBuNyt1OQp0UHd5U2kzU21vRGZtVFV0QllHZnJKMzZyb3ovY2lGdkxON3Z6d0lEQVFBQm8yRXdYekFPQmdOVkhROEJBZjhFCkJBTUNBcVF3SFFZRFZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01BOEdBMVVkRXdFQi93UUYKTUFNQkFmOHdIUVlEVlIwT0JCWUVGS0tlMW05UHBTUUZPS0JWNGc4VXlJNGVHK2dZTUEwR0NTcUdTSWIzRFFFQgpDd1VBQTRJQkFRQ1R1M0xTVjBqeVBSNFp2RWpVTUIyeGIzVHZTKzNIdTVmRWE1U3l0Um5mWXNlakcwTGdScHE2Cndna2d0OVFNMzNVVUZNT1dES0RqZHpUM1kySGsxdUtZY3VBb2FMN25Wd1pKVm1PTUtoUjB5bVJ3VXFzTGhTbGEKdUxtSWNvZDdDU2dsNXV0NHB6dlNHZDlkcm5XQ1NuY1pYQVd3UTRyZEkwY1Q4Skw4Qzl4NUZvN2orZzFRWXZ1RgpJVmdMdDNaa3NhSUJFVjJpb0x2TTdxdDltMFYyZ3JWY2xGTnpCdFIyNmlZaDY4QXQ1eDZsUGlCTG9QMit6YTFICmVTN0FYYnR3TWhpZHRSbWxrUE8rbEJCT2dMSEtqcDRFZVY0eUowaWFrMytBYWZlWUtuWDBzMG95OVQyODNhSWUKem9MTmNoTUlnS2pYdmVJdkFpWHIzL1ZxOHQvZ0VjNksKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=,hubble.tls.ca.key=[--- REDACTED WHEN PRINTING TO TERMINAL (USE --redact-helm-certificate-keys=false TO PRINT) ---]
✨ Patching ConfigMap cilium-config to enable Hubble...
🚀 Creating ConfigMap for Cilium version 1.12.7...
♻️  Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
🚀 Creating Peer Service...

Error: Unable to enable Hubble: services "hubble-peer" already exists

Hmm.

Error parsing helm cli secret: unable to retrieve helm values secret kube-system/cilium-cli-helm-values: secrets "cilium-cli-helm-values" not found

Marked as fixed, but regressed? #959

Error: Unable to enable Hubble: services "hubble-peer" already exists

Don't know yet.....

I really wonder how people are using it. WHat is a recommended low-overhead default setup where the observability UI can be enabled/disabled as needed. Maybe it is better to add a notes to the docs and call out that this does not yet work as you want?

[nbusseneau]

WHat is a recommended low-overhead default setup where the observability UI can be enabled/disabled as needed.

Either going full Helm or going full CLI (i.e. cilium install and then cilium hubble ... should work, unlike Helm install + cilium hubble...). AFAIK it's only mixing the two that causes issues, so you should be safe to stick with Helm rollouts to enable/disable Hubble without using cilium hubble commands.

Maybe it is better to add a notes to the docs and call out that this does not yet work as you want?

I agree the docs are not that great in that regard, since the docs mostly recommend using cilium install and implicitly expect it will work, but of course there are still many users that use Helm instead (e.g. because of CI versioning). I think everybody kinda hoped the situation would be resolved fast enough for it not to be a huge issue, but it dragged on. If you see something specific that would benefit from improvements, may I suggest submitting a PR? It should be a quick win, that would be very appreciated :)

cc @tklauser in case there are complementary comments.

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has not seen any activity since it was marked stale.
Closing.