Chicken-egg loop issue with pods that depend on secrets created by kubernetes-external-secrets?

[moseb]

Hi!

Thanks for sharing kubernetes-external-secrets as software libre! 👍

We ran into a bit of a chicken-egg problem with kubernetes-external-secrets (version 1.2.0):

• We have a helm chart that contains a pod and an external secret.
• The pod can only be created when the secret is around.
• The external secret makes kubernetes-external-secrets create the secret eventually.
• But the regular secret is not available during helm upgrade --install ... so the the upgrade/installation fails.

For a workaround, I break the loop by "downloading" secrets JSON off AWS secrets manager using AWS CLI, turning that JSON into a base64-encoded YAML secret resource for k8s and apply it using kubectl.

That same process is needed when adding new keys to the secret. Because helm install would fail and kubernetes-external-secrets will not copy the new keys because the external secret still lacks these keys. Ticket #94 to be related.

I wonder if I have to accept this problem as conceptual and permanent, because it's not very convenient.
Any ideas?

Thanks in advance!

[moseb]

Any ideas, any feedback?

[moseb]

Please please 😄

[anarcher]

IMHO, I think this problem can’t solve with external secerts. Humm..

[anarcher]

I have an idea about it that using Argocd PreSync 1 and custom health checking. 2 can be addressed if you use Argocd for deployment.

But, We need a way to check two resources (ExternalSecret and Secret) . If ExternalSecret`s annotations copies to Secret, We can check the annotations are equals or not. (Or Using ExternalSecret own version)

What do you think that? :->

[moseb]

(I have no experience with Argocd, so I cannot really comment on that, yet.)

[jbuettnerbild]

Damn, that was exactly was i plan to do too. Than i need a second livecycle to deploy the external secrets before i deploy the helm charts.

[j-zimnowoda]

Today, I was experimenting with helm upgrade command and non existing secrets at the time of release deployment. It seems that chicken-egg phenomena is not a problem, because helm provides timeout parameter and Kubernetes implements various retry mechanisms. Here are my findings:

helm upgrade --install myrelease mychart

Above command will install the release and exists without errors.

# helm list
NAME     	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART        	APP VERSION
myrelease	default  	1       	2021-01-14 19:09:12.324568 +0100 CET	deployed	mychart-0.1.0	1.16.0

The deployed Pods will be CreateContainerConfigError state as there is no secret available and the k8s retry mechanism takes place.

# k get po
NAME                                      READY   STATUS                       RESTARTS   AGE
myrelease-mychart-6f7cd8f886-fln5j        0/1     CreateContainerConfigError   0          5s

If later, an expected secret is created the Pod changes its status to Running

# k get po
NAME                                      READY   STATUS    RESTARTS   AGE
myrelease-mychart-6f7cd8f886-fln5j        1/1     Running   0          2m5s

The problem with helm may occur if:

1. helm upgrade command is performed with --atomic, --wait, --timeout flags. In this case increasing timeout may help but it does not guarantee that expected Secret will be created in time by kubernetes-external-secrets controller. By default helm sets timeout to 5m0s..

2. the helm chart defines tests: For that purpose, I recommend defining tests as Jobs (not Pods) with backoffLimit: 3, so Secret must appear within 60 seconds.

3. there are helm hooks: solution similar as in 2)

My helm is in version v3.4.2

I hope it clarifies the issue.

[Flydiverny]

Closing this as I don't really see how we can improve / resolve this with KES.
But, yes, there can be a chicken-egg scenario depending on how you deploy your resources.