Failed to get /openapi/v2 and /swagger.json: Created Component, but require templated one.

[anden-dev]

Hi there,

I have the current kubernetes-external-secrets 6.0.0 running on a couple of eks clusters, and after a pod restart it is crash looping now.

npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~prestart: [email protected]
npm info lifecycle [email protected]~start: [email protected]

> [email protected] start /app
> ./bin/daemon.js

{"level":"info","time":1605871823096,"pid":17,"hostname":"kube-external-secrets-kubernetes-external-secrets-76dbbf4cwkwmg","msg":"loading kube specs"}
Error: Failed to get /openapi/v2 and /swagger.json: Created Component, but require templated one. This is a bug. Please report: https://github.com/silasbw/fluent-openapi/issues
at /app/node_modules/kubernetes-client/lib/swagger-client.js:58:15
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async main (/app/bin/daemon.js:33:3)
npm info lifecycle [email protected]~start: Failed to exec start script
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `./bin/daemon.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm timing npm Completed in 3308ms

the clusters are:

eks 1.17 / eks.3 - v1.17.9
eks 1.18 / eks.2 - v1.18.9

It is not a cert problem like #511 but also related to swagger-client.
silasbw/swagger-fluent#55
godaddy/kubernetes-client#666

[anden-dev]

hmm #543 looks like my problem, but it was closed with:
Found the solution for our problem, we needed to revert deployment of kube-metrics-adapter from v0.1.9 to v0.1.8.

Yet I do not have kube-matrics-adapter running in the clusters.

[Flydiverny]

Ugh, this seems to come back every now and then. Underlying issue has been that some api resource in the cluster can't be parsed by the dependency KES uses (or well the dependency of our dependency), when the kubernetes api is generated in runtime. :\

[Flydiverny]

It's hard to patch without knowing what fails to be processed. Have you upgrades any other resource in the cluster recently since it was last working? Probably something that provides a CRD

[anden-dev]

You touch quite the subject there. I am advocating constantly updating charts, and always do: I guess we will have that discussion in my team again.

But CRDS is a good pointer. Thanks.

I will check and get back here: my bet is on prometheus-adapter or kube-prometheus-stack.

[anden-dev]

I narrowed it down to kube-prometheus-stack the CRDS coming with it
are causing the crash-loop.

Testet Helm Chart Versions:
12.2.2 (current)
11.1.1
I am still looking for one that is not breaking external-secrets

[anden-dev]

I was wrong.

The problem is coming from k8s-prometheus-adapter.

image: v0.8.2 modifies the openapi

k get apiservice v1beta1.custom.metrics.k8s.io 

image v0.7.0 is not doing that.

remediation steps:

1. redploy k8s-prometheus-adapter with image v0.7.0
2. delete the apiservice coming from v0.8.2
3. restart kubernetes-external-secrets

k delete apiservice v1beta1.custom.metrics.k8s.io

Everything back to normal.

[patrickleet]

so this is still a thing with newer version of prometheus adapter a year later it seems

[Flydiverny]

Well, nothing has really changed on KES side and since its one of the dependencies in KES complaining even if prometheus adapter could probably implement some workaround its not really their issue.

Anyway check out #864