unable to retrieve the complete list of server APIs: karpenter.sh/v1alpha5

[akunduru9]

panic: failed to setup machine provider id indexer: failed to get API group resources: unable to retrieve the complete list of server APIs: karpenter.sh/v1alpha5: the server could not find the requested resource

goroutine 1 [running]:
github.com/samber/lo.must({0x25f73e0, 0xc0005f9800}, {0xc000b0f730, 0x1, 0x1})
github.com/samber/[email protected]/errors.go:51 +0x1c5
github.com/samber/lo.Must0(...)
github.com/samber/[email protected]/errors.go:72
github.com/aws/karpenter-core/pkg/operator.NewOperator()
github.com/aws/[email protected]/pkg/operator/operator.go:182 +0x16ce
main.main()
github.com/aws/karpenter/cmd/controller/main.go:33 +0x25

I am using latest tag of karpenter helm charts and EKS version we have is 1.28

[jonathan-innis]

Which version of Karpenter are you running? You typically see this error when you don't have the Machine CRD installed, which means that the controller can't register the indexer that it needs to start running.

[akunduru9]

public.ecr.aws/karpenter/controller:v0.32.3@sha256:afa0d0fd5ac375859dc3d239ec992f197cdf01f6c8e3413e3845a43c2434621e is the version being used

[jonathan-innis]

You need the Machine CRD installed when you are running v0.32.x. This should already be included in the chart. How are you installing Karpenter?

[jonathan-innis]

It's hard for me to parse through the script to see what it's doing. Are you installing all of the CRDs that are part of the chart in the crds/ directory? There's a callout for installing all of the CRDs for the chart here in the upgrade guide: https://karpenter.sh/v0.32/upgrading/upgrade-guide/#:~:text=In%20general%2C%20you%20can%20reapply%20the%20CRDs%20in%20the%20crds%20directory%20of%20the%20Karpenter%20helm%20chart

[akunduru9]

ya I am installing crds using this command helm template karpenter "${SCRIPTPATH}/karpenter-provider-aws-${VERSION#v}/charts/karpenter-crd" --namespace karpenter >| "${CRDMANIFEST}" it would install everything under https://github.com/aws/karpenter-provider-aws/tree/main/charts/karpenter-crd

[jonathan-innis]

I think the difference here is that you need it to install the CRDs based on the version that you are on. If you are installing from main for a version that isn't latest, this is going to break. Try https://github.com/aws/karpenter-provider-aws/tree/v0.32.4/charts/karpenter-crd/templates

[akunduru9]

sure do we also need to install chart we had here https://github.com/aws/karpenter-provider-aws/blob/v0.32.4/charts/karpenter-crd/Chart.yaml

[akunduru9]

used this command to execute helm template karpenter "${SCRIPTPATH}/karpenter-provider-aws-0.32.4/charts/karpenter-crd/templates" --namespace karpenter >| "${CRDMANIFEST}" it errored out saying Error: Chart.yaml file is missing

[jonathan-innis]

I think based on your script, you need to move up one directory so the Chart.yaml file is at the top of the path. The main point is that you need to point to the versioned thing, not just the latest thing. The version you are installing the CRDs from should match the version of the image you are installing.

[akunduru9]

I successfully managed and installed versioned CRDs (Custom Resource Definitions). However, this time I'm encountering a different error. Any ideas on what might be causing this new issue?

ERROR:

webhook.ValidationWebhook","message":"Reconcile error","commit":"3a61217","knative.dev/traceid":"454a0c6a-e510-4d5f-a0df-3fda8b797e9b","knative.dev/key":"validation.webhook.karpenter.sh","duration":"55.054059ms","error":"failed to update webhook: Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io "validation.webhook.karpenter.sh": the object has been modified; please apply your changes to the latest version and try again

{"level":"DEBUG","time":"2024-01-05T15:20:37.358Z","logger":"controller.provisioner","message":"ignoring pod, configured to not run on a Karpenter provisioned node via karpenter.sh/provisioner-name DoesNotExist and karpenter.sh/nodepool DoesNotExist requirements","commit":"3a61217","pod":"karpenter/karpenter-8c994fc78-tw9pp"}

Thanks !

[jonathan-innis]

These are both expected. The first one is a transient error which is discussed here: #2902. The other one is just telling you that a Karpenter deployment can't run on capacity that is launched by Karpenter. This is to just keep you from having a cyclical dependency where you have Karpenter deployed to infra, but then the infra that Karpenter is running on goes bad and Karpenter can't recover itself because it manages the infra that it's running on.

If you are seeing that error continually, that means that you don't have enough static capacity (we recommend using MNG or Fargate) to run the Karpenter controller.

[akunduru9]

is there any documentation for using fargate to run karpenter.if we pass reource limits in manifest file will it resolve the issue without fargate below is configuration

  containers:
    - name: controller
      resources:
        requests:
          memory: "128Mi "  
          cpu: "250m"    
        limits:
          memory: "256Mi" 
          cpu: "500m"

[jonathan-innis]

Are you using MNG or Fargate? If MNG, how many nodes are provisioned as part of your MNG?

[akunduru9]

we are using managed node groups and currently we have two node groups provisioned

[akunduru9]

I was able to fix all the errors mentioned above, and the pods are running fine. However, when I tried to test Karpenter by deploying an app, seeing nodepool errors can you point me to any reference for defining nodepools .

{"level":"INFO","time":"2024-01-11T01:16:11.699Z","logger":"controller","message":"Starting workers","commit":"3a61217","controller":"nodeclass","controllerGroup":"karpenter.k8s.aws","controllerKind":"EC2NodeClass","worker count":10}
{"level":"INFO","time":"2024-01-11T01:16:11.761Z","logger":"controller","message":"Starting workers","commit":"3a61217","controller":"state.provisioner","controllerGroup":"karpenter.sh","controllerKind":"Provisioner","worker count":10}
{"level":"INFO","time":"2024-01-11T01:16:11.761Z","logger":"controller","message":"Starting workers","commit":"3a61217","controller":"state.nodepool","controllerGroup":"karpenter.sh","controllerKind":"NodePool","worker count":10}
{"level":"DEBUG","time":"2024-01-11T01:16:12.029Z","logger":"controller.pricing","message":"updated spot pricing with instance types and offerings","commit":"3a61217","instance-type-count":730,"offering-count":2667}
{"level":"DEBUG","time":"2024-01-11T01:16:14.718Z","logger":"controller.pricing","message":"updated on-demand pricing","commit":"3a61217","instance-type-count":760}
{"level":"INFO","time":"2024-01-11T14:23:07.257Z","logger":"controller.provisioner","message":"no nodepools or provisioners found","commit":"3a61217"}
{"level":"INFO","time":"2024-01-11T14:23:17.258Z","logger":"controller.provisioner","message":"no nodepools or provisioners found","commit":"3a61217"}
{"level":"INFO","time":"2024-01-11T14:23:27.258Z","logger":"controller.provisioner","message":"no nodepools or provisioners found","commit":"3a61217"}
{"level":"INFO","time":"2024-01-11T14:23:37.275Z","logger":"controller.provisioner","message":"no nodepools or provisioners found","commit":"3a61217"}
{"level":"INFO","time":"2024-01-11T14:23:47.260Z","logger":"controller.provisioner","message":"no nodepools or provisioners found","commit":"3a61217"}

[radhikathorbole]

Hi @akunduru9
I'm stuck with the same error {"level":"INFO","time":"2024-04-08T07:57:36.388Z","logger":"controller.provisioner","message":"no nodepools or provisioners found","commit":"3a61217"}
Can you guide me how did you resolve it?