Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nodeClaim label domain is invalid after 1.1.0 update #7471

Closed
joke opened this issue Dec 2, 2024 · 8 comments
Closed

nodeClaim label domain is invalid after 1.1.0 update #7471

joke opened this issue Dec 2, 2024 · 8 comments
Labels
bug Something isn't working triage/needs-information Marks that the issue still needs more information to properly triage

Comments

@joke
Copy link

joke commented Dec 2, 2024

Description

Observed Behavior:

After upgrading to 1.1.0 karpenter can not create nodeclaims any more.

level: ERROR
time: '2024-12-02T11:57:49.729Z'
logger: controller
message: Reconciler error
commit: 0a85efb
controller: provisioner
namespace: ''
name: ''
reconcileID: 65e78b27-e60a-4763-9a77-820b60957497
error: >-
  creating node claim, NodeClaim.karpenter.sh "infrastructure-7zhm2" is invalid:
  spec.requirements[5].key: Invalid value: "string": label domain
  "karpenter.k8s.aws" is restricted; creating node claim, NodeClaim.karpenter.sh
  "infrastructure-t62zd" is invalid: spec.requirements[7].key: Invalid value:
  "string": label domain "karpenter.k8s.aws" is restricted; creating node claim,
  NodeClaim.karpenter.sh "infrastructure-hchvj" is invalid:
  spec.requirements[6].key: Invalid value: "string": label domain
  "karpenter.k8s.aws" is restricted
errorCauses:
  - error: >-
      creating node claim, NodeClaim.karpenter.sh "infrastructure-7zhm2" is
      invalid: spec.requirements[5].key: Invalid value: "string": label domain
      "karpenter.k8s.aws" is restricted
  - error: >-
      creating node claim, NodeClaim.karpenter.sh "infrastructure-t62zd" is
      invalid: spec.requirements[7].key: Invalid value: "string": label domain
      "karpenter.k8s.aws" is restricted
  - error: >-
      creating node claim, NodeClaim.karpenter.sh "infrastructure-hchvj" is
      invalid: spec.requirements[6].key: Invalid value: "string": label domain
      "karpenter.k8s.aws" is restricted

Expected Behavior:

The node claims should be created without problem and new node should join the cluster.

Reproduction Steps (Please include YAML):

The node pool definition. If you need any more information I happy to help.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: infrastructure
spec:
  disruption:
    consolidateAfter: 1m
    consolidationPolicy: WhenEmptyOrUnderutilized
  template:
    metadata:
      labels:
        istio-version: 1.24.1
    spec:
      expireAfter: 168h
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: bottlerocket
      requirements:
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values:
        - m8g
        - m7g
        - m7a
        - m6g
        - m6a
        - m5a
        - c8g
        - c7g
        - c7a
        - c6g
        - c6a
        - c5a
        - r8g
        - r7g
        - r7a
        - r6g
        - r6a
        - r5a
      - key: kubernetes.io/arch
        operator: In
        values:
        - arm64
        - amd64
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - on-demand
        - spot
      - key: kubernetes.io/os
        operator: In
        values:
        - linux
      startupTaints:
        - key: cni.istio.io/not-ready
          value: 'true'
          effect: NoExecute

Versions:

  • Chart Version: 1.1.0
  • Kubernetes Version (kubectl version): v1.31.2-eks-7f9249a
@joke joke added bug Something isn't working needs-triage Issues that need to be triaged labels Dec 2, 2024
@RomainBelorgey
Copy link
Contributor

Seeing the same issue and quite blocked because of this.
Is that related to this PR #7423 ? (so these lines = https://github.com/aws/karpenter-provider-aws/blob/main/pkg/apis/crds/karpenter.sh_nodepools.yaml#L209-L210)

@thanhtruc123456
Copy link

Any update anymore ? I also got this error when upgrade to 1.1.0

@jmdeal
Copy link
Contributor

jmdeal commented Dec 2, 2024

Can you ensure that you've upgraded your CRDs? I suspect you're still on the v1.0 version of the CRDs, which don't allowlist the karpenter.k8s.aws/ec2nodeclass label.

@jmdeal jmdeal added triage/needs-information Marks that the issue still needs more information to properly triage and removed needs-triage Issues that need to be triaged labels Dec 2, 2024
@RomainBelorgey
Copy link
Contributor

Can you ensure that you've upgraded your CRDs? I suspect you're still on the v1.0 version of the CRDs, which don't allowlist the karpenter.k8s.aws/ec2nodeclass label.

For me it is installing the helm chart 1.1.0, so it should be good ? 🤔

@ssh2003
Copy link

ssh2003 commented Dec 2, 2024

Have the same issue

@omkensey
Copy link

omkensey commented Dec 2, 2024

Can you ensure that you've upgraded your CRDs? I suspect you're still on the v1.0 version of the CRDs, which don't allowlist the karpenter.k8s.aws/ec2nodeclass label.

For me it is installing the helm chart 1.1.0, so it should be good ? 🤔

If you installed the CRDs using just the karpenter chart that has the CRDs in the special crds directory, subsequent Helm upgrades won't update or alter the already-installed CRDs (this is a Helm behavior, not something specific to Karpenter). You probably need to update them manually per the Note in step 5 of the Upgrade procedure.

@jmdeal
Copy link
Contributor

jmdeal commented Dec 2, 2024

subsequent Helm upgrades won't update or alter the already-installed CRDs

Exactly this. This is called out explicitly in the v1 migration guide, but it should be implicit for all versions. I've cut a PR to correct the wording in the upgrade guide.

@joke
Copy link
Author

joke commented Dec 3, 2024

@omkensey your right my fault. It was a bit more complicated in my case. I removed the karpenter helm CRD charts from the ArgoCd setup and forgot to re-enabled CRD processing for the main karpenter chart.

Thanks.

@joke joke closed this as completed Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage/needs-information Marks that the issue still needs more information to properly triage
Projects
None yet
Development

No branches or pull requests

6 participants