Conversion webhook hardcoded at kube-system namespace #6818

marcofranssen · 2024-08-21T08:30:24Z

Description

Observed Behavior:

When upgrading our karpenter to the v1.0.0 chart it fails at the conversion webhook which is targeting the kube-system namespace. Our karpenter is deployed in the karpenter namespace.

controller logs

{"level":"ERROR","time":"2024-08-21T09:02:09.801Z","logger":"controller","message":"k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.NodeCl │
│ aim: failed to list *v1.NodeClaim: conversion webhook for karpenter.sh/v1beta1, Kind=NodeClaim failed: Post \"https://karpenter.kube-system.svc:8443/conversion/karpenter.sh?timeout=30s\": service  │
│ \"karpenter\" not found","commit":"5bdf9c3"}

Expected Behavior:

The conversion webhook targets the Helm Release namespace.

Reproduction Steps (Please include YAML):

Install Karpenter in the karpenter namespace using the release prior to v1.0.0. Then upgrade karpenter to v1.0.0.

Versions:

Chart Version: v1.0.0
Kubernetes Version (kubectl version): 1.30

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

The text was updated successfully, but these errors were encountered:

marcofranssen · 2024-08-21T09:19:48Z

Turns out this is controller by

https://github.com/aws/karpenter-provider-aws/blob/main/charts/karpenter-crd/values.yaml#L5

This is something that should be made very clear in the documentation or migration guide.

It isn't mentioned here https://aws.amazon.com/blogs/containers/announcing-karpenter-1-0/

woehrl01 · 2024-08-21T11:41:21Z

@marcofranssen We just stumbled over this as well. At best, this could be by default the namespace of the helm chart deployment.

marcofranssen · 2024-08-21T11:49:17Z

yes indeed. that would even be better

denniszag · 2024-08-21T11:49:32Z

I encountered the same problem using ArgoCD. The karpenter-crd chart is installed correctly in the karpenter namespace. But the karpenter chart also installs the same CRDs (using ArgoCD) but wants to change it to kube-system namespace.
An option to disable installing CRDs using the karpenter chart can help.

marcofranssen · 2024-08-21T11:51:35Z

I encountered the same problem using ArgoCD. The karpenter-crd chart is installed correctly in the karpenter namespace. But the karpenter chart also installs the same CRDs (using ArgoCD) but wants to change it to kube-system namespace. An option to disable installing CRDs using the karpenter chart can help.

You can already do so by --skip-crds.

woehrl01 · 2024-08-21T12:20:15Z

@marcofranssen using that value only works if you install the "karpenter-crd" helm chart, if you use the "normal" one, they are symlinked and replacement does not happen. (https://github.com/aws/karpenter-provider-aws/tree/28da0b96b6086679f75e656d31ac65bd7fca2bc0/charts/karpenter)

woehrl01 · 2024-08-21T12:35:01Z

Looks like the modification is made with a post job hook, and the referenced image digest is not compatible with ARM

woehrl01 · 2024-08-21T15:08:32Z

Related to #6819 #6765

Vinaum8 · 2024-08-22T20:10:05Z

Related to #6544

marcofranssen · 2024-08-26T09:55:19Z

Seems to be resolved in #6827

jmdeal · 2024-08-26T23:53:02Z

There's some additional justification given here, but if you need to template the CRDs (required if installing Karpenter outside of kube-system or customizing the service), you will need to use the karpenter-crd chart. This is why the karpenter-crd chart is used in the v1 migration guide, with an alternative of manually patching the CRDs.

As far as what's missing from our docs, I do think we can be a little more explicit. We do instruct users to install the CRD chart as part of the installation guide, but without a justification of why I can understand why users would continue to just use the standard chart since it already includes the CRDs, just without templating.

booleanbetrayal · 2024-08-27T13:52:20Z

As far as what's missing from our docs, I do think we can be a little more explicit. We do instruct users to install the CRD chart as part of the installation guide, but without a justification of why I can understand why users would continue to just use the standard chart since it already includes the CRDs, just without templating.

@jmdeal - In our case, in an ArgoCD environment, it's not the lack of clear instructions or justifications for using the karpenter-crd chart ... it's that you cannot deploy the CRDs from the karpenter-crd at all over ArgoCD managed resources, because of .Release.Service / managed-by and release-name stamping. You get fun errors like:

Error: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition "ec2nodeclasses.karpenter.k8s.aws" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "karpenter-crd"

So you're stuck with the situation of having to figure it out yourself and potentially dealing with the Application construction as we are discussing in #6847.

frimik · 2024-09-12T16:05:41Z

For now, "fixing" it with an override on top of the helm chart. Because the validation in the webhook is still useful for me on older Kubernetes versions.

From Tanka with ❤️.

{
          local conversionWebhookSpecMixin = {
            spec+: {
              conversion+: {
                webhook+: {
                  clientConfig+: {
                    service+: {
                      namespace: c.karpenter.namespace,
                    },
                  },
                },
              },
            },
          },

      karpenter: helm.template(releaseName, './vendor/charts/karpenter', {
          apiVersions: apiVersions,
          includeCrds: includeCrds,
          kubeVersion: kubeVersion,
          namespace: c.karpenter.namespace,
          noHooks: noHooks,
          values: values,
        }) + {
          // https://github.com/aws/karpenter-provider-aws/issues/6818
          'custom_resource_definition_ec_2nodeclasses.karpenter.k_8s.aws'+: conversionWebhookSpecMixin,
          'custom_resource_definition_nodeclaims.karpenter.sh'+: conversionWebhookSpecMixin,
          'custom_resource_definition_nodepools.karpenter.sh'+: conversionWebhookSpecMixin,
       },
}

frimik · 2024-09-12T16:33:18Z

Seems to be backported to v1.0.2. See #6855, #6849 and then there are possibly several other open issues at least partially related to this issue and fix. Right?

icicimov · 2024-10-23T05:41:07Z

I'm seeing the same error:

{"level":"ERROR","time":"2024-10-23T04:35:12.676Z","logger":"controller","message":"k8s.io/[email protected]/tools/cache/reflector.go:232: Failed to watch *v1.NodePool: failed to list *v1.NodePool: conversion webhook for karpenter.sh/v1beta1, Kind=NodePool failed: Post \"https://karpenter.kube-system.svc:8443/conversion/karpenter.sh?timeout=30s\": service \"karpenter\" not found","commit":"f587167"}
panic: Conversion webhook enabled but unable to complete call: Timeout: failed waiting for *v1.NodePool Informer to sync

but for karpenter chart 0.37.5 (not using karpenter-crd chart) installed in karpenter NameSpace:

$ kubectl get pods -n karpenter karpenter-5bdb98d7c-clpn9 -o yaml | grep 'image:'
    image: public.ecr.aws/karpenter/controller:0.37.5@sha256:df978d853e0d96f905eac1e7abd87075e9fb617ce610bee3dd9a4f947c31d573

I thought this should be an issue with 1.0+ as per the above conversations? Bit suspicious that it is looking for version v1.NodePool though right (since 0.37.5 is still v1beta1 API) ?

honarkhah · 2024-10-23T18:38:04Z

Thanks to all this worked out for me

$ export KARPENTER_VERSION=1.0.6
$ helm registry logout public.ecr.aws
$ helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd  --version "${KARPENTER_VERSION}" --namespace karpenter --create-namespace
$ helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" -f values/karpenter-values.yaml -n karpenter --skip-crds

ketozhang · 2024-11-01T00:43:19Z

Those that installs Karpenter as a Helm subchart are susceptible to this issue.

Options are to move Karpenter out of the parent chart OR to install karpenter-crd separately.

Makeshift · 2024-11-07T16:26:31Z

Additionally, the service that Karpenter creates is named after the release name.

The webhook seems to be hardcoded to look for a service named karpenter, so mine was erroring due to me having called the release karpenter-controller.

corkupine · 2024-12-05T19:26:22Z

I thought this should be an issue with 1.0+ as per the above conversations? Bit suspicious that it is looking for version v1.NodePool though right (since 0.37.5 is still v1beta1 API) ?

@icicimov - I am running into the same issue with 0.35.11 while preparing to migrate to 1.0. Were you able to resolve it?

icicimov · 2024-12-09T01:24:54Z

@corkupine in preparation to 1.0 move I just split the charts into separate dependency as recommended in the docs:

dependencies:
  - name: karpenter-crd
    version: 0.37.5
    repository: oci://public.ecr.aws/karpenter
  - name: karpenter
    version: 0.37.5
    repository: oci://public.ecr.aws/karpenter

Vinaum8 · 2024-12-11T11:49:01Z

@corkupine in preparation to 1.0 move I just split the charts into separate dependency as recommended in the docs:
dependencies:
  - name: karpenter-crd
    version: 0.37.5
    repository: oci://public.ecr.aws/karpenter
  - name: karpenter
    version: 0.37.5
    repository: oci://public.ecr.aws/karpenter

I did this too, I'm using argocd and it works.

marcofranssen added bug Something isn't working needs-triage Issues that need to be triaged labels Aug 21, 2024

Vinaum8 mentioned this issue Aug 22, 2024

no service port 8443 found for service "karpenter" after migrating from cluster autoscaler #6544

Open

jmdeal mentioned this issue Aug 26, 2024

Hardcoded conversion webhook namespace in primary Karpenter helm chart incompatible with running in different namespaces #6864

Closed

jakeskyaws mentioned this issue Sep 24, 2024

NodeClaim namespace hardcoded to kube-system aws-samples/karpenter-blueprints#17

Closed

awoimbee mentioned this issue Oct 23, 2024

TLS handshake error from API server #6898

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conversion webhook hardcoded at kube-system namespace #6818

Conversion webhook hardcoded at kube-system namespace #6818

marcofranssen commented Aug 21, 2024 •

edited

Loading

marcofranssen commented Aug 21, 2024

woehrl01 commented Aug 21, 2024

marcofranssen commented Aug 21, 2024

denniszag commented Aug 21, 2024

marcofranssen commented Aug 21, 2024

woehrl01 commented Aug 21, 2024 •

edited

Loading

woehrl01 commented Aug 21, 2024 •

edited

Loading

woehrl01 commented Aug 21, 2024

Vinaum8 commented Aug 22, 2024

marcofranssen commented Aug 26, 2024

jmdeal commented Aug 26, 2024

booleanbetrayal commented Aug 27, 2024

frimik commented Sep 12, 2024

frimik commented Sep 12, 2024 •

edited

Loading

icicimov commented Oct 23, 2024

honarkhah commented Oct 23, 2024 •

edited

Loading

ketozhang commented Nov 1, 2024 •

edited

Loading

Makeshift commented Nov 7, 2024

corkupine commented Dec 5, 2024 •

edited

Loading

icicimov commented Dec 9, 2024

Vinaum8 commented Dec 11, 2024

Conversion webhook hardcoded at kube-system namespace #6818

Conversion webhook hardcoded at kube-system namespace #6818

Comments

marcofranssen commented Aug 21, 2024 • edited Loading

Description

marcofranssen commented Aug 21, 2024

woehrl01 commented Aug 21, 2024

marcofranssen commented Aug 21, 2024

denniszag commented Aug 21, 2024

marcofranssen commented Aug 21, 2024

woehrl01 commented Aug 21, 2024 • edited Loading

woehrl01 commented Aug 21, 2024 • edited Loading

woehrl01 commented Aug 21, 2024

Vinaum8 commented Aug 22, 2024

marcofranssen commented Aug 26, 2024

jmdeal commented Aug 26, 2024

booleanbetrayal commented Aug 27, 2024

frimik commented Sep 12, 2024

frimik commented Sep 12, 2024 • edited Loading

icicimov commented Oct 23, 2024

honarkhah commented Oct 23, 2024 • edited Loading

ketozhang commented Nov 1, 2024 • edited Loading

Makeshift commented Nov 7, 2024

corkupine commented Dec 5, 2024 • edited Loading

icicimov commented Dec 9, 2024

Vinaum8 commented Dec 11, 2024

marcofranssen commented Aug 21, 2024 •

edited

Loading

woehrl01 commented Aug 21, 2024 •

edited

Loading

woehrl01 commented Aug 21, 2024 •

edited

Loading

frimik commented Sep 12, 2024 •

edited

Loading

honarkhah commented Oct 23, 2024 •

edited

Loading

ketozhang commented Nov 1, 2024 •

edited

Loading

corkupine commented Dec 5, 2024 •

edited

Loading