Hardcoded conversion webhook namespace in primary Karpenter helm chart incompatible with running in different namespaces

[dschaaff]

Description

Description

We run Karpenter in the karpenter namespace and not kube-system. This is primarily for historical reasons. We were early adopters of Karpenter and it is risky and not worthwhile to move the install over the kube-system namespace. We also do not use the karpenter-crd chart. Flux manages the helm install and has a feature to ensure CRDs are upgraded when the helm chart updates. See https://fluxcd.io/flux/components/helm/helmreleases/#controlling-the-lifecycle-of-custom-resource-definitions

The Karpenter chart installs CRDs with conversion webhooks client config hardcoded to the kube-system namespace. Unfortunately this now blocks us from updating to v1 without also migrating to the karpenter-crd chart.

Reproduction Steps
Manage karpenter using flux with crds: CreateReplace set on the install and upgrade specs of the flux helmrelease manifest

---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: karpenter
  namespace: flux-system
spec:
  chart:
    spec:
      chart: karpenter
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: karpenter
      version: 0.37.1
  install:
    createNamespace: true
    crds: CreateReplace
  upgrade:
    crds: CreateReplace
    remediation:
      retries: 1
  interval: 10m0s
  releaseName: karpenter
  targetNamespace: karpenter
  storageNamespace: karpenter

• update to 0.37.1

The controller logs an error

{"level":"INFO","time":"2024-08-23T15:23:56.820Z","logger":"controller","message":"k8s.io/[email protected]/tools/cache/reflector.go:232: failed to list *v1.NodePool: conversion webhook for karpenter.sh/v1beta1, Kind=NodePool failed: Post \"https://karpenter.kube-system.svc:8443/conversion/karpenter.sh?timeout=30s\": service \"karpenter\" not found","commit":"5bdf9c3"}
karpenter-5657b5578c-w6wqj controller 

This blocks the v1 upgrade for user's relying on flux when running Karpenter in any namespace besides kube-system.

Expected Behavior:

The karpenter helm chart should support being installed in any namespace with full functionality.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jmdeal]

Going to close this issue in favor of #6818. If you have any additional feedback please feel free to add it there.