feat: allow setting service annotations via helm chart

[jamesduffy]

Fixes #N/A

Description
Allows setting annotations on the karpenter service. We need this so that we can add annotations to setup datadog to monitor karpenter. There are probably other use cases.

How was this change tested?

Generated a helm chart locally with the additional values:

service:
  annotations:
    ad.datadoghq.com/service.checks: |
      {
        "karpenter": {
          "cluster_check": true,
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:8000/metrics",
              "exclude_metrics": [
                "^karpenter\\.go\\..*$"
              ]
            }
          ]
        }
      }

Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: #
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	e8752b6
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/6716c0bc3445cf000831eac8

[jmdeal]

What's preventing you from using the existing additionalAnnotations field? Unless absolutely necessary, we'd like to avoid additional configuration in the helm chart.

[jamesduffy]

What's preventing you from using the existing additionalAnnotations field? Unless absolutely necessary, we'd like to avoid additional configuration in the helm chart.

additionalAnnotations adds the annotations to a lot of different things. This would be picked up from the Datadog agent as separate checks and instead of running just one check for Karpenter it would run duplicate checks for each place additionalAnnotations is injected. Therefore, we would need a way to add the annotation in a single place. Datadog annotations are often on the service.

[github-actions]

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

[jamesduffy]

@jmdeal I would like to move forward with this if my explanation makes sense.

[jmdeal]

Sorry this has slipped, could you elaborate on what the consequences of this are?

This would be picked up from the Datadog agent as separate checks and instead of running just one check for Karpenter it would run duplicate checks for each place additionalAnnotations is injected.

Unless this is strictly blocking, and no other workarounds exist, we don't want to expand the configuration surface for the helm chart we ship.

[jamesduffy]

Sorry this has slipped, could you elaborate on what the consequences of this are?

This would be picked up from the Datadog agent as separate checks and instead of running just one check for Karpenter it would run duplicate checks for each place additionalAnnotations is injected.

Unless this is strictly blocking, and no other workarounds exist, we don't want to expand the configuration surface for the helm chart we ship.

The datadog agent watches for annotations to configure it's checks/metric collection. Using the existing additionalAnnotations would create annotations on more than one resource. This would configure the datadog agent to run multiple collections of the same metrics making it unusable.

If anyone wants to use datadog to collect metrics using datadog's autodiscovery they will also have this problem.

[jmdeal]

Got it, I assume for most resources this would be a no-op, but you can get duplicate metrics since it would scrape both the Karpenter pods and the service?

[jamesduffy]

Correct. In summary, using `additionalAnnotations` would duplicate metrics sent to Datadog. Adding `service.annotations` on the service would allow us to create annotations to create only one Datadog metric collection.

[jmdeal]

Discussed offline with other team members, the use-case seems strong enough. LGTM 🚀

[coveralls]

Pull Request Test Coverage Report for Build 11448527366

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 82.883%

---

Totals
Change from base Build 11445189374:	0.0%
Covered Lines:	5641
Relevant Lines:	6806

---

💛 - Coveralls