make node names unique in tests

[tzneal]

1. Issue, if available:

N/A

2. Description of changes:

• make node names unique with a sequential ID
• if the node already exists, log a debug message

3. How was this change tested?

Unit tests & scaling up/down inflate on EKS.

4. Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: link to issue
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	8b286f8
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/624481aae7a811000910ffa8

[tzneal]

Added this log as it would have saved me a few hours this morning. In a test if you somehow happen to get a duplicate node name, there are no errors reported since it could happen normally if kubelet comes up. The end result is that your test doesn't fail with some "node already exists" error and if it does fail it will look like something is really wrong. In my case, a zonal topology spread was trying to schedule a node in "test-zone-2", the node already existed in "test-zone-1", no error occurred but the test for zone skew failed.

[bwagner5]

I would guess the comment is correct that this is rare, but did you check this in a real cluster to make sure that debug doesn't spam in the common case?

otherwise, lgtm

[tzneal]

I would guess the comment is correct that this is rare, but did you check this in a real cluster to make sure that debug doesn't spam in the common case?

otherwise, lgtm

Yup, nothing normally logs in a real cluster in my testing, I'm not sure how often kubelet would beat us to the node creation, but it doesn't seem likely:

karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:05.858Z	INFO	controller.provisioning	Batched 5 pods in 1.071391856s	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:06.009Z	DEBUG	controller.provisioning	Discovered security groups: [sg-00c5fed6615db429d sg-063c542ddd4538485]	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:06.011Z	DEBUG	controller.provisioning	Discovered kubernetes version 1.21	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:06.062Z	DEBUG	controller.provisioning	Discovered ami-05f60cea94ba81bec for query /aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:06.227Z	DEBUG	controller.provisioning	Created launch template, Karpenter-tnealt-karpenter-demo-1937467200527745945	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:09.255Z	INFO	controller.provisioning	Launched instance: i-0983db01fa351a7a6, hostname: ip-192-168-24-90.us-west-2.compute.internal, type: t3a.2xlarge, zone: us-west-2b, capacityType: on-demand	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:09.273Z	INFO	controller.provisioning	Created node with 5 pods requesting {"cpu":"5125m","pods":"8"} from types c1.xlarge, c3.2xlarge, c4.2xlarge, c6i.2xlarge, c5d.2xlarge and 205 other(s)	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:09.326Z	INFO	controller.provisioning	Waiting for unschedulable pods	{"commit": "8b286f8", "provisioner": "default"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:09.326Z	DEBUG	controller.selection	Relaxing soft constraints for pod since it previously failed to schedule, adding: toleration for PreferNoSchedule taints	{"commit": "8b286f8", "pod": "default/inflate-6b88c9fb68-qlm94"}
karpenter-84894b6b77-7j5gb controller 2022-03-30T16:13:10.327Z	INFO	controller.provisioning	Batched 1 pods in 1.000529468s	{"commit": "8b286f8", "provisioner": "default"}

If it does log, it will be Debug only.