Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix aws cp caching empty instance types set #785

Merged
merged 2 commits into from
Nov 10, 2021

Conversation

bwagner5
Copy link
Contributor

@bwagner5 bwagner5 commented Nov 9, 2021

1. Issue, if available:
N/A

2. Description of changes:

  1. The AWS CP InstanceTypes provider was using a defer func to cache results. If the DescribeInstanceTypes call err'd for any reason, the defer block still runs and caches an empty value. On subsequent reconciliations, the scheduler would not fill in well known labels causing this logging:
2021-11-08T22:39:28.831Z	INFO	controller.allocation.provisioner/default	Ignored pod test1-basic-0-774f48fcd-hkf85/koi-test3, kubernetes.io/arch is too constrained: 
kubernetes.io/os is too constrained: 
node.kubernetes.io/instance-type is too constrained: 
  1. The test I was running received a RequestLimitExceeded on the DescribeInstanceTypes call and resulted in an empty set of instance types cached.
2021-11-08T22:39:23.117Z	ERROR	controller.controller.Allocation	Reconciler error	{"commit": "82226ca", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "name": "default", "namespace": "", "error": "getting instance types, retrieving all instance types, fetching instance types using ec2.DescribeInstanceTypes, RequestLimitExceeded: Request limit exceeded.\n\tstatus code: 503

Since DescribeInstanceTypes should change very rarely, I set a custom cache TTL for the instance types and zones cache. The default it was using was set to 1 minute. I set the new value to 5 minutes.

  1. I cleaned up some comment typos

  2. Cleaned up some redundant error messages since we were wrapping and propagating errors in the InstanceTypes provider.

3. Does this change impact docs?

  • Yes, PR includes docs updates
  • Yes, issue opened: link to issue
  • No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@bwagner5 bwagner5 requested a review from ellistarn November 9, 2021 17:43
@netlify
Copy link

netlify bot commented Nov 9, 2021

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: 1fc775d

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/618c1631eda5f30008dcd4c7

@bwagner5 bwagner5 requested a review from ellistarn November 10, 2021 18:57
@bwagner5 bwagner5 force-pushed the fix-instance-type-empty-cache branch from 2ef451c to 1fc775d Compare November 10, 2021 18:57
@bwagner5 bwagner5 merged commit fb06300 into aws:main Nov 10, 2021
@bwagner5 bwagner5 deleted the fix-instance-type-empty-cache branch November 10, 2021 20:55
gfcroft pushed a commit to gfcroft/karpenter-provider-aws that referenced this pull request Nov 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants