Pricing Endpoints in EKS Private Clusters

[hekki]

Tell us about your request

Existing implementations automatically select the pricingAPIRegion by region.
https://github.com/aws/karpenter/blob/077c099b4419d0caac447904d9ffae1d871e37b9/pkg/cloudprovider/aws/pricing.go#L86-L91

However, I would like to determine the pricingAPIRegion myself without following this logic.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

This problem occurs, for example, when karpenter is used in a region starting with "ap-" and the use of ap-south-1 is restricted by AWS SCPs.
Removing the restriction on the use of ap-south-1 by AWS SCPs would solve this problem, but we do not want to do that as much as possible because it would increase the security risk. Instead, we want to use us-east-1 for pricingAPIRegion even if the region starts with "ap-".

Are you currently working around this issue?

No, we have not been able to work around this issue.

Additional Context

No response

Attachments

No response

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[bwagner5]

We're trying to understand the pricing API endpoints a little better ourselves to be able to implement something more flexible. Endpoint configuration is likely something we can support. Regardless, it looks like it would be possible to specify an SCP that only allows the pricing API getPrices in ap-south-1 which would give you the lowest latency request to the pricing endpoint. Is that a possible workaround for you?

[hekki]

@bwagner5

Thanks for the reply. Also, thank you for considering supporting Endpoint configuration.

it looks like it would be possible to specify an SCP that only allows the pricing API getPrices in ap-south-1 which would give you the lowest latency request to the pricing endpoint. Is that a possible workaround for you?

Yes, we're working around the issue that way now. So we're not in a hurry to resolve this issue. However, we believe that we need to support Pricing API Endpoint configuration to make the SCP policy more robust :)

[samuelthan]

Hi, anymore updates on this ? I'm in similar boat where by our organisation lock down AWS Regions that we don't use.

[ellistarn]

@samuelthan does the static pricing not work for you? We hardcode fallback pricing at release time, and bake it into the binary.

[samuelthan]

@samuelthan does the static pricing not work for you? We hardcode fallback pricing at release time, and bake it into the binary.

@ellistarn it doesn't seem to work as i deploy my EKS cluster in the ap-southeast-2 region. It seem to be grabbing the endpoint of ap-south-1 as it's endpoint which is a region we don't support at the moment.

Here's a question, if i were to update the karpenter's chart's deployment value of AWS_REGION environment to us-east-1
at https://github.com/aws/karpenter/blob/main/charts/karpenter/values.yaml#L104. Will that "tell" karpenter to use the pricing endpoint with us-east-1 even though my EKS cluster is located at ap-southeast-2 ?

[tbugfinder]

meanwhile the code got slightly updated covering us, ap, eu pricing endpoints.

Actually PR #4495 adds a regression to deployments running in eu-* and having SCPs in place which deny use of eu-central-1.

https://github.com/aws/karpenter/blob/main/pkg/providers/pricing/pricing.go#L80-L93

// NewPricingAPI returns a pricing API configured based on a particular region
func NewAPI(sess *session.Session, region string) pricingiface.PricingAPI {
	if sess == nil {
		return nil
	}
	// pricing API doesn't have an endpoint in all regions
	pricingAPIRegion := "us-east-1"
	if strings.HasPrefix(region, "ap-") {
		pricingAPIRegion = "ap-south-1"
	} else if strings.HasPrefix(region, "cn-") {
		pricingAPIRegion = "cn-northwest-1"
	} else if strings.HasPrefix(region, "eu-") {
		pricingAPIRegion = "eu-central-1"
	}
	return pricing.New(sess, &aws.Config{Region: aws.String(pricingAPIRegion)})
}

So if SCPs restrict region to eu-west-1 the pricing api in eu-central-1 is queried and that region has to be enabled.
Overall it would be very beneficial if we could override this value by an environment variable or a fallback to us-east-1 is in place (e.g. if SCP denies access to picked princing endpoint).

Verification outside of karpenter:

$ aws pricing get-products --service-code AmazonEC2 --filters "Type=TERM_MATCH,Field=instanceType,Value=m7a.xlarge" "Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)" --region eu-central-1 | jq -rc '.PriceList[]' | jq -r '[ .product.attributes.servicecode, .product.attributes.location, .product.attributes.instancesku?, .product.attributes.instanceType, .product.attributes.usagetype, .product.attributes.operatingSystem, .product.attributes.memory, .product.attributes.physicalProcessor, .product.attributes.processorArchitecture, .product.attributes.vcpu, .product.attributes.currentGeneration, .terms.OnDemand[].priceDimensions[].unit, .terms.OnDemand[].priceDimensions[].pricePerUnit.USD, .terms.OnDemand[].priceDimensions[].description] | @csv'

An error occurred (AccessDeniedException) when calling the GetProducts operation: User: arn:aws:sts::012345678901:assumed-role/AWSReservedSSO_User_0122121211/adminadmin is not authorized to perform: pricing:GetProducts with an explicit deny in a service control policy

[njtran]

@tbugfinder, just like the author of the issue, do you want to rely on us-east-1 for all your pricing queries?

Does the post that @bwagner5 suggested work for you as a workaround for the eu partition?

it looks like it would be possible to specify an SCP that only allows the pricing API getPrices in ap-south-1 which would give you the lowest latency request to the pricing endpoint. Is that a possible workaround for you?

[tbugfinder]

@njtran Relying on us-east-1 (at least as a fallback) would be ok for my usecase, although I like the approach to use closest API endpoints.

I don't have control on SCPs so I asked the responsible team to review and open the endpoint in eu-central-1 - awaiting feedback on that...

[shivkonaje]

Any updates on this? scp's are blocking eu-central-1. Would be ideal to have an override env variable for pricing end point region

[njtran]

@shivkonaje
#5704 This was recently merged which should ignore the ISOLATED_VPC environment variable when doing spot pricing lookups. This should be included in the next release.