Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subnetSelector and securityGroupSelector expect " or n, but found [ #1850

Closed
FernandoMiguel opened this issue May 24, 2022 · 2 comments
Closed
Labels
bug Something isn't working

Comments

@FernandoMiguel
Copy link
Contributor

Version

Karpenter: "0.10.1"

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
WARNING: version difference between client (1.24) and server (1.22) exceeds the supported minor version skew of +/-1
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.6-eks-14c7a48", GitCommit:"35f06c94ad99b78216a3d8e55e04734a85da3f7b", GitTreeState:"clean", BuildDate:"2022-04-01T03:18:05Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Expected Behavior

to be able to use a list in the selector, like it does in the AZs, to use jsonencode instead of join in the code below

Actual Behavior

│ Error: default failed to run apply: error when applying patch:
│ {
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"karpenter.sh/v1alpha5\",\"kind\":\"Provisioner\",\"metadata\":{\"annotations\":{},\"name\":\"default\"},\"spec\":{\"limits\":{\"resources\":{\"cpu\":1000}},\"provider\":{\"instanceProfile\":\"KarpenterNodeInstanceProfile--precise-mako\",\"securityGroupSelector\":{\"aws-ids\":\"sg-XXXX, sg-XXXX, sg-XXXX\"},\"subnetSelector\":{\"aws-ids\":[\"subnet-XXXXX\",\"subnet-XXXXX\",\"subnet-XXXXX\",\"subnet-XXXXX\",\"subnet-XXXXX\"]},\"tags\":{\"BusinessUnit\":\"cts\",\"Component\":\"eks-modules\",\"Environment\":\"dev\",\"OWNER\":\"Fernando\",\"Product\":\"XXX\",\"SourceCode\":\"https://gitXXX\",\"Team\":\"XXXX\",\"TerraformCreated\":\"true\"}},\"requirements\":[{\"key\":\"topology.kubernetes.io/zone\",\"operator\":\"In\",\"values\":[\"us-east-1a\",\"us-east-1b\",\"us-east-1c\",\"us-east-1d\",\"us-east-1f\"]},{\"key\":\"karpenter.sh/capacity-type\",\"operator\":\"In\",\"values\":[\"spot\",\"on-demand\"]},{\"key\":\"node.kubernetes.io/instance-type\",\"operator\":\"In\",\"values\":[\"c6a.large\",\"c6i.large\"]},{\"key\":\"kubernetes.io/arch\",\"operator\":\"In\",\"values\":[\"arm64\",\"amd64\"]}],\"topologySpreadConstraints\":[{\"maxSkew\":1,\"topologyKey\":\"topology.kubernetes.io/zone\",\"whenUnsatisfiable\":\"ScheduleAnyway\"},{\"maxSkew\":1,\"topologyKey\":\"kubernetes.io/hostname\",\"whenUnsatisfiable\":\"ScheduleAnyway\"}],\"ttlSecondsAfterEmpty\":120,\"ttlSecondsUntilExpired\":2592000}}\n"
        }
    },
    "spec": {
        "limits": {
            "resources": {
                "cpu": 1000
            }
        },
        "provider": {
            "subnetSelector": {
                "aws-ids": [
                    "subnet-XXXXX",
                    "subnet-XXXXX",
                    "subnet-XXXXX",
                    "subnet-XXXXX",
                    "subnet-XXXXX"
                ]
            }
        },
        "topologySpreadConstraints": [
            {
                "maxSkew": 1,
                "topologyKey": "topology.kubernetes.io/zone",
                "whenUnsatisfiable": "ScheduleAnyway"
            },
            {
                "maxSkew": 1,
                "topologyKey": "kubernetes.io/hostname",
                "whenUnsatisfiable": "ScheduleAnyway"
            }
        ]
    }
}
│ to:
│ Resource: "karpenter.sh/v1alpha5, Resource=provisioners", GroupVersionKind: "karpenter.sh/v1alpha5, Kind=Provisioner"
│ Name: "default", Namespace: ""
│ for: "/var/folders/0l/wk5qj7wj6sxf8pkkx6phm3y80000gn/T/847115903kubectl_manifest.yaml": admission webhook "validation.webhook.provisioners.karpenter.sh" denied the request: validation failed: v1alpha1.AWS.SubnetSelector: ReadString: expects " or n, but found [, error found in #10 byte of ...|aws-ids": [
    "subnet-XXXXX|..., bigger context ...|-XXXX"
},
"subnetSelector": {
    "aws-ids": [
        "subnet-XXXXX","subnet-XXXX|...:
│
│   with module.base_system.module.karpenter[
            0
        ].kubectl_manifest.karpenter_provisioner,
│   on /Users/fernando/modules/karpenter/karpenter.tf line 112, in resource "kubectl_manifest" "karpenter_provisioner":
│  112: resource "kubectl_manifest" "karpenter_provisioner" {

Resource Specs and Logs

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: "topology.kubernetes.io/zone"
      operator: In
      values: ${azs}
    - key: "karpenter.sh/capacity-type"
      operator: In
      values: ${capacity-type}
    - key: "node.kubernetes.io/instance-type" #If not included, all instance types are considered
      operator: In
      values: ${instance-type}
    - key: "kubernetes.io/arch" #If not included, all architectures are considered
      operator: In
      values: ${arch}
  limits:
    resources:
      cpu: ${limits_resources_cpu}
  provider:
    instanceProfile: ${iam-instance-profile-id}
    subnetSelector:
      aws-ids: ${private_subnets}
    securityGroupSelector:
      aws-ids: ${security_groups}
    tags: ${tags}
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "topology.kubernetes.io/zone"
      whenUnsatisfiable: ScheduleAnyway
    - maxSkew: 1
      topologyKey: "kubernetes.io/hostname"
      whenUnsatisfiable: ScheduleAnyway
  ttlSecondsAfterEmpty: ${ttlSecondsAfterEmpty}
  ttlSecondsUntilExpired: ${ttlSecondsUntilExpired}
# Deploying default provisioner for Karpenter autoscaler
data "kubectl_path_documents" "karpenter_provisioners" {
  pattern = "${path.module}/provisioners/default_provisioner.yaml"
  vars = {
    azs                     = jsonencode(var.aws_availability_zones)
    capacity-type           = jsonencode(var.capacity-type)
    instance-type           = jsonencode(var.instance-type)
    arch                    = jsonencode(var.arch)
    tags                    = jsonencode(data.aws_default_tags.aws_default_tags.tags)
    private_subnets         = join(", ", var.private_subnets)
    security_groups         = join(", ", var.security_groups)
    limits_resources_cpu    = var.limits_resources_cpu
    iam-instance-profile-id = aws_iam_instance_profile.karpenter.name
    eks-cluster-id          = var.eks_cluster_id
    ttlSecondsAfterEmpty    = var.ttlSecondsAfterEmpty
    ttlSecondsUntilExpired  = var.ttlSecondsUntilExpired
  }
}
@FernandoMiguel FernandoMiguel added the bug Something isn't working label May 24, 2022
@tzneal
Copy link
Contributor

tzneal commented May 24, 2022

This is related to #1327 , we couldn't make that change now as it's backwards incompatible and would break existing customers.

@FernandoMiguel
Copy link
Contributor Author

This is related to #1327 , we couldn't make that change now as it's backwards incompatible and would break existing customers.

Thanks for linking it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants