Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replacing cluster specs doesn't work with amazon-vpc-routed-eni networking #5537

Closed
MiLk opened this issue Jul 27, 2018 · 5 comments
Closed

Replacing cluster specs doesn't work with amazon-vpc-routed-eni networking #5537

MiLk opened this issue Jul 27, 2018 · 5 comments
Assignees
@mikesplain

Comments

@MiLk
Copy link

MiLk commented Jul 27, 2018

1. What kops version are you running? The command kops version, will display
this information.
Version 1.10.0-beta.1 (git-dc9154528)
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-18T11:37:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:34:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
kops replace -f cluster.yaml
5. What happened after the commands executed?
The following message was displayed.
error: error replacing cluster: Spec.Networking: Invalid value: "amazon-vpc-routed-eni": amazon-vpc-routed-eni networking is supported only in AWS
6. What did you expect to happen?
Correctly update the spec.
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2018-07-18T10:27:53Z
  name: k8s-stg.example.net
spec:
  additionalNetworkCIDRs:
  - 10.15.0.0/16
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": ["acm:DescribeCertificate", "acm:ListCertificates"],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:AuthorizeSecurityGroupIngress",
            "ec2:CreateSecurityGroup",
            "ec2:CreateTags",
            "ec2:DeleteSecurityGroup",
            "ec2:DescribeInstances",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets",
            "ec2:DescribeTags",
            "ec2:DescribeVpcs",
            "ec2:ModifyInstanceAttribute",
            "ec2:RevokeSecurityGroupIngress"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "elasticloadbalancing:AddTags",
            "elasticloadbalancing:CreateListener",
            "elasticloadbalancing:CreateLoadBalancer",
            "elasticloadbalancing:CreateRule",
            "elasticloadbalancing:CreateTargetGroup",
            "elasticloadbalancing:DeleteListener",
            "elasticloadbalancing:DeleteLoadBalancer",
            "elasticloadbalancing:DeleteRule",
            "elasticloadbalancing:DeleteTargetGroup",
            "elasticloadbalancing:DeregisterTargets",
            "elasticloadbalancing:DescribeListeners",
            "elasticloadbalancing:DescribeLoadBalancers",
            "elasticloadbalancing:DescribeLoadBalancerAttributes",
            "elasticloadbalancing:DescribeRules",
            "elasticloadbalancing:DescribeTags",
            "elasticloadbalancing:DescribeTargetGroups",
            "elasticloadbalancing:DescribeTargetGroupAttributes",
            "elasticloadbalancing:DescribeTargetHealth",
            "elasticloadbalancing:ModifyListener",
            "elasticloadbalancing:ModifyLoadBalancerAttributes",
            "elasticloadbalancing:ModifyRule",
            "elasticloadbalancing:ModifyTargetGroup",
            "elasticloadbalancing:ModifyTargetGroupAttributes",
            "elasticloadbalancing:RegisterTargets",
            "elasticloadbalancing:RemoveTags",
            "elasticloadbalancing:SetIpAddressType",
            "elasticloadbalancing:SetSecurityGroups",
            "elasticloadbalancing:SetSubnets",
            "elasticloadbalancing:SetWebACL"
          ],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": ["iam:GetServerCertificate", "iam:ListServerCertificates"],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": ["waf-regional:GetWebACLForResource"],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": ["tag:GetResources"],
          "Resource": "*"
        },
        {
          "Effect": "Allow",
          "Action": [
            "waf:GetWebACL",
            "waf:AssociateWebACL",
            "waf:DisassociateWebACL"
          ],
          "Resource": "*"
        }
      ]
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://REDACTED
  dnsZone: k8s-stg.example.net
  etcdClusters:
  - enableEtcdTLS: true
    etcdMembers:
    - instanceGroup: master-ap-northeast-1a
      name: a
    - instanceGroup: master-ap-northeast-1c
      name: c
    - instanceGroup: master-ap-northeast-1d
      name: d
    name: main
    version: 3.2.18
  - enableEtcdTLS: true
    etcdMembers:
    - instanceGroup: master-ap-northeast-1a
      name: a
    - instanceGroup: master-ap-northeast-1c
      name: c
    - instanceGroup: master-ap-northeast-1d
      name: d
    name: events
    version: 3.2.18
  hooks:
  - before:
    - kubelet.service
    manifest: |
      [Unit]
      Description=Download AWS Authenticator configs from S3
      [Service]
      Type=oneshot
      ExecStart=/bin/mkdir -p /srv/kubernetes/aws-iam-authenticator
      ExecStart=/usr/local/bin/aws s3 cp --recursive s3://REDACTED/addons/authenticator /srv/kubernetes/aws-iam-authenticator/
    name: kops-hook-authenticator-config.service
    roles:
    - Master
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    authenticationTokenWebhookConfigFile: /srv/kubernetes/aws-iam-authenticator/kubeconfig.yaml
    authorizationMode: RBAC
    authorizationRbacSuperUser: admin
    oidcClientID: REDACTED
    oidcGroupsClaim: groups
    oidcIssuerURL: https://accounts.google.com
    oidcUsernameClaim: email
    runtimeConfig:
      admissionregistration.k8s.io/v1alpha1: "true"
  kubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"
  masterKubelet:
    kubeletCgroups: "/systemd/system.slice"
    runtimeCgroups: "/systemd/system.slice"
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.10.5
  masterInternalName: api.internal.k8s-stg.example.net
  masterPublicName: api.k8s-stg.example.net
  networkCIDR: 10.10.0.0/16
  networkID: vpc-REDACTED
  networking:
    amazonvpc: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 10.15.32.0/19
    name: ap-northeast-1a
    type: Private
    zone: ap-northeast-1a
  - cidr: 10.15.64.0/19
    name: ap-northeast-1c
    type: Private
    zone: ap-northeast-1c
  - cidr: 10.15.96.0/19
    name: ap-northeast-1d
    type: Private
    zone: ap-northeast-1d
  - cidr: 10.15.0.0/22
    name: utility-ap-northeast-1a
    type: Utility
    zone: ap-northeast-1a
  - cidr: 10.15.4.0/22
    name: utility-ap-northeast-1c
    type: Utility
    zone: ap-northeast-1c
  - cidr: 10.15.8.0/22
    name: utility-ap-northeast-1d
    type: Utility
    zone: ap-northeast-1d
  topology:
    bastion:
      bastionPublicName: bastion.k8s-stg.example.net
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-07-18T10:27:53Z
  labels:
    kops.k8s.io/cluster: k8s-stg.example.net
  name: bastions
spec:
  image: kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11
  machineType: t2.micro
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: bastions
  role: Bastion
  subnets:
  - utility-ap-northeast-1a
  - utility-ap-northeast-1c
  - utility-ap-northeast-1d

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-07-18T10:27:53Z
  labels:
    kops.k8s.io/cluster: k8s-stg.example.net
  name: master-ap-northeast-1a
spec:
  image: kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11
  machineType: m4.large
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-ap-northeast-1a
  role: Master
  subnets:
  - ap-northeast-1a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-07-18T10:27:53Z
  labels:
    kops.k8s.io/cluster: k8s-stg.example.net
  name: master-ap-northeast-1c
spec:
  image: kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11
  machineType: m4.large
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-ap-northeast-1c
  role: Master
  subnets:
  - ap-northeast-1c

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-07-18T10:27:53Z
  labels:
    kops.k8s.io/cluster: k8s-stg.example.net
  name: master-ap-northeast-1d
spec:
  image: kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11
  machineType: m4.large
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-ap-northeast-1d
  role: Master
  subnets:
  - ap-northeast-1d

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-07-18T10:27:53Z
  labels:
    kops.k8s.io/cluster: k8s-stg.example.net
  name: nodes
spec:
  image: kope.io/k8s-1.9-debian-jessie-amd64-hvm-ebs-2018-03-11
  machineType: m4.large
  maxSize: 2
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - ap-northeast-1a
  - ap-northeast-1c
  - ap-northeast-1d

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

I0727 14:39:39.705510   53200 factory.go:68] state store s3://REDACTED
I0727 14:39:39.804741   53200 aws_cloud.go:981] Querying EC2 for all valid zones in region "ap-northeast-1"
I0727 14:39:39.805294   53200 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I0727 14:39:40.111042   53200 status.go:57] Querying AWS for etcd volumes
I0727 14:39:40.111080   53200 status.go:68] Listing EC2 Volumes
I0727 14:39:40.111628   53200 request_logger.go:45] AWS request: ec2/DescribeVolumes
I0727 14:39:40.242891   53200 status.go:40] Cluster status (from cloud): {}
I0727 14:39:41.082688   53200 s3context.go:198] Checking default bucket encryption "REDACTED"
I0727 14:39:41.082725   53200 s3context.go:203] Calling S3 GetBucketEncryption Bucket="REDACTED"
I0727 14:39:41.219325   53200 s3context.go:182] Found bucket "REDACTED" in region "ap-northeast-1" with default encryption set to true
I0727 14:39:41.219364   53200 s3fs.go:216] Reading file "s3://REDACTED/config"
I0727 14:39:41.252151   53200 s3fs.go:216] Reading file "s3://REDACTED/config"
F0727 14:39:41.274728   53200 helpers.go:119] error: error replacing cluster: Spec.Networking: Invalid value: "amazon-vpc-routed-eni": amazon-vpc-routed-eni networking is supported only in AWS

9. Anything else do we need to know?
@mikesplain
Copy link
Contributor

Ahh yes, I was able to replicate this. Fix incoming.

@mikesplain
Copy link
Contributor

/assign

@mikesplain mikesplain mentioned this issue Jul 27, 2018
Merged
@sergiorua
Copy link

Which kops version fixes this issue? I'm still seeing it on 1.10.0

@adammw
Copy link
Contributor

adammw commented Oct 4, 2018

The fix is in #5540, which is on the master branch. Doesn't seem to have made it to a release yet.

@MiLk
Copy link
Author

MiLk commented Oct 5, 2018

Yes it's in master. I've confirmed that the fix is working.
You need to either build kops & nodeup yourself or wait for the next release.

@adammw adammw mentioned this issue Oct 5, 2018
Merged
@sbinev sbinev mentioned this issue Jan 17, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mikesplain mikesplain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@MiLk @adammw @mikesplain @sergiorua