Launching t3.medium EC2 instance failed

[idealhack]

1. What kops version are you running? The command kops version, will display
this information.

Version 1.10.0 (git-782ff1358)

This is branch release-1.10 with #5681 cherry-picked

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.15", GitCommit:"c2bd642c70b3629223ea3b7db566a267a1e2d0df", GitTreeState:"clean", BuildDate:"2018-07-11T17:59:56Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.11", GitCommit:"b13f2fd682d56eab7a6a2b5a1cab1a3d2c8bdd55", GitTreeState:"clean", BuildDate:"2017-11-25T17:51:39Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

1. update ig to use t3.medium
2. kops update cluster --yes
3. kops rolling-update --yes

5. What happened after the commands executed?

Node drained, instance terminated but not created, Auto scaling group reported:

Enhanced networking with the Elastic Network Adapter (ENA) is required for the 't3.medium' instance type. Ensure that you are using an AMI that is enabled for ENA. Launching EC2 instance failed.

I'm using the kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-03-11 AMI.

Related: #3868

6. What did you expect to happen?

a) kops rolling-update works as normal

b) kops knew t3.medium is not going to work, so refuse to update or rolling-update cluster at first.

[idealhack]

btw, r5 also fails because of the same reason.

[idealhack]

While m5 failed for the same reason with k8s-1.7-debian-jessie-amd64-hvm-ebs-2018-03-11 (ami-226a2244), it seems works with k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-03-11 (ami-b6571fd0)

[zkanda]

@idealhack thanks for testing this out, 👍 we are eager to move to t3 instance, 💃

Does it work with stretch?

[idealhack]

@zkanda I don't think so. You should try it on testing environment only.

[Globegitter]

I guess the last question is if it works with latest stable CoreOS? If I have time to test on our cluster I will do so and report back - if in the meantime someone else has tried I would be curios to hear if it works.

[geojaz]

I believe t3 support on the kops side ended up landed just after 1.10 and will be includedin 1.11alpha1 which arrives soon. m5 support works with 1.10 and stretch (but not jessie).

[tatobi]

A silly question (probably):
Why kops checks AWS machine type at all? Shouldn't be running AWS class agnostic? I headed the same problem now with 1.10 :

Invalid value: "t3.small": machine type specified is invalid

[macropin]

Related request #5714 for t3 instances

[b0ric]

Also do not understand why kops care about instance types. Tried to bootstrap a cluster with t3.* instances and got the same error as above.

[zkanda]

I think it's more about ami used, kops use a specific one and not all support the new ENA from aws.

[etehtsea]

@idealhack thanks for testing this out, 👍 we are eager to move to t3 instance, 💃

Does it work with stretch?

I successfully upgraded to the t3 family for both master and nodes using this ami kope.io/k8s-1.10-debian-stretch-amd64-hvm-ebs-2018-08-17.

[chrkaatz]

when will be 1.11 out of testing phase?

[cjbottaro]

This is why we use amazon/amzn2-ami-hvm-2.0.20181024-x86_64-gp2 for all our instance groups (including masters).

We are about to build our own version of kops 1.10.0 with that exact cherry picked commit and upgrade some of our nodes to the t3 class. Will let you know how it goes (I mean, if I remember)!

[cjbottaro]

when will be 1.11 out of testing phase?

We tried using the 1.11 beta and after doing a kops update cluster for something relatively simple, the amount of diffs it spit out was too much for us to be comfortable proceeding with.

[cjbottaro]

Upgrading to the t3 class worked perfectly using kops 1.10.0 with cherry picked #5681 and amazon/amzn2-ami-hvm-2.0.20181024-x86_64-gp2 for the image. 👍

[emwalker]

We just attempted to use the t3.xlarge instance class for nodes. Kops 1.11.0 recognized the new instance type and allowed the rolling update to proceed, but when the nodes came up, the networking did not seem to be working. Presumably this was because an AMI different from amazon/amzn2-ami-hvm-2.0.20181024-x86_64-gp2 was used, although I'm not sure.

Is the expectation that kops will prevent users from using an instance type that is not yet supported?

[idealhack]

Since t3 and other new types were supported since kops 1.11.0, I'm going to close this. Other issues can be filed as new issues. Thanks for all the comments and feedback here.