Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kops fails when using c5 instances on aws #3868

Closed
dharmisha opened this issue Nov 16, 2017 · 36 comments
Closed

kops fails when using c5 instances on aws #3868

dharmisha opened this issue Nov 16, 2017 · 36 comments
Labels
blocks-next
Milestone
1.10

Comments

@dharmisha
Copy link

dharmisha commented Nov 16, 2017 •
edited
Loading

I am running kops version 1.7.1 in aws, i used the amazon/amzn-ami-hvm-2017.09.1.20171103-x86_64-gp2 image, which has ena support. However, when I specify the --node-size c5.large, it fails.

c5 instances are launched recently, however is there a way kops can launch c5 instances?
@dharmisha dharmisha changed the title kops does not provide support for c5.large kops fails when using c5 instances on aws Nov 16, 2017
@pieterza
Copy link

+1; perhaps kops should not care what size we specify, and let aws tell us if we were wrong? Else it's always catchup

@chrislovecnm
Copy link
Contributor

It is fixed in master and the current beta. Also the Debian base image that kops uses out of the box does not support ena driver. C5 requires that.

So use the beta and you need a different ami

@justinsb
Copy link
Member

If we can't get this supported in the default image, this at least has to be a warning.

@adrianchifor
Copy link

adrianchifor commented Nov 16, 2017 •
edited
Loading

I agree, please put a warning up, was banging my head against the table all day of why pods were failing randomly on c5.4xlarge nodes.

@coderanger
Copy link
Member

Copying the specific error here to aid others in googling:

Launching a new EC2 instance. Status Reason: Enhanced networking with the Elastic Network Adapter (ENA) is required for the 'c5.large' instance type. Ensure that you are using an AMI that is enabled for ENA. Launching EC2 instance failed.

@chrislovecnm
Copy link
Contributor

Here is the issue for ENA driver with the base image kubernetes-retired/kube-deploy#292

@dharmisha
Copy link
Author

It is fixed in master and the current beta. Also the Debian base image that kops uses out of the box does not support ena driver. C5 requires that.So use the beta and you need a different ami

I used the beta Version 1.8.0-beta.1 (git-9b71713) binary with amazon/amzn-ami-hvm-2017.09.1.20171103-x86_64-gp2 ami which has support for ena. It still gives an error saying

nodes.spec.machineType: Invalid value: "c5.2xlarge": machine type specified is invalid

Please let me know what needs to be done. This solution is urgently required

@chrislovecnm
Copy link
Contributor

kubernetes-retired/kube-deploy#292

Thanks needs to be tested and completed

@bcorijn
Copy link
Contributor

bcorijn commented Nov 20, 2017

Having ENA support by default would be great for all instances, not just C5. I've been running the modified image from my PR for several months now without any issues and was hoping it could get merged before the 1.8 stable release.

@chrislovecnm
Copy link
Contributor

@bcorjin the image and the kops releases are not always tied together

This was referenced Nov 21, 2017
Closed
Closed
@chrislovecnm
Copy link
Contributor

#1558 is related, and kubernetes-retired/kube-deploy#292 needs to be fixed in the base image

@justinsb justinsb added this to the 1.8.0 milestone Nov 25, 2017
@justinsb
Copy link
Member

There are three aspects:

  1. recognize c5 instances (done)
  2. build a new image supporting ENA (I've merged the supporting fixes to the image and am validating them, so more or less done)
  3. support NVME volume mounting in protokube, if we want to run a master on c5. not done Kubernetes won't support nvme for 1.8, but we're debating whether we can get it into 1.9. Likely we'll punt this aspect out of the 1.8 milestone, but we'll see...

@chrislovecnm
Copy link
Contributor

In regards to number 3. This means that k8s will not support mount pv as well, I assume.

@justinsb
Copy link
Member

Exactly - so I have two PRs (with much the same code), one that adds nvme mounting to protokube, one that adds it to k8s.

@dharmisha
Copy link
Author

@justinsb @chrislovecnm
Hi, No new binary has been released which support c5 instances. Is there a specific image that I need to use other than amazon/amzn-ami-hvm-2017.09.1.20171103-x86_64-gp2 to have kops recognize c5? If not what needs to be done for having it resolved.

@thuandt
Copy link

thuandt commented Dec 5, 2017

kops 1.8.1 has support c5 instances but EC2 instance will failed. We need AMI update also

@dharmisha
Copy link
Author

dharmisha commented Dec 5, 2017 via email

@onyxraven
Copy link

@justinsb mind linking those PRs for easy tracking from this ticket?

@chrislovecnm
Copy link
Contributor

This is fixed in 1.8. Please see release notes, let us know if you need more details!!

@chrislovecnm
Copy link
Contributor

kops 1.8 which is backwards compatible with k8s

@thuandt
Copy link

thuandt commented Dec 14, 2017

FYI, if you want use c5 please ensure you didn't use EBS volume. Because kubernetes 1.8 can't mount it as devices path.

Waiting for 1.9 you want to use c5 with EBS

@chrislovecnm
Copy link
Contributor

To be clear pvc volumes are not supported in k8s 1.8 - will be supported in 1.9

@erickt erickt mentioned this issue Jan 9, 2018
Closed
@justinsb justinsb modified the milestones: 1.8.0, 1.9 Feb 21, 2018
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 22, 2018
@justinsb justinsb modified the milestones: 1.9.0, 1.10 May 26, 2018
@diegocn
Copy link
Contributor

diegocn commented Jun 7, 2018

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 7, 2018
@diegocn
Copy link
Contributor

diegocn commented Jun 7, 2018

Is c5 instances supposed to work on 1.9.x now? I've started a cluster with c5.large nodes and the root volumes just got 8GB on / partition (but the volume starts with 128GB) and I got errors with PODs trying to attach PVCs.

@thuandt
Copy link

thuandt commented Jun 7, 2018

@diegocn PVCs is only works with stretch image not jessie image.

current stable still use jessie as i see

@diegocn
Copy link
Contributor

diegocn commented Jun 7, 2018

Hello @thuandt, thank you.
I'll try to use supported AWS Instances, such as t2.large instead of c5.large for now.

@chenrui333
Copy link
Member

+1

@diegocn
Copy link
Contributor

diegocn commented Jun 20, 2018

Running a few POCs with other workloads (not K8S), c5 instances are far better than t2. It would be really nice to have support for these instance type.

@rtrive
Copy link

rtrive commented Jun 25, 2018

Today i try to launch a cluster with c5 instances but the cluster doesn't work...

@faheem-nadeem
Copy link

C5 instances have been working for us, using the debian stretch AMI for about two months. Haven't had any problems relating to mounting volumes or any instability.

@thuandt
Copy link

thuandt commented Jun 26, 2018

@faheem-cliqz problem is upgrade. when do kops upgrade it will replaced stretch with jessie :)

@chenrui333
Copy link
Member

@thuandt do you have any code reference for that?

@faheem-nadeem
Copy link

@thuandt yup the upgrade part is a problem. We still get the upgrade messages. Just ignoring them for now, and try to upgrade as much as we can with cluster and instance group specs :)

@bcorijn
Copy link
Contributor

bcorijn commented Jun 27, 2018

@chenrui333 upgrade pulls it's information from the channels, which have jessie images. https://github.com/kubernetes/kops/blob/master/channels/stable

@thomaspeitz thomaspeitz mentioned this issue Aug 17, 2018
Closed
@geojaz
Copy link
Member

geojaz commented Aug 18, 2018

closed in #5660

@geojaz geojaz closed this as completed Aug 18, 2018
@idealhack idealhack mentioned this issue Sep 6, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
blocks-next
Projects
None yet
Milestone
1.10
Development

No branches or pull requests