Investigate VMSS (Virtual Machine Scale Sets)

[justaugustus]

/kind feature
/assign
/milestone next

[justaugustus]

/priority important-longterm

[alexeldeib]

I'm also interested in this, not sure how high priority this should be considered. VMSS makes a lot of sense for implementing a higher level machineset or machinedeployment, but it's not strictly necessary for the Machine spec.

Off the top of my head one immediate question is whether this would be user-toggleable or not. If so, something in the MachineSet** spec needs to communicate that. If not, one thing to keep in mind is the providerID format will suddenly be slightly different.

[alexeldeib]

@justaugustus any thoughts?

[feiskyer]

@justaugustus @alexeldeib any blocking issues on VMSS support? We'd like to prioritize this.

[alexeldeib]

@feiskyer Not as far as I'm aware. This seems like an easy win to me.

[justaugustus]

It's a win for everyone, but I wouldn't necessarily call it an easy one.
What we need to consider here is the fact that there is no native paradigm for VMSS in Cluster API.

As the VMSS and virtual machine implementations for Azure are different, I would request that this be something toggleable for users.

Juan-Lee expressed interest in working on this, so assigning to him.
/assign @juan-lee

Also tagging @detiber, as I believe we were talking about what the right way to make this happen is, during KubeCon.

[justaugustus]

/unassign

[detiber]

There are two potential approaches:

• Add an (optional) extension point for MachineSets that could be leveraged to directly interact with VMSS (or other cloud-based scaling mechanisms).
• Add a new type similar to MachineSet with an extension point for interacting with VMSS (or other cloud-based scaling mechanisms)

The second option would probably be the least controversial in the community, since there is some concern that overloading MachineSets could potentially lead to user confusion.

The proper place to start for this would be to start putting together a design proposal using this template and bringing it up for discussion at the weekly Cluster API meeting.

[alexeldeib]

@detiber if you're planning to bring this up in cluster-api weekly, would you mind pushing it to the meeting on the 19th rather than the 12th? I was hoping to attend the kubebuilder monthly meeting which conflicts with this. I've been traveling and mostly have tried to keep up async with meetings, docs, and slack, but I'd like to attend in person if the group will discuss ASG/VMSS/MIG proposals.

If that's too much to ask, I'd be happy to collaborate on a proposal in advance of a meeting where I won't be present. Or if you end up hashing something out with @juan-lee, I'd love it if that were a public doc from Day 1 (at the very least for viewing, I can understand it may be difficult to crowdsource everything from scratch with too many cooks in the kitchen).

I agree it should be an extension/toggle at MachineSet level (which is partially why I said this is an 'easy win' -- I'm struggling to imagine something much different which fits into the model of cluster API; details can be hashed out).

For the use cases I'm interested in, using cluster API without this functionality is basically a non-starter (partially due to connection with HA). I'd like to see this move forward.

[justaugustus]

@detiber -- thanks for the feedback!!

@alexeldeib -- just to be clear, this is assigned to @juan-lee, not Jason (I just tagged him in for feedback). Once a public proposal is put together, we can look at getting it on the calendar with the other Cluster API folks.

[detiber]

I won't have time for a while to work on a proposal for this, but I'm happy to review/provide feedback. I also wasn't planning on adding it explicitly as an agenda item to the Cluster API meeting until a proposal exists.

[juan-lee]

@justaugustus @detiber @alexeldeib

Here's my plan:

@CecileRobertMichon and I will work on a design and prototype based on the current state of cluster-api. This exercise should lead us to concrete requirements that will inform a proposal to upstream cluster-api. It's entirely possible that we won't encounter any blockers and our feedback is mostly around usability. Please let me know if there are any objections to this plan.

[justaugustus]

@juan-lee -- sounds good! Please link the Google Doc here once it's in motion.

[puja108]

We are using VMSS already, but from talking to Azure folks it seems not everyone is convinced you should use them, yet. At least it seems that Azure people in this thread would support it, which makes me happy.

In general, we would welcome a cluster API mode that allows for higher level provider abstractions to machines (VMSS/ASG/MIG) as there're some good arguments for using them, e.g. AWS is adding lots of features to ASGs that make manual management of reserved vs spot vs normal instances obsolete (/cc @teemow). The approaches suggested by @detiber should be applicable to any of those.

[juan-lee]

@puja108 can you provide some additional context around Azure folks not being convinced of using VMSS? Are we talking about just for k8s or broader?

[puja108]

@juan-lee I think it was a broader/general hesitancy to use VMSS as they are still relatively new and there wasn't much experience with it. I sadly have no deeper insights there and as I said at least there's also some Azure folks who would support using VMSS, so I'm optimistic there.

[CecileRobertMichon]

Here a first iteration of a proposed design for implementing VMSS given the current state:
https://docs.google.com/document/d/1nbOqCIC0-ezdMXubZIV6EQrzD0QYPrpcdCBB4oSjWeQ/edit?usp=sharing

@detiber @justaugustus PTAL

[CecileRobertMichon]

@detiber @justaugustus I don't see any comments from you on the doc, have you had a chance to review it? Should I share it with a wider audience?

[justaugustus]

@CecileRobertMichon @juan-lee -- Will review this week!
Email sent to SIG CL and Azure for comments: https://groups.google.com/d/topic/kubernetes-sig-cluster-lifecycle/hhTm8u4rSMU/discussion

cc: @ritazh

[ncdc]

I wrote a comment in the proposal document, but I'd really like to see support for scaling sets/groups (Azure VMSS, AWS ASG, etc) in Cluster API itself, and not implemented entirely independently per provider. This could be added to MachineSet, or via a new type similar to MachineSet but specifically for taking advantage of provider-specific scaling functionality.

[vincepri]

+1 to what @ncdc mentioned, having support in CAPI (especially now that we're working on v1alpha2) would be a major win for the project.

If you can join, I'd love if one of you can bring the discussion in today's Cluster API community meeting https://docs.google.com/document/d/1Ys-DOR5UsgbMEeciuG0HOgDQc8kZsaWIWJeKJ1-UfbY/edit

[CecileRobertMichon]

@vincepri sounds great. Unfortunately I'm not able to make it today but @juan-lee and I will both attend next week's meeting. Hopefully that will also give time for others to share their perspective on the doc before we have a discussion.

[vincepri]

Sounds great, thanks!

[justaugustus]

/assign
@juan-lee and I will be working on the MachinePool + AzureMachinePool (VMSS) proposal(s).

[CecileRobertMichon]

Link to the MachinePool proposal: https://docs.google.com/document/d/17oyJsUYWq75by5xIMT3KyEHm0Ln__DNZp9b8AK0jszg/edit

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[awesomenix]

/remove-lifecycle stale

[awesomenix]

/remove-lifecycle rotten

[devigned]

/assign

[devigned]

/active

[CecileRobertMichon]

/close

@devigned can you please open a new issue to track implementing MachinePools?

[k8s-ci-robot]

@CecileRobertMichon: Closing this issue.

In response to this:

/close

@devigned can you please open a new issue to track implementing MachinePools?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.