Add support for defining/using AWS ASG Lifecycle Hooks

[andersosthus]

We would like support for defining AWS ASG Lifecycle hooks on an InstanceGroup.

Use case:
We use AWS ASG Lifecycle hooks with a custom script that watches for AWS Spot terminations, and if detected runs a drain on the node and then sends a COMPLETED signal to the Lifecycle hook, that then allows the instance to be terminated.
The fact that we have this Lifecycle hooks kinda "breaks" kops rolling-update cluster, since the instance won't be terminated until either a COMPLETED signal is sent or it reaches the timeout value.

Our proposed solution would look something like this:
Add an awsAsgLifecycle property to InstanceGroup where one can set the Lifecycle properties (name, transition, default result, heartbeat timeout, notification arn, role arn).

When kops is doing a node drain, if the InstanceGroup has the awsAsgLifecycle set, it should send a COMPLETED signal when the drain is done.

If this sounds ok, I can do the implementation (though I probably need some guidance since I'm not that familiar with the kops codebase)

[paalkr]

Yes, please!

[mikesplain]

Out of curiosity, is there a reason you chose to go this way and not use something like https://github.com/pusher/k8s-spot-termination-handler?

[paalkr]

I'm actually working with @andersosthus on this one, so I will make an answer. Yes, there is a good reason why we don't use the component you linked to. It does only support spot termination and we need a solution that also works on AGS termination. We do NOT use the common Cluster Autoscaler because we have memory bound and spiky workloads. So we export utilization metrics from Prometheus to CloudWatch, and the we use a combination of TargetTracking and StepScaling to modify the Autoscaling Groups desired count.

Also the proposed solution from @andersosthus will allow to attach custom ARNs, like a Lambda function to do arbitrary activities when machines are added or removed from an Autoscaling group.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[olemarkus]

This sounds good to me

[rifelpet]

/remove-lifecycle stale
/kind feature

[rdrgmnzs]

@andersosthus are you still interested in implementing this? If not I'm going to take a shot at it.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[olemarkus]

/remove-lifecycle stale

Anyone had any luck with this issue?

This is to some extent related to #7119

[paalkr]

Currently we handled this using a custom systemd unit in the instance group. But the use case will not longer apply when we get this NTH implementation in place.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[bigman3]

Currently we handled this using a custom systemd unit in the instance group. But the use case will not longer apply when we get this NTH implementation in place.

@paalkr can you share/guide how the systemd unit should like? I am trying to replicate, with increased timeouts (TimeoutStopSec, TimeoutSec) but hitting 2 minutes when it stops not gracefully

[bigman3]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[olemarkus]

With the support for NTH in SQS mode, are the use cases covered?
/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[olemarkus]

Closing it is
/close

[k8s-ci-robot]

@olemarkus: Closing this issue.

In response to this:

Closing it is
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.