Docs: How to purge old workers (status: Not responding)

[RaiaN]

Documentation Issue

Hi there,

We are trying to figure out how to properly purge old workers?

Due to active development we have many worker with status: Not responding. How to delete them via deadline tool?

Best wishes,
Petr

[leongdl]

Hi,

Thanks for your question, please try the delete-worker API from the AWS CLI?
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/deadline/delete-worker.html

For example,
aws deadline delete-worker --farm-id <farm-12345> --fleet-id <fleet-98765> --worker-id <worker-abcdef>

Question, why did the workers get stuck at at Not responding ? Is this a Customer Managed Fleet, or Service Managed Fleet?

[mwiebe]

We also have a sample fleet health check that you can deploy via CloudFormation, that will monitor the health of workers to clean them up and notify you if there are sustained issues.

https://github.com/aws-deadline/deadline-cloud-samples/tree/mainline/cloudformation/farm_templates/cmf_templates

[RaiaN]

Thank you @leongdl @mwiebe
We are using CMF (Windows nodes). I didn't know we can use aws CLI, I thought we should use deadline (this repo basically) to deal with deadline workers so I created a PR: #598

Question, why did the workers get stuck at at Not responding

We've been using Windows Deadline worker service and sometimes we hit this bug. In other cases Deadline service failed to start (i.e. there was a timezone bug on CMF fleet like time wasn't properly synchronized with global time so Worker couldn't start, see error message below).

Error:

API.Resp 📥 [deadline:AssumeFleetRoleForWorker](403) error={'Message': 'Signature not yet current: 20250207T232743Z is still later than 20250207T213245Z (20250207T212745Z + 5 min.)', 'Code': 'InvalidSignatureException'} params={} request_id=93c366be-d351-4f4e-8bb5-54299b43a44c

Any comments?

[RaiaN]

https://github.com/aws-deadline/deadline-cloud-samples/tree/mainline/cloudformation/farm_templates/cmf_templates

This is really good. We can use that I think. Is it compatible with CMF?

[mwiebe]

https://github.com/aws-deadline/deadline-cloud-samples/tree/mainline/cloudformation/farm_templates/cmf_templates

This is really good. We can use that I think. Is it compatible with CMF?

Yes, this template was created for a CMF with an EC2 Auto Scaling Group and event-based auto scaling enabled.