Eventually shut down scheduler when no workers return

[hendrikmakait]

Currently, a scheduler is not considered idle if it has tasks waiting to be processed but no workers that could process them: https://github.com/hendrikmakait/distributed/blob/4b10aa7068ff8a67cc756ef0c91f5553429963f3/distributed/scheduler.py#L7966-L7972

This makes sense in adaptive scaling scenarios or when using ephemeral workers to give new workers time to spin up. However, we should eventually shut down the scheduler if no workers return instead of leaving it running indefinitely.

We could do this by adjusting the definition of the idle timeout to include time spent without workers or adding a dedicated timeout mechanism.

[FTang21]

Helllo @hendrikmakait,

I'm Franke Tang, a graduate student currently taking a Distributed Computing course, and part of my final project encourages us to contribute to open issues on GitHub relating to distributed systems. I would like to work on this issue if this feature has not been implemented yet.

[fjetter]

Welcome @FTang21 ! Contributions are very welcome.

You'll find the logic for this here

distributed/distributed/scheduler.py

Line 8076 in e98dcb1

def check_idle(self) -> float | None:

Tracking the time w/out workers is likely not trivial but it shouldn't be too difficult. If you have questions or need help, I encourage you to open a "Draft" PR and we can have a look at how far you got

[FTang21]

Hello,

Sorry with the late response, I was busy with other PRs. I created PR, could you guys look and see if my logic is correct or on the right track.