RFC#0192 - Ensure workers do not get unnecessarily killed #192
Conversation
JohanLorenzo
requested review from
lotas,
petemoore and
matt-boris
and removed request for
a team
JohanLorenzo
changed the title
|
|
||
| ### What if we deploy new worker images? | ||
|
|
||
| Long-lived workers will have to be killed if there's a change in their config, including |
There was a problem hiding this comment.
This goes along with the #191 proposal quite well.
With introduction of launch configurations, workers would be able to query w-m periodically to see if config is still active, and restart/shutdown if not.
We could probably create a more generic endpoint which would answer if worker should terminate:
yesif config changednoifminCapacityis not observed.
However, I'm not totally sure how to solve this issue if all workers at once ask if they could shutdown, and all get yes.. just to have zero running workers after
There was a problem hiding this comment.
This goes along with the #191 proposal quite well.
Nice! 👍
However, I'm not totally sure how to solve this issue if all workers at once ask if they could shutdown, and all get yes.. just to have zero running workers after
I guess that's an okay behavior, at least for a first iteration. Maybe worker-manager could remember how many workers are being shut down so it doesn't tell more workers to turn off.
|
|
||
| ## When `minCapacity` is not yet met | ||
|
|
||
| Here, `worker-manager` should increase `afterIdleSeconds` to a much higher value (e.g.: |
There was a problem hiding this comment.
"tweaking" idle seconds could be a good solution for existing workers that are unlikely to get upgraded soon.
There should be some upper limit of what worker-manager could set for such workers to balance minCapacity promise and avoiding additional costs from running them for too long
There was a problem hiding this comment.
To me, if we set minCapacity then we expect to have a number of workers at any time of the day. I would expect these workers to keep running. If some worker-types shouldn't be running for too long, I wonder if we should set minCapacity to 0.
There was a problem hiding this comment.
that's true! Indeed, if someone sets min value he probably knows what he's doing, no need to restrict
There was a problem hiding this comment.
Sorry term I think this isn't a problem.
Long term something probabilistic could work: p(no) = 0.5 if exactly at minCapacity, p lower if higher than minCapacity
There was a problem hiding this comment.
I wonder if we need a probabilistic model at this stage. To me, there are already several random factors impacting workers (e.g.: spot instance termination, network issues). So having a deterministic model here may help diagnosing what goes wrong with the implementation. I may be missing something in my understanding of the problem, though.
|
@lotas 👋 Do you see anything more we should discuss? If not, is okay to open up the discussion in the next community meeting? |
please bring it up during the next community meeting, would be best! Thanks |
No description provided.