fix: resolve indefinitely queued (STOPPING_COMPLETED) trials

[carolinaecalderon]

Ticket

RM-368

Description

When a cluster restarts, it restarts running trials. For large experiments with several trials (like a hyperparameter experiment), some of these restored trails end up in STOPPING_COMPLETED indefinitely versus COMPLETED state. Fix this bug.

I was able to reproduce this on AWS by killing the agent service, stopping the master service, and then restarting them both. WIth my fix, the experiments naturally resolved themselves.

Test Plan

See new e2e test. No additional testing needed.

Checklist

• Changes have been manually QA'd
• New features have been approved by the corresponding PM
• User-facing API changes have the "User-facing API Change" label
• Release notes have been added as a separate file under docs/release-notes/
  See Release Note for details.
• Licenses have been included for new code which was copied and/or modified from any external code

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	6d109be
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/66996394bd2d4f0008490e6f

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.

Project coverage is 53.42%. Comparing base (e4a9ae3) to head (6d109be).
Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9605      +/-   ##
==========================================
- Coverage   53.44%   53.42%   -0.02%     
==========================================
  Files        1254     1254              
  Lines      152636   152633       -3     
  Branches     3268     3267       -1     
==========================================
- Hits        81572    81548      -24     
- Misses      70913    70934      +21     
  Partials      151      151              

Flag	Coverage Δ
backend	44.69% <83.33%> (-0.05%)	⬇️
harness	72.84% <ø> (ø)
web	51.81% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
master/internal/trial.go	42.10% <83.33%> (+0.24%)	⬆️

... and 5 files with indirect coverage changes

[hamidzr]

When a cluster restarts, it restarts running trials

always? why? is this because we're not handling missed metrics, and progress reports?

[carolinaecalderon]

When a cluster restarts, it restarts running trials

always? why? is this because we're not handling missed metrics, and progress reports?

Not sure what you mean -- In the case of a long-running hp experiment, when a cluster goes down/restarts in the middle of that progress, uncompleted trials are re-allocated/re-started. I'm not sure what you mean by missed metrics/progress reports. I see that the trials restart through the master service logs & also on the webUI.

[hamidzr]

does the order matter here that which one dies first?

[carolinaecalderon]

Yes, if you try to kill the agent after the master, it gives you an error (can't remember what it is off the top of my head though)