Delete non-terminal jobs and subworkflow invocations when cancelling invocation

[mvdbeek]

Came up as a wish during the last UI/UX call with @nomadscientist. I contemplated making the job deletion part optional, but couldn't think of a reason why we wouldn't make this the only default. No problem to make it configurable though if there is a reason to keep the old default.

Not sure this is completely free of race conditions, but worst case you gotta delete again ? That seems fine.

I think it'd also be nice to have various flavors of bulk deletion (delete all datasets produced by an invocation, delete all non-output datasets, maybe other modalities ?).

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[mvdbeek]

| 5 - Concatenate datasets (with sleep) on data 2 (HID - NAME) 
INFO:     127.0.0.1:54788 - "GET /api/histories/b492c5865e953d13/contents/f4569757dd099abb HTTP/1.1" 200 OK
| Dataset State:
|  running
| Dataset Blurb:
|  deleted
| Dataset Info:
|  *Dataset info is empty.*
| Peek:
|  <table cellspacing="0" cellpadding="3"><tr><td>Job deleted</td></tr></table>
INFO:     127.0.0.1:54798 - "GET /api/histories/b492c5865e953d13/contents/f4569757dd099abb/provenance HTTP/1.1" 200 OK
| Dataset Job Standard Output:
|  *Standard output was empty.*
| Dataset Job Standard Error:
|  *Standard error was empty.*

hmm, I guess that's the race condition I anticipated.

[mvdbeek]

This is ready for review now, the conda unit test is unrelated.

[nsoranzo]

Failed API tests seem relevant.

[mvdbeek]

yeah, and the crazy thing is reverting 3e4380f fixes this

[mvdbeek]

ouch workflow_invocation.state = workflow_invocation.mark_cancelled() ... well that sets the state to None 😆

[mvdbeek]

This probably also fixes #1450 ... if it is still an issue
A similar approach where we prevent changing the state from cancelling, cancelled, failed to new, ready, scheduled would probably also help with #1450

[mvdbeek]

Failing tests look unrelated, but I triggered a rerun anyway.