Interrupting Workflow Does Not Terminate AWS Batch Job(s)

[zcking]

Steps to Reproduce

1. Clone the tutorials:

metaflow tutorials pull
cd metaflow-tutorials

2. Configure AWS

metaflow configure aws

3. Start Tutorial Episode 5: "Hello AWS"

python 05-helloaws/helloaws.py run

4. As soon as you see the output "Task is starting (status STARTING)..." perform a keyboard interrupt (Ctrl+C) to stop the workflow. Note: because this hello AWS example runs so quickly, it may be easier if you add a time.sleep(10) and interrupt it during that delay

5. View the AWS Batch Job console and notice the Job is not terminated

[savingoyal]

Thanks for the bug report. Looks like a race condition. We do a best effort kill of Batch jobs. We will investigate.

[romain-intel]

@zcking: I played around with this issue and found the following:

• Batch does not immediately terminate; it may take 30s (or sometimes slightly more) for a termination API call to batch to be effective (in other words, MF does send termination commands to Batch jobs when you CTRL-C it but those termination commands do not have an immediate effect -- this is not something we have control over). Note that for very short jobs, this may manifest as the job being 'SUCCESSFUL' as opposed to 'FAILED' because the termination signal was handled after the job succeeded. If you add a time.sleep(60) for example, you are much more likely to see the job state in the AWS Batch Job console transition to 'FAILED' (after the aforementioned 30 seconds or so delay).
• In some cases, the termination command is not sent from MF to Batch. This is a bug on our end and we will work on improving this. As @savingoyal mentioned, we do a best-effort kill of Batch jobs so we may still not be able to catch call cases but we will see if we can make it somewhat better. This is a fairly rare case so I suspect that what you are observing is more the delayed Batch termination issue listed above. You can change the sleep to 60 seconds for example to confirm -- if the jobs eventually transition to 'FAILED', it is the delayed batch issue, if not, it is this race (or some other issue that we haven't replicated yet); let me know what you observe and I can dig further if it is something I have not yet seen.

[zcking]

Thanks for the details @romain-intel I think that's a reasonable behavior, as is. I'm still very new to AWS Batch, and Metaflow even moreso, but I appreciate the willingness to improve for even these small details :)