Workflow hangs after clicking stop and terminate

[JPZ13]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

When clicking stop or terminate within the UI, we are noticing that workflows are left hanging in a running state. The only way to get them to actually stop is to delete the workflow. This causes a problem for users who want to stop the workflow but keep the record in their archives

Here's a list of some other issues that I think might be related. I will keep this list up to date as more arise:

• Pod hangs when container in ContainerSet is OOM Killed #10063
• OOM error not caught by the emissary executor forcing the workflow to hang in "Running" state #8680
• Pod failed with error: Pod was active on the node longer than the specified deadline -remain in status Running  #9934

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Unfortunately, this happens intermittently. I will update this issue with some videos and see if I can find some workflows that consistently trigger it using only public images

Logs from the workflow controller

see above

Logs from in your workflow's wait container

See above

[JPZ13]

Grain.Highlight.7vm9f15b0ukV93KlO59k6kH8m8eTMiPiwxysYezG.mp4

The above video demonstrates what we're seeing along with the controller logs. It doesn't look like there's any logging output unfortunately

[JPZ13]

On a call with @juliev0 - we need the Argo Server logs to get more clarity into what's happening @tico24 or any others, if you see this happen again, can you post the Argo Server logs as well?

Also, if we're seeing any logs being sent from the main or wait container that can inform us whether the SIGINT or SIGTERM signals are going out, that would be super helpful as well

[tico24]

Naturally, every workflow I've tried to stop today just stops gracefully. I'll add logs as soon as I can recreate.