daemon container won't be delete if deadline exceeded happen

[henrywangx]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

We use the daemon container of argo workflow, but the workflow happened hang forever sometimes.
The processes workflow:

1. running computing steps;
2. delete daemon container when all computing steps finished.

logs:

 level=error msg="error in entry template execution" error="Deadline exceeded"

And I checked source code as bellow, whether ErrDeadlineExceeded the workflow won't be reconciled if workflow only one daemon container in the end unless restart controller?
https://github.com/argoproj/argo-workflows/blob/master/workflow/controller/operator.go#L346-L353

Version

v3.2.6

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Hard to reproduce.

Logs from the workflow controller

level=error msg="error in entry template execution" error="Deadline exceeded"

Logs from in your workflow's wait container

no useful log

[alexec]

I think we’d need a workflow to reproduce this if we want to investigate it.

[HRusby]

To Note, I believe this is the same issue as in #9934 which does have reproducible workflows attached

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity and needs more information. It will be closed if no further activity occurs.

[github-actions]

This issue has been closed due to inactivity and lack of information. If you still encounter this issue, please add the requested information and re-open.

[agilgur5]

To Note, I believe this is the same issue as in #9934 which does have reproducible workflows attached

That one doesn't use a daemon container, so would not apply to this issue.

Otherwise that issue is now resolved.

If you're still encountering this, please provide a reproducible Workflow with a daemon on the :latest version