when main container recreate，the wait container still wait old main container id， and return err

[huangjiasingle]

Summary

What happened/what you expected to happen?
when main container recreate，the wait container still wait old main container id， and return err： failed to wait for main container to complete: timed out waiting for the condition: Error response from daemon: No such container: containerID
Before we start, around 2/3 of issues can be fixed by one of the following:

• Have you double-checked your configuration? Maybe 30% of issues are wrong configuration.
• Have you tested to see if it is fixed in the latest version? Maybe 20% of issues are fixed by this.
• Have you tried using the PNS executer instead of Docker? Maybe 50% of artifact related issues are fixed by this.

Diagnostics

👀 Yes! We need all of your diagnostics, please make sure you add it all, otherwise we'll go around in circles asking you for it:

What Kubernetes provider are you using?
v1.19.7
What version of Argo Workflows are you running?
v3.0.2
What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
Docker
Did this work in a previous version? I.e. is it a regression?
time="2021-06-29T06:31:12.987Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label "io.kubernetes.container.name"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=argo --f
ilter=label=io.kubernetes.pod.name=prod-datafilter-batch-1.2-cxmk1buk-3785657846"
time="2021-06-29T06:31:13.031Z" level=info msg="mapped container name "main" to container ID "ee3f0bbaf70d19fc179a14cad051b30a01840f44bbac9fd21517f1d2266cff5d" (created at 2021-06-29 06:29:28 +0000 UTC, status
Created)"
time="2021-06-29T06:31:13.284Z" level=info msg="docker wait ee3f0bbaf70d19fc179a14cad051b30a01840f44bbac9fd21517f1d2266cff5d"
time="2021-06-29T06:31:13.314Z" level=error msg="docker wait ee3f0bbaf70d19fc179a14cad051b30a01840f44bbac9fd21517f1d2266cff5d failed: Error response from daemon: No such container: ee3f0bbaf70d19fc179a14cad051b30
a01840f44bbac9fd21517f1d2266cff5d\n"
time="2021-06-29T06:31:50.635Z" level=info msg="mapped container name "main" to container ID "8fd538622134a96d9cbf2e79e127b9ceaedfe6956500149c300fa32706ae73a7" (created at 2021-06-29 06:31:49 +0000 UTC, status
Up)"
time="2021-06-29T06:31:50.635Z" level=info msg="ignoring created container "main" that would Up -> Created"
time="2021-06-29T06:31:53.175Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label "io.kubernetes.container.name"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=argo --f
ilter=label=io.kubernetes.pod.name=prod-datafilter-batch-1.2-cxmk1buk-3785657846"
time="2021-06-29T06:31:53.227Z" level=info msg="mapped container name "main" to container ID "ee3f0bbaf70d19fc179a14cad051b30a01840f44bbac9fd21517f1d2266cff5d" (created at 2021-06-29 06:29:28 +0000 UTC, status
Created)"
time="2021-06-29T06:31:54.818Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label "io.kubernetes.container.name"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=argo --f
ilter=label=io.kubernetes.pod.name=prod-datafilter-batch-1.2-cxmk1buk-3785657846"
time="2021-06-29T06:31:54.866Z" level=info msg="mapped container name "main" to container ID "8fd538622134a96d9cbf2e79e127b9ceaedfe6956500149c300fa32706ae73a7" (created at 2021-06-29 06:31:49 +0000 UTC, status
Up)"
time="2021-06-29T06:31:54.866Z" level=info msg="ignoring created container "main" that would Up -> Created"
time="2021-06-29T06:31:55.618Z" level=info msg="docker ps --all --no-trunc --format={{.Status}}|{{.Label "io.kubernetes.container.name"}}|{{.ID}}|{{.CreatedAt}} --filter=label=io.kubernetes.pod.namespace=argo --f
ilter=label=io.kubernetes.pod.name=prod-datafilter-batch-1.2-cxmk1buk-3785657846"
Are you pasting thousands of log lines? That's too much information.

# Either a workflow that reproduces the bug, or paste you whole workflow YAML, including status, something like:
kubectl get wf -o yaml ${workflow}

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

Interesting, so the main container ID can change because it is re-created, that might explain some other issues.

I will fix. Would you be able to test out an engineering build?

[alexec]

I've created a speculative fix. Github Actions will build argoproj/argoexec:dev-docker in the next 1h. Can I please ask you to test it and see if it resolves your issue?

[huangjiasingle]

@alexec not only main container re-created, in my prod env, include two case:
case one:
when main container recreated and the old main container was be gc collector deleted, the docker wait old container will return err: No such container. but pollContainerIDs will update the main container id to new created one, so we get the latest main container id and wait it. (update code)
case two:
sometimes when execute docker wait containerID, the docker daemon return No such container: containerID, but the
container id exist on the node, maybe the os loader vary high or disk io-wait is vary big and so on cause that, when first execute docker wait containerID, return err, if we do retry some times, may be no error.

[alexec]

Did you try the new image?

[huangjiasingle]

@alexec yeah.