Stopped workflow has no artifacts

[rajaie-sg]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I have 'archiveLogs' enabled in my Argo controller.

artifactRepository:
  # archiveLogs will archive the container logs as an artifact in S3
  archiveLogs: true
  s3:
    endpoint: s3.amazonaws.com
    region: us-west-2
    useSDKCreds: true

and controller.persistence.archive set to true.

When I Stop a running Workflow, I expect the archived Workflow to have a list of artifacts of the archived logs.

What happens is that the Workflow's output.artifacts value is empty.

In the 'wait' container, I can see that the logs were successfully archived to S3, and I can also see the log files in S3, but it seems that the Argo controller is not properly saving the output.artifacts associated with the workflow to the Workflow's metadata.

Version

v3.4.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: example
  namespace: argo
  labels:
spec:
  automountServiceAccountToken: false
  executor:
    serviceAccountName: "argo-workflow"
  templates:
    - name: entrypoint
      metadata:
      containerSet:
        containers:
          - name: main
            imagePullPolicy: Always
            resources:
              requests:
                memory: "500Mi"
                cpu: "500m"
              limits:
                memory: "500Mi"
            image: "ubuntu"
            command:
              - bash
              - '-c'
            args:
              - |
                /bin/bash <<'EOF'
                echo "hello one"
                sleep 600
                EOF
  entrypoint: entrypoint

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

The logs below are from our actual Workflow, not the sample workflow I shared in this bug report.

time="2023-01-09T16:40:00.033Z" level=info msg="Starting deadline monitor"
time="2023-01-09T16:45:00.033Z" level=info msg="Alloc=5947 TotalAlloc=12421 Sys=18642 NumGC=6 Goroutines=7"
time="2023-01-09T16:49:59.443Z" level=info msg="Deadline monitor stopped"
time="2023-01-09T16:49:59.443Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2023-01-09T16:50:00.033Z" level=info msg="Alloc=6026 TotalAlloc=12797 Sys=18642 NumGC=8 Goroutines=4"
time="2023-01-09T16:50:00.257Z" level=warning msg="Non-transient error: context canceled"
time="2023-01-09T16:50:00.257Z" level=info msg="Main container completed" error="context canceled"
time="2023-01-09T16:50:00.257Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-01-09T16:50:00.257Z" level=info msg="No output parameters"
time="2023-01-09T16:50:00.257Z" level=info msg="No output artifacts"
time="2023-01-09T16:50:00.258Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/init-wf.log, key: path/to/logs/init-wf.log"
time="2023-01-09T16:50:00.258Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-01-09T16:50:00.421Z" level=info msg="Saving file to s3" bucket=XXXXXXXX endpoint=s3.amazonaws.com key=path/to/logs/init-wf.log path=/tmp/argo/outputs/logs/init-wf.log
time="2023-01-09T16:50:00.568Z" level=info msg="Save artifact" artifactName=init-wf-logs duration=309.898733ms error="<nil>" key=path/to/logs/init-wf.log
time="2023-01-09T16:50:00.568Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/init-wf.log
time="2023-01-09T16:50:00.568Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/init-wf.log"
time="2023-01-09T16:50:00.568Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/build-artifact.log, key: path/to/logs/build-artifact.log"
time="2023-01-09T16:50:00.568Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-01-09T16:50:00.584Z" level=info msg="Saving file to s3" bucket=XXXXXXXX endpoint=s3.amazonaws.com key=path/to/logs/build-artifact.log path=/tmp/argo/outputs/logs/build-artifact.log
time="2023-01-09T16:50:00.677Z" level=info msg="Save artifact" artifactName=build-pex-logs duration=108.811963ms error="<nil>" key=path/to/logs/build-artifact.log
time="2023-01-09T16:50:00.677Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/build-artifact.log
time="2023-01-09T16:50:00.677Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/build-artifact.log"
time="2023-01-09T16:50:00.677Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/upload-artifact.log, key: path/to/logs/upload-artifact.log"
time="2023-01-09T16:50:00.677Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-01-09T16:50:00.691Z" level=info msg="Saving file to s3" bucket=XXXXXXXX endpoint=s3.amazonaws.com key=path/to/logs/upload-artifact.log path=/tmp/argo/outputs/logs/upload-artifact.log
time="2023-01-09T16:50:00.791Z" level=info msg="Save artifact" artifactName=upload-pex-logs duration=113.400335ms error="<nil>" key=path/to/logs/upload-artifact.log
time="2023-01-09T16:50:00.791Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/upload-artifact.log
time="2023-01-09T16:50:00.791Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/upload-artifact.log"
time="2023-01-09T16:50:00.791Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: path/to/logs/main.log"
time="2023-01-09T16:50:00.791Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-01-09T16:50:00.807Z" level=info msg="Saving file to s3" bucket=XXXXXXXX endpoint=s3.amazonaws.com key=path/to/logs/main.log path=/tmp/argo/outputs/logs/main.log
time="2023-01-09T16:50:00.911Z" level=info msg="Save artifact" artifactName=main-logs duration=120.191245ms error="<nil>" key=path/to/logs/main.log
time="2023-01-09T16:50:00.911Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2023-01-09T16:50:00.911Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2023-01-09T16:50:00.911Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/upload-data.log, key: path/to/logs/upload-data.log"
time="2023-01-09T16:50:00.911Z" level=info msg="Creating minio client using AWS SDK credentials"
time="2023-01-09T16:50:00.931Z" level=info msg="Saving file to s3" bucket=XXXXXXXX endpoint=s3.amazonaws.com key=path/to/logs/upload-data.log path=/tmp/argo/outputs/logs/upload-data.log
time="2023-01-09T16:50:01.038Z" level=info msg="Save artifact" artifactName=upload-data-logs duration=126.703133ms error="<nil>" key=path/to/logs/upload-data.log
time="2023-01-09T16:50:01.038Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/upload-data.log
time="2023-01-09T16:50:01.038Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/upload-data.log"
time="2023-01-09T16:50:01.060Z" level=info msg="Create workflowtaskresults 201"
time="2023-01-09T16:50:01.061Z" level=info msg="Alloc=8168 TotalAlloc=22200 Sys=23250 NumGC=11 Goroutines=17"