Step not getting latest overwritten global artifact

[WeiTang114]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened:
A step creates a global artifact global-art with value "A". The next step is a nested workflow with a loop, where in each iteration, there is a step creates another artifact with globalName global-art with value "B", and another step reads the global-art. As "B" is written after "A", I expect to see the reader got "B", but it got "A".

(simplified graph)

 ✔ workflow
 ├---✔ A             # write A to global-art
 └-·-✔ loop(0:B)
   | ├---✔ B         # write B to global-art
   | └---✔ read-B      # expected to read "B" but got "A"
   └-✔ loop(1:C)
     ├---✔ C         # write C to global-art
     └---✔ read-C      # expected to read "C" but got "A"

Besides, in my use case, the loop is sequential (parallelism=1), so I can definitely say I expect to see read-B gets B and read-C gets C. But in other use cases where many parallel steps are writing to the same global artifact, I wonder how we define what we should expect to get. "The latest generated" global artifact is a reasonable one. Another possible option is "the most nearly generated" global artifact. Is there a defined behavior currently?

In the "latest generated" case, if the loop is parallel, read-B and read-C may read (B,B), (B,C), (C,C) [(C,B) is not possible]. while in the "most nearly generated" case, they always read (B,C).

What you expected to happen:
The steps inside nested workflows read the latest written global artifact.

How to reproduce it (as minimally and precisely as possible):

https://gist.github.com/WeiTang114/ed3813ea801180c3cf860a3caa42eac9

# This example shows that the consume-artifact steps in the loop will 
# wrongly get "AAA" as global-art output, while it should get "BBB" and
# "CCC" which are generated by loop-echo later than "AAA" generated by 
# generate-artifact-A.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: global-artifact-passing-
spec:
  entrypoint: global-artifact-example
  templates:
  - name: global-artifact-example
    parallelism: 1
    steps:
    - - name: generate-artifact-A
        template: echo
        arguments:
          parameters:
          - name: msg
            value: "AAA"
    - - name: loop
        template: loop
        arguments:
          parameters:
          - name: msg
            value: "{{item}}"
        withItems:
        - "BBB"
        - "CCC"
        
  - name: loop
    inputs:
      parameters:
        - name: msg
    steps:
    - - name: loop-echo
        template: echo
        arguments:
          parameters:
          - name: msg
            value: "{{inputs.parameters.msg}}"
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          - name: my-art
            from: "{{workflow.outputs.artifacts.global-art}}"

  - name: echo
    inputs:
      parameters:
      - name: msg
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo {{inputs.parameters.msg}} | tee /tmp/echo.txt"]
    outputs:
      artifacts:
      - name: echo-art
        path: /tmp
        globalName: global-art

  - name: print-message
    inputs:
      artifacts:
      - name: my-art
        path: /tmp
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/echo.txt"]

Anything else we need to know?:

Environment:

• Argo version:

argo: v2.2.0
  BuildDate: 2018-08-30T08:51:40Z
  GitCommit: af636ddd8455660f307d835814d3112b90815dfd
  GitTreeState: clean
  GitTag: v2.2.0
  GoVersion: go1.10.3
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

clientVersion:
  buildDate: 2018-09-10T11:44:26Z
  compiler: gc
  gitCommit: a4529464e4629c21224b3d52edfe0ea91b072862
  gitTreeState: clean
  gitVersion: v1.11.3
  goVersion: go1.11
  major: "1"
  minor: "11"
  platform: darwin/amd64
serverVersion:
  buildDate: 2018-03-26T16:44:10Z
  compiler: gc
  gitCommit: fc32d2f3698e36b93322a3465f63a14e9f0eaead
  gitTreeState: clean
  gitVersion: v1.10.0
  goVersion: go1.9.3
  major: "1"
  minor: "10"
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

Name:                global-artifact-passing-zc2rh
Namespace:           default
ServiceAccount:      default
Status:              Succeeded
Created:             Sun Nov 11 02:30:10 +0800 (37 seconds ago)
Started:             Sun Nov 11 02:30:10 +0800 (37 seconds ago)
Finished:            Sun Nov 11 02:30:38 +0800 (9 seconds ago)
Duration:            28 seconds
Output Artifacts:
  global-art:        http://storage.googleapis.com/aip-argo-artifacts/global-artifact-passing-zc2rh/global-artifact-passing-zc2rh-4273248531/echo-art.tgz

STEP                              PODNAME                                   DURATION  MESSAGE
 ✔ global-artifact-passing-zc2rh
 ├---✔ generate-artifact-A        global-artifact-passing-zc2rh-3249287693  4s
 └-·-✔ loop(0:BBB)
   | ├---✔ loop-echo              global-artifact-passing-zc2rh-2356867109  4s
   | └---✔ consume-artifact       global-artifact-passing-zc2rh-1779394067  5s
   └-✔ loop(1:CCC)
     ├---✔ loop-echo              global-artifact-passing-zc2rh-4273248531  4s
     └---✔ consume-artifact       global-artifact-passing-zc2rh-1867768273  5s

 $ argo logs global-artifact-passing-zc2rh-1779394067 
AAA

 $ argo logs global-artifact-passing-zc2rh-1867768273
AAA

[jessesuen]

Fixed in PR #1086