Can't run example "Volumes" due to PVC already exists error

[BobLiu20]

Is this a BUG REPORT or FEATURE REQUEST?:
BUG REPORT
What happened:
Can't run example "Volumes" due to PVC already exists error
What you expected to happen:
Run example Volumes correctly.
How to reproduce it (as minimally and precisely as possible):
Copy from https://argoproj.github.io/docs/argo/examples/readme.html#volumes
Just submit this yaml to run:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: volumes-pvc-
spec:
  entrypoint: volumes-pvc-example
  volumeClaimTemplates:                 # define volume, same syntax as k8s Pod spec
  - metadata:
      name: workdir                     # name of volume claim
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi                  # Gi => 1024 * 1024 * 1024

  templates:
  - name: volumes-pvc-example
    steps:
    - - name: generate
        template: whalesay
    - - name: print
        template: print-message

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["echo generating message in volume; cowsay hello world | tee /mnt/vol/hello_world.txt"]
      # Mount workdir volume at /mnt/vol before invoking docker/whalesay
      volumeMounts:                     # same syntax as k8s Pod spec
      - name: workdir
        mountPath: /mnt/vol

  - name: print-message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo getting message from volume; find /mnt/vol; cat /mnt/vol/hello_world.txt"]
      # Mount workdir volume at /mnt/vol before invoking docker/whalesay
      volumeMounts:                     # same syntax as k8s Pod spec
      - name: workdir
        mountPath: /mnt/vol

Anything else we need to know?:
The PVC is not exist before I run argo submit. It is created after I run submit and error appear. I don't konw why?
Environment:

• Argo version:

$ argo version
argo: v2.2.1
  BuildDate: 2018-10-11T16:26:28Z
  GitCommit: 3b52b26190163d1f72f3aef1a39f9f291378dafb
  GitTreeState: clean
  GitTag: v2.2.1
  GoVersion: go1.10.3
  Compiler: gc
  Platform: linux/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-04-08T17:11:31Z"
  compiler: gc
  gitCommit: b7394102d6ef778017f2ca4046abbaa23b88c290
  gitTreeState: clean
  gitVersion: v1.14.1
  goVersion: go1.12.1
  major: "1"
  minor: "14"
  platform: linux/amd64
serverVersion:
  buildDate: "2019-05-16T16:14:56Z"
  compiler: gc
  gitCommit: 66049e3b21efe110454d67df4fa62b08ea79a19b
  gitTreeState: clean
  gitVersion: v1.14.2
  goVersion: go1.12.5
  major: "1"
  minor: "14"
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

$ argo get <workflowname>

• executor logs:

$ kubectl logs <failedpodname> -c init
$ kubectl logs <failedpodname> -c wait

• workflow-controller logs:

$ kubectl logs -n kube-system $(kubectl get pods -l app=workflow-controller -n kube-system -o name)
time="2019-07-30T05:25:59Z" level=info msg="Processing workflow" namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=info msg="Updated phase  -> Running" namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=info msg="Creating pvc volumes-pvc-tv9v6-workdir" namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=error msg="volumes-pvc-tv9v6 pvc create error: persistentvolumeclaims \"volumes-pvc-tv9v6-workdir\" already exists" namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=info msg="Updated message  -> persistentvolumeclaims \"volumes-pvc-tv9v6-workdir\" already exists" namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=info msg="Marking workflow completed" namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=info msg="Checking daemoned children of " namespace=default workflow=volumes-pvc-tv9v6
time="2019-07-30T05:25:59Z" level=info msg="Workflow update successful" namespace=default workflow=volumes-pvc-tv9v6

[sarabala1979]

This is resolved in #1363. 2.4 should have this fix

[BobLiu20]

@sarabala1979 Seems this is a break using issue. Is possible to have a hot release v2.3.1?