Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug] condition sample stuck in running state with argo v3.1.0 #5937

Closed
Bobgy opened this issue Jun 29, 2021 · 6 comments · Fixed by #5971
Closed

[bug] condition sample stuck in running state with argo v3.1.0 #5937

Bobgy opened this issue Jun 29, 2021 · 6 comments · Fixed by #5971
Assignees
@Bobgy
Labels
area/sdk kind/bug priority/p0

Comments

@Bobgy
Copy link
Contributor

Bobgy commented Jun 29, 2021 •
edited
Loading

What steps did you take

  1. run v2 sample test

What happened:

  1. it times out waiting for condition sample to finish
  2. the sample runs and gets cached as usual
    image
  3. but the workflow itself is always stuck in running state

What did you expect to happen:

the sample should finish as normal

Environment:

  • How do you deploy Kubeflow Pipelines (KFP)? KFP standalone 1.6.0-alpha.1
  • KFP SDK version: master

Anything else you would like to add:

Here's status of the argo workflow, it's also stuck in running. So this is not a problem with persistence agent.

status:
  artifactRepositoryRef:
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  finishedAt: null
  nodes:
    single-condition-pipeline-xb2hw:
      children:
      - single-condition-pipeline-xb2hw-690456091
      displayName: single-condition-pipeline-xb2hw
      finishedAt: null
      id: single-condition-pipeline-xb2hw
      inputs:
        parameters:
        - name: force_flip_result
          value: heads
        - name: text
          value: condition test
      name: single-condition-pipeline-xb2hw
      phase: Running
      progress: 4/4
      startedAt: "2021-06-28T13:58:12Z"
      templateName: single-condition-pipeline
      templateScope: local/single-condition-pipeline-xb2hw
      type: DAG
    single-condition-pipeline-xb2hw-151083147:
      boundaryID: single-condition-pipeline-xb2hw
      children:
      - single-condition-pipeline-xb2hw-329071385
      - single-condition-pipeline-xb2hw-1404454342
      displayName: condition-1
      finishedAt: "2021-06-28T13:59:30Z"
      id: single-condition-pipeline-xb2hw-151083147
      inputs:
        parameters:
        - name: text
          value: condition test
      name: single-condition-pipeline-xb2hw.condition-1
      outboundNodes:
      - single-condition-pipeline-xb2hw-1421231961
      - single-condition-pipeline-xb2hw-1404454342
      phase: Succeeded
      progress: 3/3
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-28T13:58:50Z"
      templateName: condition-1
      templateScope: local/single-condition-pipeline-xb2hw
      type: DAG
    single-condition-pipeline-xb2hw-329071385:
      boundaryID: single-condition-pipeline-xb2hw-151083147
      children:
      - single-condition-pipeline-xb2hw-1421231961
      displayName: flip-coin-2
      finishedAt: "2021-06-28T13:58:56Z"
      id: single-condition-pipeline-xb2hw-329071385
      name: single-condition-pipeline-xb2hw.condition-1.flip-coin-2
      outputs:
        artifacts:
        - name: flip-coin-2-Output
          path: /tmp/outputs/Output/data
          s3:
            key: artifacts/single-condition-pipeline-982vv/2021/06/28/single-condition-pipeline-982vv-3001282121/flip-coin-2-Output.tgz
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-982vv/2021/06/28/single-condition-pipeline-982vv-3001282121/main.log
        exitCode: "0"
        parameters:
        - name: flip-coin-2-Output
          value: heads
          valueFrom:
            path: /tmp/outputs/Output/data
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-28T13:58:50Z"
      templateName: flip-coin-2
      templateScope: local/single-condition-pipeline-xb2hw
      type: Pod
    single-condition-pipeline-xb2hw-690456091:
      boundaryID: single-condition-pipeline-xb2hw
      children:
      - single-condition-pipeline-xb2hw-151083147
      displayName: flip-coin
      finishedAt: "2021-06-28T13:58:41Z"
      hostNodeName: gke-kfp-std-default-pool-1c1207aa-2eyx
      id: single-condition-pipeline-xb2hw-690456091
      inputs:
        parameters:
        - name: force_flip_result
          value: heads
      name: single-condition-pipeline-xb2hw.flip-coin
      outputs:
        artifacts:
        - name: flip-coin-Output
          path: /tmp/outputs/Output/data
          s3:
            key: artifacts/single-condition-pipeline-xb2hw/2021/06/28/single-condition-pipeline-xb2hw-690456091/flip-coin-Output.tgz
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-xb2hw/2021/06/28/single-condition-pipeline-xb2hw-690456091/main.log
        exitCode: "0"
        parameters:
        - name: flip-coin-Output
          value: heads
          valueFrom:
            path: /tmp/outputs/Output/data
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 7
        memory: 2
      startedAt: "2021-06-28T13:58:12Z"
      templateName: flip-coin
      templateScope: local/single-condition-pipeline-xb2hw
      type: Pod
    single-condition-pipeline-xb2hw-1404454342:
      boundaryID: single-condition-pipeline-xb2hw-151083147
      displayName: print-msg-3
      finishedAt: "2021-06-28T13:59:02Z"
      hostNodeName: gke-kfp-std-default-pool-1c1207aa-2eyx
      id: single-condition-pipeline-xb2hw-1404454342
      inputs:
        parameters:
        - name: text
          value: condition test
      name: single-condition-pipeline-xb2hw.condition-1.print-msg-3
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-982vv/2021/06/28/single-condition-pipeline-982vv-3511891414/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-28T13:58:50Z"
      templateName: print-msg-3
      templateScope: local/single-condition-pipeline-xb2hw
      type: Pod
    single-condition-pipeline-xb2hw-1421231961:
      boundaryID: single-condition-pipeline-xb2hw-151083147
      displayName: print-msg-2
      finishedAt: "2021-06-28T13:59:09Z"
      id: single-condition-pipeline-xb2hw-1421231961
      inputs:
        parameters:
        - name: flip-coin-2-Output
          value: heads
      name: single-condition-pipeline-xb2hw.condition-1.print-msg-2
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-982vv/2021/06/28/single-condition-pipeline-982vv-3528669033/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-28T13:59:00Z"
      templateName: print-msg-2
      templateScope: local/single-condition-pipeline-xb2hw
      type: Pod
    single-condition-pipeline-xb2hw-3316531579:
      boundaryID: single-condition-pipeline-xb2hw
      displayName: print-msg
      finishedAt: "2021-06-28T13:59:04Z"
      hostNodeName: gke-kfp-std-default-pool-1c1207aa-2eyx
      id: single-condition-pipeline-xb2hw-3316531579
      inputs:
        parameters:
        - name: flip-coin-Output
          value: heads
      name: single-condition-pipeline-xb2hw.print-msg
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-982vv/2021/06/28/single-condition-pipeline-982vv-1648329963/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-28T13:58:50Z"
      templateName: print-msg
      templateScope: local/single-condition-pipeline-xb2hw
      type: Pod
  phase: Running
  progress: 5/5
  resourcesDuration:
    cpu: 7
    memory: 2
  startedAt: "2021-06-28T13:58:12Z"

Labels

/area sdk

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@Bobgy Bobgy added the kind/bug label Jun 29, 2021
@Bobgy Bobgy self-assigned this Jun 29, 2021
@Bobgy Bobgy mentioned this issue Jun 29, 2021
Closed
10 tasks
@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 29, 2021

I tried to delete cache db and run the condition pipeline again without cache, but it is stuck in running state again.
image

Conclusion: we can rule out caching from possible reasons for this.

@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 29, 2021 

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    pipelines.kubeflow.org/kfp_sdk_version: 1.6.4
    pipelines.kubeflow.org/pipeline_compilation_time: 2021-06-29T15:04:57.873529
    pipelines.kubeflow.org/pipeline_spec: '{"inputs": [{"default": "condition test",
      "name": "text", "optional": true, "type": "String"}, {"default": "", "name":
      "force_flip_result", "optional": true, "type": "String"}], "name": "single-condition-pipeline"}'
    pipelines.kubeflow.org/run_name: my_pipeline 2021-06-29 15-04-57
  creationTimestamp: "2021-06-29T15:05:23Z"
  generateName: single-condition-pipeline-
  generation: 5
  labels:
    pipeline/runid: 2f797483-ea0f-47d5-ba09-4eb19fa1e271
    pipelines.kubeflow.org/kfp_sdk_version: 1.6.4
    workflows.argoproj.io/phase: Running
    manager: workflow-controller
    operation: Update
    time: "2021-06-29T15:05:53Z"
  name: single-condition-pipeline-cmn6x
  namespace: kubeflow
  resourceVersion: "30785245"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/kubeflow/workflows/single-condition-pipeline-cmn6x
  uid: 44151d24-2ac8-4a05-b28f-66e556901f89
spec:
  arguments:
    parameters:
    - name: text
      value: condition test
    - name: force_flip_result
      value: heads
  entrypoint: single-condition-pipeline
  serviceAccountName: pipeline-runner
  templates:
  - dag:
      tasks:
      - arguments: {}
        name: flip-coin-2
        template: flip-coin-2
      - arguments:
          parameters:
          - name: flip-coin-2-Output
            value: '{{tasks.flip-coin-2.outputs.parameters.flip-coin-2-Output}}'
        dependencies:
        - flip-coin-2
        name: print-msg-2
        template: print-msg-2
      - arguments:
          parameters:
          - name: text
            value: '{{inputs.parameters.text}}'
        name: print-msg-3
        template: print-msg-3
    inputs:
      parameters:
      - name: text
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        pipelines.kubeflow.org/cache_enabled: "true"
    name: condition-1
    outputs: {}
  - container:
      args:
      - --force-flip-result
      - '{{inputs.parameters.force_flip_result}}'
      - '----output-paths'
      - /tmp/outputs/Output/data
      command:
      - sh
      - -ec
      - |
        program_path=$(mktemp)
        printf "%s" "$0" > "$program_path"
        python3 -u "$program_path" "$@"
      - |
        def flip_coin(force_flip_result = ''):
            """Flip a coin and output heads or tails randomly."""
            if force_flip_result:
                return force_flip_result
            import random
            result = 'heads' if random.randint(0, 1) == 0 else 'tails'
            return result

        def _serialize_str(str_value: str) -> str:
            if not isinstance(str_value, str):
                raise TypeError('Value "{}" has type "{}" instead of str.'.format(str(str_value), str(type(str_value))))
            return str_value

        import argparse
        _parser = argparse.ArgumentParser(prog='Flip coin', description='Flip a coin and output heads or tails randomly.')
        _parser.add_argument("--force-flip-result", dest="force_flip_result", type=str, required=False, default=argparse.SUPPRESS)
        _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
        _parsed_args = vars(_parser.parse_args())
        _output_files = _parsed_args.pop("_output_paths", [])

        _outputs = flip_coin(**_parsed_args)

        _outputs = [_outputs]

        _output_serializers = [
            _serialize_str,

        ]

        import os
        for idx, output_file in enumerate(_output_files):
            try:
                os.makedirs(os.path.dirname(output_file))
            except OSError:
                pass
            with open(output_file, 'w') as f:
                f.write(_output_serializers[idx](_outputs[idx]))
      image: python:3.7
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: force_flip_result
    metadata:
      annotations:
        pipelines.kubeflow.org/arguments.parameters: '{"force_flip_result": "{{inputs.parameters.force_flip_result}}"}'
        pipelines.kubeflow.org/component_ref: '{}'
        pipelines.kubeflow.org/component_spec: '{"description": "Flip a coin and output
          heads or tails randomly.", "implementation": {"container": {"args": [{"if":
          {"cond": {"isPresent": "force_flip_result"}, "then": ["--force-flip-result",
          {"inputValue": "force_flip_result"}]}}, "----output-paths", {"outputPath":
          "Output"}], "command": ["sh", "-ec", "program_path=$(mktemp)\nprintf \"%s\"
          \"$0\" > \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n", "def
          flip_coin(force_flip_result = ''''):\n    \"\"\"Flip a coin and output heads
          or tails randomly.\"\"\"\n    if force_flip_result:\n        return force_flip_result\n    import
          random\n    result = ''heads'' if random.randint(0, 1) == 0 else ''tails''\n    return
          result\n\ndef _serialize_str(str_value: str) -> str:\n    if not isinstance(str_value,
          str):\n        raise TypeError(''Value \"{}\" has type \"{}\" instead of
          str.''.format(str(str_value), str(type(str_value))))\n    return str_value\n\nimport
          argparse\n_parser = argparse.ArgumentParser(prog=''Flip coin'', description=''Flip
          a coin and output heads or tails randomly.'')\n_parser.add_argument(\"--force-flip-result\",
          dest=\"force_flip_result\", type=str, required=False, default=argparse.SUPPRESS)\n_parser.add_argument(\"----output-paths\",
          dest=\"_output_paths\", type=str, nargs=1)\n_parsed_args = vars(_parser.parse_args())\n_output_files
          = _parsed_args.pop(\"_output_paths\", [])\n\n_outputs = flip_coin(**_parsed_args)\n\n_outputs
          = [_outputs]\n\n_output_serializers = [\n    _serialize_str,\n\n]\n\nimport
          os\nfor idx, output_file in enumerate(_output_files):\n    try:\n        os.makedirs(os.path.dirname(output_file))\n    except
          OSError:\n        pass\n    with open(output_file, ''w'') as f:\n        f.write(_output_serializers[idx](_outputs[idx]))\n"],
          "image": "python:3.7"}}, "inputs": [{"default": "", "name": "force_flip_result",
          "optional": true, "type": "String"}], "name": "Flip coin", "outputs": [{"name":
          "Output", "type": "String"}]}'
        sidecar.istio.io/inject: "false"
      labels:
        pipelines.kubeflow.org/cache_enabled: "true"
        pipelines.kubeflow.org/kfp_sdk_version: 1.6.4
        pipelines.kubeflow.org/pipeline-sdk-type: kfp
    name: flip-coin
    outputs:
      artifacts:
      - name: flip-coin-Output
        path: /tmp/outputs/Output/data
      parameters:
      - name: flip-coin-Output
        valueFrom:
          path: /tmp/outputs/Output/data
  - container:
      args:
      - --force-flip-result
      - ""
      - '----output-paths'
      - /tmp/outputs/Output/data
      command:
      - sh
      - -ec
      - |
        program_path=$(mktemp)
        printf "%s" "$0" > "$program_path"
        python3 -u "$program_path" "$@"
      - |
        def flip_coin(force_flip_result = ''):
            """Flip a coin and output heads or tails randomly."""
            if force_flip_result:
                return force_flip_result
            import random
            result = 'heads' if random.randint(0, 1) == 0 else 'tails'
            return result

        def _serialize_str(str_value: str) -> str:
            if not isinstance(str_value, str):
                raise TypeError('Value "{}" has type "{}" instead of str.'.format(str(str_value), str(type(str_value))))
            return str_value

        import argparse
        _parser = argparse.ArgumentParser(prog='Flip coin', description='Flip a coin and output heads or tails randomly.')
        _parser.add_argument("--force-flip-result", dest="force_flip_result", type=str, required=False, default=argparse.SUPPRESS)
        _parser.add_argument("----output-paths", dest="_output_paths", type=str, nargs=1)
        _parsed_args = vars(_parser.parse_args())
        _output_files = _parsed_args.pop("_output_paths", [])

        _outputs = flip_coin(**_parsed_args)

        _outputs = [_outputs]

        _output_serializers = [
            _serialize_str,

        ]

        import os
        for idx, output_file in enumerate(_output_files):
            try:
                os.makedirs(os.path.dirname(output_file))
            except OSError:
                pass
            with open(output_file, 'w') as f:
                f.write(_output_serializers[idx](_outputs[idx]))
      image: python:3.7
      name: ""
      resources: {}
    inputs: {}
    metadata:
      annotations:
        pipelines.kubeflow.org/arguments.parameters: '{"force_flip_result": ""}'
        pipelines.kubeflow.org/component_ref: '{}'
        pipelines.kubeflow.org/component_spec: '{"description": "Flip a coin and output
          heads or tails randomly.", "implementation": {"container": {"args": [{"if":
          {"cond": {"isPresent": "force_flip_result"}, "then": ["--force-flip-result",
          {"inputValue": "force_flip_result"}]}}, "----output-paths", {"outputPath":
          "Output"}], "command": ["sh", "-ec", "program_path=$(mktemp)\nprintf \"%s\"
          \"$0\" > \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n", "def
          flip_coin(force_flip_result = ''''):\n    \"\"\"Flip a coin and output heads
          or tails randomly.\"\"\"\n    if force_flip_result:\n        return force_flip_result\n    import
          random\n    result = ''heads'' if random.randint(0, 1) == 0 else ''tails''\n    return
          result\n\ndef _serialize_str(str_value: str) -> str:\n    if not isinstance(str_value,
          str):\n        raise TypeError(''Value \"{}\" has type \"{}\" instead of
          str.''.format(str(str_value), str(type(str_value))))\n    return str_value\n\nimport
          argparse\n_parser = argparse.ArgumentParser(prog=''Flip coin'', description=''Flip
          a coin and output heads or tails randomly.'')\n_parser.add_argument(\"--force-flip-result\",
          dest=\"force_flip_result\", type=str, required=False, default=argparse.SUPPRESS)\n_parser.add_argument(\"----output-paths\",
          dest=\"_output_paths\", type=str, nargs=1)\n_parsed_args = vars(_parser.parse_args())\n_output_files
          = _parsed_args.pop(\"_output_paths\", [])\n\n_outputs = flip_coin(**_parsed_args)\n\n_outputs
          = [_outputs]\n\n_output_serializers = [\n    _serialize_str,\n\n]\n\nimport
          os\nfor idx, output_file in enumerate(_output_files):\n    try:\n        os.makedirs(os.path.dirname(output_file))\n    except
          OSError:\n        pass\n    with open(output_file, ''w'') as f:\n        f.write(_output_serializers[idx](_outputs[idx]))\n"],
          "image": "python:3.7"}}, "inputs": [{"default": "", "name": "force_flip_result",
          "optional": true, "type": "String"}], "name": "Flip coin", "outputs": [{"name":
          "Output", "type": "String"}]}'
        sidecar.istio.io/inject: "false"
      labels:
        pipelines.kubeflow.org/cache_enabled: "true"
        pipelines.kubeflow.org/kfp_sdk_version: 1.6.4
        pipelines.kubeflow.org/pipeline-sdk-type: kfp
    name: flip-coin-2
    outputs:
      artifacts:
      - name: flip-coin-2-Output
        path: /tmp/outputs/Output/data
      parameters:
      - name: flip-coin-2-Output
        valueFrom:
          path: /tmp/outputs/Output/data
  - container:
      args:
      - --msg
      - '{{inputs.parameters.flip-coin-Output}}'
      command:
      - sh
      - -ec
      - |
        program_path=$(mktemp)
        printf "%s" "$0" > "$program_path"
        python3 -u "$program_path" "$@"
      - |
        def print_msg(msg):
            """Print a message."""
            print(msg)

        import argparse
        _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
        _parser.add_argument("--msg", dest="msg", type=str, required=True, default=argparse.SUPPRESS)
        _parsed_args = vars(_parser.parse_args())

        _outputs = print_msg(**_parsed_args)
      image: python:3.7
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: flip-coin-Output
    metadata:
      annotations:
        pipelines.kubeflow.org/arguments.parameters: '{"msg": "{{inputs.parameters.flip-coin-Output}}"}'
        pipelines.kubeflow.org/component_ref: '{}'
        pipelines.kubeflow.org/component_spec: '{"description": "Print a message.",
          "implementation": {"container": {"args": ["--msg", {"inputValue": "msg"}],
          "command": ["sh", "-ec", "program_path=$(mktemp)\nprintf \"%s\" \"$0\" >
          \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n", "def print_msg(msg):\n    \"\"\"Print
          a message.\"\"\"\n    print(msg)\n\nimport argparse\n_parser = argparse.ArgumentParser(prog=''Print
          msg'', description=''Print a message.'')\n_parser.add_argument(\"--msg\",
          dest=\"msg\", type=str, required=True, default=argparse.SUPPRESS)\n_parsed_args
          = vars(_parser.parse_args())\n\n_outputs = print_msg(**_parsed_args)\n"],
          "image": "python:3.7"}}, "inputs": [{"name": "msg", "type": "String"}],
          "name": "Print msg"}'
        sidecar.istio.io/inject: "false"
      labels:
        pipelines.kubeflow.org/cache_enabled: "true"
        pipelines.kubeflow.org/kfp_sdk_version: 1.6.4
        pipelines.kubeflow.org/pipeline-sdk-type: kfp
    name: print-msg
    outputs: {}
  - container:
      args:
      - --msg
      - '{{inputs.parameters.flip-coin-2-Output}}'
      command:
      - sh
      - -ec
      - |
        program_path=$(mktemp)
        printf "%s" "$0" > "$program_path"
        python3 -u "$program_path" "$@"
      - |
        def print_msg(msg):
            """Print a message."""
            print(msg)

        import argparse
        _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
        _parser.add_argument("--msg", dest="msg", type=str, required=True, default=argparse.SUPPRESS)
        _parsed_args = vars(_parser.parse_args())

        _outputs = print_msg(**_parsed_args)
      image: python:3.7
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: flip-coin-2-Output
    metadata:
      annotations:
        pipelines.kubeflow.org/arguments.parameters: '{"msg": "{{inputs.parameters.flip-coin-2-Output}}"}'
        pipelines.kubeflow.org/component_ref: '{}'
        pipelines.kubeflow.org/component_spec: '{"description": "Print a message.",
          "implementation": {"container": {"args": ["--msg", {"inputValue": "msg"}],
          "command": ["sh", "-ec", "program_path=$(mktemp)\nprintf \"%s\" \"$0\" >
          \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n", "def print_msg(msg):\n    \"\"\"Print
          a message.\"\"\"\n    print(msg)\n\nimport argparse\n_parser = argparse.ArgumentParser(prog=''Print
          msg'', description=''Print a message.'')\n_parser.add_argument(\"--msg\",
          dest=\"msg\", type=str, required=True, default=argparse.SUPPRESS)\n_parsed_args
          = vars(_parser.parse_args())\n\n_outputs = print_msg(**_parsed_args)\n"],
          "image": "python:3.7"}}, "inputs": [{"name": "msg", "type": "String"}],
          "name": "Print msg"}'
        sidecar.istio.io/inject: "false"
      labels:
        pipelines.kubeflow.org/cache_enabled: "true"
        pipelines.kubeflow.org/kfp_sdk_version: 1.6.4
        pipelines.kubeflow.org/pipeline-sdk-type: kfp
    name: print-msg-2
    outputs: {}
  - container:
      args:
      - --msg
      - '{{inputs.parameters.text}}'
      command:
      - sh
      - -ec
      - |
        program_path=$(mktemp)
        printf "%s" "$0" > "$program_path"
        python3 -u "$program_path" "$@"
      - |
        def print_msg(msg):
            """Print a message."""
            print(msg)

        import argparse
        _parser = argparse.ArgumentParser(prog='Print msg', description='Print a message.')
        _parser.add_argument("--msg", dest="msg", type=str, required=True, default=argparse.SUPPRESS)
        _parsed_args = vars(_parser.parse_args())

        _outputs = print_msg(**_parsed_args)
      image: python:3.7
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: text
    metadata:
      annotations:
        pipelines.kubeflow.org/arguments.parameters: '{"msg": "{{inputs.parameters.text}}"}'
        pipelines.kubeflow.org/component_ref: '{}'
        pipelines.kubeflow.org/component_spec: '{"description": "Print a message.",
          "implementation": {"container": {"args": ["--msg", {"inputValue": "msg"}],
          "command": ["sh", "-ec", "program_path=$(mktemp)\nprintf \"%s\" \"$0\" >
          \"$program_path\"\npython3 -u \"$program_path\" \"$@\"\n", "def print_msg(msg):\n    \"\"\"Print
          a message.\"\"\"\n    print(msg)\n\nimport argparse\n_parser = argparse.ArgumentParser(prog=''Print
          msg'', description=''Print a message.'')\n_parser.add_argument(\"--msg\",
          dest=\"msg\", type=str, required=True, default=argparse.SUPPRESS)\n_parsed_args
          = vars(_parser.parse_args())\n\n_outputs = print_msg(**_parsed_args)\n"],
          "image": "python:3.7"}}, "inputs": [{"name": "msg", "type": "String"}],
          "name": "Print msg"}'
        sidecar.istio.io/inject: "false"
      labels:
        pipelines.kubeflow.org/cache_enabled: "true"
        pipelines.kubeflow.org/kfp_sdk_version: 1.6.4
        pipelines.kubeflow.org/pipeline-sdk-type: kfp
    name: print-msg-3
    outputs: {}
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: text
            value: '{{inputs.parameters.text}}'
        dependencies:
        - flip-coin
        name: condition-1
        template: condition-1
        when: '"{{tasks.flip-coin.outputs.parameters.flip-coin-Output}}" == "heads"'
      - arguments:
          parameters:
          - name: force_flip_result
            value: '{{inputs.parameters.force_flip_result}}'
        name: flip-coin
        template: flip-coin
      - arguments:
          parameters:
          - name: flip-coin-Output
            value: '{{tasks.flip-coin.outputs.parameters.flip-coin-Output}}'
        dependencies:
        - flip-coin
        name: print-msg
        template: print-msg
    inputs:
      parameters:
      - name: force_flip_result
      - name: text
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        pipelines.kubeflow.org/cache_enabled: "true"
    name: single-condition-pipeline
    outputs: {}
status:
  artifactRepositoryRef:
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  finishedAt: null
  nodes:
    single-condition-pipeline-cmn6x:
      children:
      - single-condition-pipeline-cmn6x-511824490
      displayName: single-condition-pipeline-cmn6x
      finishedAt: null
      id: single-condition-pipeline-cmn6x
      inputs:
        parameters:
        - name: force_flip_result
          value: heads
        - name: text
          value: condition test
      name: single-condition-pipeline-cmn6x
      phase: Running
      progress: 4/4
      startedAt: "2021-06-29T15:05:23Z"
      templateName: single-condition-pipeline
      templateScope: local/single-condition-pipeline-cmn6x
      type: DAG
    single-condition-pipeline-cmn6x-511824490:
      boundaryID: single-condition-pipeline-cmn6x
      children:
      - single-condition-pipeline-cmn6x-1818944638
      displayName: flip-coin
      finishedAt: "2021-06-29T15:05:24Z"
      id: single-condition-pipeline-cmn6x-511824490
      inputs:
        parameters:
        - name: force_flip_result
          value: heads
      name: single-condition-pipeline-cmn6x.flip-coin
      outputs:
        artifacts:
        - name: flip-coin-Output
          path: /tmp/outputs/Output/data
          s3:
            key: artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1478846994/flip-coin-Output.tgz
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1478846994/main.log
        exitCode: "0"
        parameters:
        - name: flip-coin-Output
          value: heads
          valueFrom:
            path: /tmp/outputs/Output/data
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-29T15:05:23Z"
      templateName: flip-coin
      templateScope: local/single-condition-pipeline-cmn6x
      type: Pod
    single-condition-pipeline-cmn6x-1818944638:
      boundaryID: single-condition-pipeline-cmn6x
      children:
      - single-condition-pipeline-cmn6x-2284007376
      - single-condition-pipeline-cmn6x-2992979395
      displayName: condition-1
      finishedAt: "2021-06-29T15:05:53Z"
      id: single-condition-pipeline-cmn6x-1818944638
      inputs:
        parameters:
        - name: text
          value: condition test
      name: single-condition-pipeline-cmn6x.condition-1
      outboundNodes:
      - single-condition-pipeline-cmn6x-2976201776
      - single-condition-pipeline-cmn6x-2992979395
      phase: Succeeded
      progress: 3/3
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-29T15:05:33Z"
      templateName: condition-1
      templateScope: local/single-condition-pipeline-cmn6x
      type: DAG
    single-condition-pipeline-cmn6x-2284007376:
      boundaryID: single-condition-pipeline-cmn6x-1818944638
      children:
      - single-condition-pipeline-cmn6x-2976201776
      displayName: flip-coin-2
      finishedAt: "2021-06-29T15:05:34Z"
      id: single-condition-pipeline-cmn6x-2284007376
      name: single-condition-pipeline-cmn6x.condition-1.flip-coin-2
      outputs:
        artifacts:
        - name: flip-coin-2-Output
          path: /tmp/outputs/Output/data
          s3:
            key: artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1354333128/flip-coin-2-Output.tgz
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1354333128/main.log
        exitCode: "0"
        parameters:
        - name: flip-coin-2-Output
          value: tails
          valueFrom:
            path: /tmp/outputs/Output/data
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-29T15:05:33Z"
      templateName: flip-coin-2
      templateScope: local/single-condition-pipeline-cmn6x
      type: Pod
    single-condition-pipeline-cmn6x-2501038730:
      boundaryID: single-condition-pipeline-cmn6x
      displayName: print-msg
      finishedAt: "2021-06-29T15:05:35Z"
      id: single-condition-pipeline-cmn6x-2501038730
      inputs:
        parameters:
        - name: flip-coin-Output
          value: heads
      name: single-condition-pipeline-cmn6x.print-msg
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3067881266/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-29T15:05:33Z"
      templateName: print-msg
      templateScope: local/single-condition-pipeline-cmn6x
      type: Pod
    single-condition-pipeline-cmn6x-2976201776:
      boundaryID: single-condition-pipeline-cmn6x-1818944638
      displayName: print-msg-2
      finishedAt: "2021-06-29T15:05:44Z"
      id: single-condition-pipeline-cmn6x-2976201776
      inputs:
        parameters:
        - name: flip-coin-2-Output
          value: tails
      name: single-condition-pipeline-cmn6x.condition-1.print-msg-2
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3432968360/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-29T15:05:43Z"
      templateName: print-msg-2
      templateScope: local/single-condition-pipeline-cmn6x
      type: Pod
    single-condition-pipeline-cmn6x-2992979395:
      boundaryID: single-condition-pipeline-cmn6x-1818944638
      displayName: print-msg-3
      finishedAt: "2021-06-29T15:05:35Z"
      id: single-condition-pipeline-cmn6x-2992979395
      inputs:
        parameters:
        - name: text
          value: condition test
      name: single-condition-pipeline-cmn6x.condition-1.print-msg-3
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3449745979/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-06-29T15:05:33Z"
      templateName: print-msg-3
      templateScope: local/single-condition-pipeline-cmn6x
      type: Pod
  phase: Running
  progress: 5/5
  resourcesDuration:
    cpu: 0
    memory: 0
  startedAt: "2021-06-29T15:05:23Z"

@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 29, 2021 •
edited
Loading

workflow controller logs related to this pipeline run:

time="2021-06-29T14:53:52.825Z" level=info msg="Processing workflow" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:53:52.921Z" level=info msg="Updated phase -> Running" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:53:52.922Z" level=info msg="DAG node single-condition-pipeline-dbznb initialized Running" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:53:52.924Z" level=info msg="All of node single-condition-pipeline-dbznb.flip-coin dependencies [] completed" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:53:52.925Z" level=info msg="Pod node single-condition-pipeline-dbznb-1478846994 initialized Pending" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:53:53.358Z" level=info msg="Created pod: single-condition-pipeline-dbznb.flip-coin (single-condition-pipeline-dbznb-1478846994)" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:53:53.496Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=30779093 workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.488Z" level=info msg="Processing workflow" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.489Z" level=info msg="Updating node single-condition-pipeline-dbznb-1478846994 exit code 0" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.489Z" level=info msg="Setting node single-condition-pipeline-dbznb-1478846994 outputs: {"parameters":[{"name":"flip-coin-Output","value":"heads","valueFrom":{"path":"/tmp/outputs/Output/data"}}],"artifacts":[{"name":"flip-coin-Output","path":"/tmp/outputs/Output/data","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1478846994/flip-coin-Output.tgz"}},{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1478846994/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.489Z" level=info msg="Updating node single-condition-pipeline-dbznb-1478846994 status Pending -> Succeeded" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.490Z" level=info msg="All of node single-condition-pipeline-dbznb.condition-1 dependencies [flip-coin] completed" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.490Z" level=info msg="DAG node single-condition-pipeline-dbznb-2442073942 initialized Running" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.490Z" level=info msg="All of node single-condition-pipeline-dbznb.condition-1.flip-coin-2 dependencies [] completed" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.490Z" level=info msg="Pod node single-condition-pipeline-dbznb-1354333128 initialized Pending" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.734Z" level=info msg="Created pod: single-condition-pipeline-dbznb.condition-1.flip-coin-2 (single-condition-pipeline-dbznb-1354333128)" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.734Z" level=info msg="All of node single-condition-pipeline-dbznb.condition-1.print-msg-3 dependencies [] completed" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:03.735Z" level=info msg="Pod node single-condition-pipeline-dbznb-3449745979 initialized Pending" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:04.237Z" level=info msg="Created pod: single-condition-pipeline-dbznb.condition-1.print-msg-3 (single-condition-pipeline-dbznb-3449745979)" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:04.237Z" level=info msg="All of node single-condition-pipeline-dbznb.print-msg dependencies [flip-coin] completed" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:04.237Z" level=info msg="Pod node single-condition-pipeline-dbznb-3067881266 initialized Pending" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:04.419Z" level=info msg="Created pod: single-condition-pipeline-dbznb.print-msg (single-condition-pipeline-dbznb-3067881266)" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:04.548Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=30779379 workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:04.555Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kubeflow/single-condition-pipeline-dbznb-1478846994/labelPodCompleted
time="2021-06-29T14:54:13.789Z" level=info msg="Processing workflow" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.790Z" level=info msg="Updating node single-condition-pipeline-dbznb-3449745979 exit code 0" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.790Z" level=info msg="Setting node single-condition-pipeline-dbznb-3449745979 outputs: {"artifacts":[{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3449745979/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.790Z" level=info msg="Updating node single-condition-pipeline-dbznb-3449745979 status Pending -> Succeeded" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.790Z" level=info msg="Updating node single-condition-pipeline-dbznb-3067881266 message: ContainerCreating" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.790Z" level=info msg="Updating node single-condition-pipeline-dbznb-1354333128 exit code 0" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.790Z" level=info msg="Setting node single-condition-pipeline-dbznb-1354333128 outputs: {"parameters":[{"name":"flip-coin-2-Output","value":"tails","valueFrom":{"path":"/tmp/outputs/Output/data"}}],"artifacts":[{"name":"flip-coin-2-Output","path":"/tmp/outputs/Output/data","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1354333128/flip-coin-2-Output.tgz"}},{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1354333128/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.790Z" level=info msg="Updating node single-condition-pipeline-dbznb-1354333128 status Pending -> Succeeded" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.791Z" level=info msg="All of node single-condition-pipeline-dbznb.condition-1.print-msg-2 dependencies [flip-coin-2] completed" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.791Z" level=info msg="Pod node single-condition-pipeline-dbznb-3432968360 initialized Pending" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.822Z" level=info msg="Created pod: single-condition-pipeline-dbznb.condition-1.print-msg-2 (single-condition-pipeline-dbznb-3432968360)" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.857Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=30779541 workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:13.866Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kubeflow/single-condition-pipeline-dbznb-3449745979/labelPodCompleted
time="2021-06-29T14:54:13.866Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kubeflow/single-condition-pipeline-dbznb-1354333128/labelPodCompleted
time="2021-06-29T14:54:23.823Z" level=info msg="Processing workflow" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:23.824Z" level=info msg="Updating node single-condition-pipeline-dbznb-3432968360 message: ContainerCreating" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:23.824Z" level=info msg="Updating node single-condition-pipeline-dbznb-3067881266 exit code 0" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:23.824Z" level=info msg="Setting node single-condition-pipeline-dbznb-3067881266 outputs: {"artifacts":[{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3067881266/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:23.824Z" level=info msg="Updating node single-condition-pipeline-dbznb-3067881266 status Pending -> Succeeded" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:23.883Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=30779712 workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:23.891Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kubeflow/single-condition-pipeline-dbznb-3067881266/labelPodCompleted
time="2021-06-29T14:54:33.920Z" level=info msg="Processing workflow" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.920Z" level=info msg="Updating node single-condition-pipeline-dbznb-3432968360 exit code 0" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.920Z" level=info msg="Setting node single-condition-pipeline-dbznb-3432968360 outputs: {"artifacts":[{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3432968360/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.920Z" level=info msg="Updating node single-condition-pipeline-dbznb-3432968360 status Pending -> Succeeded" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.921Z" level=info msg="Outbound nodes of single-condition-pipeline-dbznb-2442073942 set to [single-condition-pipeline-dbznb-3432968360 single-condition-pipeline-dbznb-3449745979]" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.921Z" level=info msg="node single-condition-pipeline-dbznb-2442073942 phase Running -> Succeeded" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.921Z" level=info msg="node single-condition-pipeline-dbznb-2442073942 finished: 2021-06-29 14:54:33.921751136 +0000 UTC" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.921Z" level=info msg="Checking daemoned children of single-condition-pipeline-dbznb-2442073942" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.953Z" level=info msg="Workflow update successful" namespace=kubeflow phase=Running resourceVersion=30779847 workflow=single-condition-pipeline-dbznb
time="2021-06-29T14:54:33.962Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kubeflow/single-condition-pipeline-dbznb-3432968360/labelPodCompleted
time="2021-06-29T14:54:43.984Z" level=info msg="Processing workflow" namespace=kubeflow workflow=single-condition-pipeline-dbznb
time="2021-06-29T15:05:33.264Z" level=info msg="Setting node single-condition-pipeline-cmn6x-511824490 outputs: {"parameters":[{"name":"flip-coin-Output","value":"heads","valueFrom":{"path":"/tmp/outputs/Output/data"}}],"artifacts":[{"name":"flip-coin-Output","path":"/tmp/outputs/Output/data","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1478846994/flip-coin-Output.tgz"}},{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1478846994/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-cmn6x
time="2021-06-29T15:05:43.308Z" level=info msg="Setting node single-condition-pipeline-cmn6x-2992979395 outputs: {"artifacts":[{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3449745979/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-cmn6x
time="2021-06-29T15:05:43.308Z" level=info msg="Setting node single-condition-pipeline-cmn6x-2284007376 outputs: {"parameters":[{"name":"flip-coin-2-Output","value":"tails","valueFrom":{"path":"/tmp/outputs/Output/data"}}],"artifacts":[{"name":"flip-coin-2-Output","path":"/tmp/outputs/Output/data","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1354333128/flip-coin-2-Output.tgz"}},{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-1354333128/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-cmn6x
time="2021-06-29T15:05:43.308Z" level=info msg="Setting node single-condition-pipeline-cmn6x-2501038730 outputs: {"artifacts":[{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3067881266/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-cmn6x
time="2021-06-29T15:05:53.339Z" level=info msg="Setting node single-condition-pipeline-cmn6x-2976201776 outputs: {"artifacts":[{"name":"main-logs","s3":{"key":"artifacts/single-condition-pipeline-dbznb/2021/06/29/single-condition-pipeline-dbznb-3432968360/main.log"}}]}" namespace=kubeflow workflow=single-condition-pipeline-cmn6x

@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 30, 2021

After some iterations, the minimal reproducing example is like:

import kfp
from kfp import compiler
from kfp import components
from kfp import dsl

flip_coin_op = components.load_component_from_text(
    '''
name: flip
inputs:
  - name: force_flip_result
    type: String
    default: 'heads'
outputs:
  - {name: flip_result, type: String}
implementation:
  container:
    image: alpine
    command: [sh, -c]
    args:
    - |
      mkdir -p "$(dirname $1)"
      echo $0 > $1
    - {inputValue: force_flip_result}
    - {outputPath: flip_result}
'''
)

print_op = components.load_component_from_text(
    '''
name: print
inputs:
  - {name: text, type: String}
implementation:
  container:
    image: alpine
    command: [sh, -c]
    args:
    - 'echo $0'
    - {inputValue: text}
'''
)

# stuck
@dsl.pipeline(name='single-condition-pipeline')
def my_pipeline_4():
    flip1 = flip_coin_op()
    print_op(flip1.output)

    with dsl.Condition(flip1.output == 'heads'):
        print_op(flip1.output)


if __name__ == '__main__':
    pipeline = my_pipeline_4
    compiler.Compiler().compile(pipeline, package_path='condition.yaml')
    # kfp.Client().create_run_from_pipeline_func(
    #     pipeline,
    #     arguments={'force_flip_result': 'heads'},
    #     # mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE
    # )

@Bobgy Bobgy mentioned this issue Jun 30, 2021
Closed
@Bobgy Bobgy changed the title [bug] condition sample stuck in running state after argo v3.1.0 [bug] condition sample stuck in running state with argo v3.1.0 Jun 30, 2021
@Bobgy
Copy link
Contributor Author

Bobgy commented Jun 30, 2021

Created upstream issue in argo after reproducing with a native argo workflow:
argoproj/argo-workflows#6254

@Bobgy
Copy link
Contributor Author

Bobgy commented Jul 3, 2021

I verified the bug has been fixed in argo v3.1.1

@Bobgy Bobgy closed this as completed Jul 3, 2021
@Bobgy Bobgy mentioned this issue Jul 3, 2021
Merged
1 task
@Bobgy Bobgy reopened this Jul 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Bobgy Bobgy

Labels
area/sdk kind/bug priority/p0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: upgrade argo to v3.1.1. Part of #5718 Bobgy/pipelines
2 participants
@Bobgy @google-oss-robot