Cloudpickle fails to deserialize a function

[kvalev]

Deserializing a python function during runtime fails in case the function has a default parameter, which uses an argo variable.

Here is a simple example:

def _test_func(
    path: str,
    local_path: str,
    remote_artifact_name: str,
    minio_bucket: str = 'mlpipeline',
    minio_key: str = 'runs/{{workflow.uid}}/{{pod.name}}',
    minio_ext: str = '.tgz',
):
    remote_path = f'minio://{minio_bucket}/{minio_key}/{remote_artifact_name}{minio_ext}'
    print(remote_path)

This can be executed fine when compiled with 0.1.19 (latest version in PyPI), but fails with the latest master (which uses cloudpickle). The error message is one of the following:

• in case only {{workflow.uid}} is used:

Traceback (most recent call last):
  File "<string>", line 9, in <module>
AttributeError: 'dict' object has no attribute 'append'

• in case {{pod.name}} is used:

Traceback (most recent call last):
  File "<string>", line 9, in <module>
_pickle.UnpicklingError: invalid load key, '5'.

Using a default value without brackets resolves the issue.

[Ark-kun]

Unfortunately this is probably "Won't fix" for now.
Argo's templating language is text-based and it brings all sorts of problems in many scenarios.
For example this command-line fails in Argo:

command: [docker inspect --format="{{index .RepoDigests 0}}" org/image]

You probably imaging what's happening in your case: Argo is replacing some text snipped inside binary data which breaks the binary format.

In your case the workaround is easy and better for future-proofing:

1. Don't use Argo templates inside the program code
2. Don't use Argo templates in component.yaml

You should only pass the templated strings as arguments when you call the component.

Argo templates are implementation detail. You should not expect them to always work.
Artifacts in Minio is an implementation detail. You should not expect them to always be there (e.g. in future we can decide to store artifacts in mounted volumes).

P.S. Why do you want to access the artifacts that would be written by your own task after it ends? They'll never be there until your code finishes...