Add option in DSL compiler to output YAML

[swiftdiaries]

It would be nice to have the generated YAML output in addition to the tarball.
Then we could use the Pipeline CLI to run the Workflow all from the CLI.

Local iteration step would be:
Write Pipeline using DSL -> compile to YAML -> Use Pipeline CLI to run -> Use Pipeline to show run link in UI.

Currently, we can do this via the notebook. We can create a Pipeline and run a Pipeline from the notebook and it creates a run link that the user can click on to view the Workflow status.
It'd be nice to have a similar experience from the CLI.

Current DSL compile step:

import kfp.compiler as compiler
compiler.Compiler().compile(basic_pipeline, __file__+'.tar.gz')

Expected DSL compile step:

import kfp.compiler as compiler
compiler.Compiler().compile(bolts_data_fetch_pipeline, __file__+'.yaml')

This shouldn't be that hard to implement because the DSL compiler currently generates the YAML for the Workflow and then bundles it into a tarball: Code.

[vicaire]

Hi @gaoning777, I thought this was already available. Could you please comment.

[gaoning777]

Current tarball only contains one yaml file, you could simply untar it.

[vicaire]

@gaoning777, is there any disadvantage in supporting this? It seems that it could be a very common request.

@swiftdiaries, if @gaoning777 does not have any objection, feel free to contribute.

/cc @IronPan

[swiftdiaries]

@gaoning777 That is how I'm doing it right now, it feels like a long-winded approach. And also it would be good to gain some visibility into what exactly is going to be run.

@vicaire Thank you for the reply. I'll wait on @gaoning777 's suggestion before proceeding

[gaoning777]

Could we change the CLI to accept the tar ball instead of the yaml? The motivation of the DSL compiler to generate the tar ball file instead of a yaml was to hide the yaml details and also potentially the pipeline might need to add more files in the future as the intermediate yaml.
On the other hand, we do need to keep the interfaces of the DSL compiler output consistent with the CLI input.

[swiftdiaries]

Makes sense. I'll add a subcommand ml pipeline run <tarball>

[gaoning777]

Thanks

[Ark-kun]

Having an option to output .yaml will help some debugging cases and simplify some tests.

[swiftdiaries]

+1
@gaoning777 Would it make sense to have it as optional? So that only when end-users want a YAML file, the DSL gives them one

[gaoning777]

#855