List available pipelines

[ajkerrigan]

I'm looking at pypyr as a substitute for some makefile / justfile usage, and one flow I'm trying to sort out is writing pipelines and exposing them to folks in a demo/education context. Given a set of YAML files that define some number of pipelines, is there a way to list available pipelines from the CLI without having to dig into the YAML?

[yaythomas]

hi @ajkerrigan, great seeing you here!

The short answer is, not really. The longer answer is, it depends.

Without being too much of a comedian, but $ ls mypipelib/*.yaml or $ find ./pipelines/*.yaml -printf "%f\n" is the way I personally do it, but I'm sure you got there yourself already!

The longer reason is:

1. since a pipeline is just a normal, parse-able yaml file with a .yaml extension, per the yaml.org spec, there is actually no ready way of differentiating some-input-config-data.yaml from pipeline-that-works-with-input-config-data.yaml.
2. pipelines can be anywhere. you can have, say,

• $ pypyr mydir/mypipe (relative to current dir)
• $ pypyr anotherdir/anotherpipe (another dir relative to current)
• $ pypyr ~mydir/mypipe (relative to $HOME)
• $ pypyr /mydir/mypipe(absolute path - the windows equivalent would be $ pypyr C:/mydir/mypipe).

So the pypyr runtime doesn't actually have any way of knowing where on your system you might have saved your pipelines, unless you tell it somehow. And the way you tell it is by providing the input.

Now, that said, I normally establish a convention whereby I order my repos and their pipelines in such a way that they are reliably in one place. This is a matter of discipline - pypyr could still of course execute a pipeline anywhere you point it at... When pypyr is doing makefile style build/package functions. . .I tend to put everything pipeline in ./ops.

This lead to relatively "natural language" when running your pipelines/cmds, like so:

# build (lint + test)
$ pypyr ops/build

# build and then package
$ pypyr ops/build package

# build, then package, then publish the package
$ pypyr ops/build package publish

# bump minor version
$ pypyr ops/bump minor

# bump major version
$ pypyr ops/bump major

So in this case, the way you answer "which pipelines are available", is pretty much $ ls ops/*.yaml

Here is an (inadequately documented) example of how I generally go about building a python project following this pattern: https://github.com/pypyr/python-ci-cd-example

If part of your educational content is to provide a pipeline to show which pipelines you have, then you could do something like this:

# ./pipes-list.yaml
steps:
  - name: pypyr.steps.py
    in:
      py: |
        from pathlib import Path
        print(*[f.stem for f in Path.cwd().glob('*.yaml') if f.is_file()], sep='\n')

Which runs like $ pypyr pipes-list. How and if this works is going to be dependent on your taxonomy/directory layout - I just used cwd here for the sake of an easy example.

Speaking more to your underlying objective (using pypyr in makefile-ish sort of ways):
here is a simple way (pick and choose which steps to run based on input args):
https://pypyr.io/docs/getting-started/conditional-logic/#use-cli-arguments-to-run-step-conditionally

or another more flexible way (pick and choose grouped steps/functionality to run based on dynamic input args):
https://pypyr.io/docs/getting-started/conditional-logic/#multiple-steps-with-the-same-condition

Hope this helps!

[blaisep]

So the pypyr runtime doesn't actually have any way of knowing where on your system you might have saved your pipelines, unless you tell it somehow. And the way you tell it is by providing the input.

does the runtime create an abstract (or maybe concrete is better) syntax tree that might have a list of symbols in sequence....?

[yaythomas]

It does, but it does so as it runs. If you're looking to do the equivalent of a Python AST operation (i.e parse the pipeline without running it), no that's not exposed externally. There are reasons - mostly that pypyr interprets pipelines and the context inputs for each step dynamically as it goes, and you don't have any way of knowing what's in context without actually running the pipeline.

You can do something of the sort if you're keen, though, like this:

from pathlib import Path
from pypyr.dsl import Step
import pypyr.loaders.file as pypyr_file_loader

pipe_def = pypyr_file_loader.get_pipeline_definition("pipeline_name", Path.cwd())

out = {}

for group_name, group in pipe_def.pipeline.items():
    steps_in_group = []

    for step in group:
        steps_in_group.append(Step(step))

    out[group_name] = steps_in_group    

print(out)

This will give you a list for each step-group, containing a pypyr.dsl.Step instance for each step. So this will give you the pipeline structure and all the properties for each step.

BUT remember this does not resolve whatever is in context.

So Imagine a pipeline like this:

steps:
  - name: pypyr.steps.cmd
    comment: this cmd should return 0 if podman api-server already running
    swallow: True
    in:
      cmd: echo some sort of cmd to check if api-server is running

So you can for example see what the pipeline is passing as in arguments to the step with something like:

>>> out['steps'][0].name
pypyr.steps.cmd

>>> out['steps'][0].in_parameters
ordereddict([('cmd', 'echo some sort of cmd to check if api-server is running')]

>>> out['steps'][0].swallow_me
True

You'll see it gives you what the in args are, but it cannot know at this point what actually would be in the context without actually running the pipeline (e.g if there were {substitution expressions} that got their values from other steps in the pipeline).

[yaythomas]

does the runtime create an abstract (or maybe concrete is better) syntax tree that might have a list of symbols in sequence....?

@blaisep

Oct 2023 update: experimenting with introducing a Python API so you can create pipelines in code. It also parses input yaml into a syntax tree when first loading, so validation happens before or separately to the pipeline running.

Details on new feature here: #333

Please feel free to comment there and give some feedback!

[yaythomas]

oh, I should probably add there is also the pipelines dir that pypyr looks in automatically as part of the discovery/lookup sequence. . . https://pypyr.io/docs/pipelines/lookup-order/#directory-locations-lookup-order

So depending on what make sense for your repo/content taxonomy, you could collect your pipelines under a directory called pipelines:

|-rootdir/
  - pipelines/
    - mydir/
      - mypipe.yaml
    - another-pipe.yaml

Which can run like:

$ cd rootdir
$ pypyr mydir/mypipe
$ pypyr another-pipe

In this case the answer to "what pipelines are there" is ls pipelines/*.yaml.

[ajkerrigan]

Hey thanks @yaythomas ! That's exactly the sort of info I was looking for :). I didn't want to go writing custom scripts or pipelines to do this if there was a native "right way" I hadn't found yet. Your explanations make sense, cheers! 🙇