current working directory vs relative paths

[yaythomas]

pypyr works from $cwd

pypyr currently resolves pipelines and custom modules like steps/context parsers/loaders from the current working directory - which is to say $PWD or wherever pypyr is invoked from.

This is a bit of a legacy feature, that has been overtaken by events.

This way of working makes pipelines less portable than they should be. When you have parent pipelines calling child pipelines via pypyr.steps.pype. It means the child pipeline has to be coded with paths relative to wherever you called the parent from.

As it stands, if you want to resolve from the pipeline directory, you currently have to invoke pypyr like this:

$ pypyr mypipeline --dir path/to/pipelinestuff

or in the case of the api

from pypyr import pipelinerunner
context = pipelinerunner.main_with_context(pipeline='mypipeline', working_dir='path/to/pipelinestuff')

using paths relative to the pipeline instead

I propose to make this a bit easier by making pypyr work relative to the pipeline itself:

• Absolute names for custom modules (custom steps. context parsers, loaders, retry backoff algorithms) will resolve relative to the pipeline path itself.
• Path inputs to any step that uses the file-system (e.g the fetchyaml, fileformat, fileplace and tar steps) will continue to be relative to the cwd/$PWD.

This means you'd be able to do

$ pypyr path/to/pipelinestuff/mypipeline

and then mypipeline will resolve custom modules and child pipeline-names relative to itself - meaning you can put supporting code/pipelines for your pipeline in path/to/pipelinestuff/ and it will work wherever you call it from.

impact

As ever, I don't want to break anything for anyone, so I'm putting out a feeler for thoughts/feedback/concerns. I think this will be very useful for making pipelines more portable, so you can create shared pipeline libraries that you can call from anywhere.

But, this does potentially break existing pipelines, if you are:

Calling the pipeline from a directory that is not its parent WITHOUT setting --dir/working_dir AND in that pipeline (

• Invoking child pipelines with pype
• OR
• Using custom steps)

Your input welcome!

[yaythomas]

Part of this change is likely to involve NOT changing sys.path anymore, which is how pypyr currently loads modules from $PWD without having to install these as packages 1st. Especially when used as a library/API, generally a library probably shouldn't be messing with the system's sys.path.

As it is, the sys.path checking/adding code is probably the area of pypyr I most dislike - the code is necessarily clumsy and doesn't sit well with the single-call API as it exists currently.

For the proposed change, module loading order will be:

1. look in pipeline directory, if available (it might not be available if you use a custom pype loader to load pipelines from elsewhere than the filesystem)
2. look in $PWD (if not the same as pipeline dir)
3. look in the current Python environment's installed packages.

[yaythomas]

See issue #239