[RFC] Tool Dependencies for CWL

[jmchilton]

Overview

This initial prototype describes a new abstract Hint type (Dependency) for CWL tools to annotate and act on software packages depended on by a tool definition. This is backed by a pluggable framework for "resolving" these dependencies across a wide variety of platforms. These resolution strategies include custom shell scripts for a local infrastructure, environment modules (http://modules.sourceforge.net/), Homebrew (http://brew.sh/), and conda (https://conda.anaconda.org/). While certainly PRs to adapt this to other package managers, such as Debian, would be welcome - these existing ones have the distinct advantage of allowing multiple versions of a package to exist on the same machine and work in a cross platform fashion.

While I believe that the work on custom packages and environment modules are important steps to allow consumers of CWL to leverage it on HPC resources - the conda resolver (and to a lesser extent the Homebrew resolver) is more exciting and more powerful because it allows for automatic installation as well mere resolution. The Mulled project for instance (https://github.com/mulled/mulled) (ping @bgruening) is work to automatically build Docker images from Homebrew and Conda dependency lists as described by this work. CWL tools that describe dependencies should therefore "get Docker for free" while also being readily usable on HPC and other non-Docker platforms using the same stack of dependencies. In addition to this increased reproducibility, Mulled Docker containers are arguably a little more structured and provide greater transparency than arbitrary Dockerfile-based containers - due to the declarative, structured nature of their construction.

This work is backed by the galaxy-lib project that makes various Galaxy functionality available as a Python library - in particular this work reuses the galaxy.tools.deps package within cwltool. The implementation could use a lot of work for sure - but I want to get a sense for what the path forward is for this.

• No thanks - if you want to submit patches that make forking cwltool a little easier feel free...
• Cool, this belongs in the reference implementation and here is a long list of things you can do to clean up the implementation...
• Cool, lets get it in the reference implementation and the hint definition should probably be described in the specification in some way, here is an outline of how to add to a future draft...

How To Use (Conda Example)

Add dependencies as hints:

hints
 - type: Dependency
   name: bwa
   version: 0.7.13

or just for a unversioned dependency:

hints
 - type: Dependency
   name: bwa

Create a dependency resolvers configuration file - sets call it /path/tool/deps/conda_resolvers_conf.yml - for example the following one will work for conda and enable advanced features such as automatic installation of conda, automatic installation of tools, and inexact version resolution if the exact version cannot be found. This file can be XML or JSON as well... but who cares...

- type: conda
  auto_init: True
  auto_install: True
- type: conda
  versionless: True
  auto_init: True
  auto_install: True

Install galaxy-lib in cwltool's environment - e.g. pip install cwltool. The only new dependency this should introduce besides galaxy-lib is six I think.

Call cwltool with a --dependency-resolvers-configuration argument.

cwltool --dependency-resolvers-configuration /path/tool/deps/conda_resolvers_conf.yml [tool] [job]

Execution of this tool should now install conda, install bwa from bioconda, and setup a conda environment just or that job with that precise version of bwa (or combination of dependencies if multiple ones are supplied), and then run the tool with that environment sourced.

Implementation

In addition to the routine plumbing of adding a command-line argument and passing the Galaxy object around - this reworked job.py to optionally run shell commands to setup an environment before actually calling Popen in another process. This is probably incorrect in some ways - I'll keep working on testing and such if there is interest in this.

Galaxy code is imported conditionally so galaxy-lib remains an optional dependency.

[mr-c]

Hey @jmchilton , thanks for opening this PR and restarting this conversation!

1. You know I'm for the general idea, yep!
2. Can galaxy.tools.deps.requirements.ToolRequirement be spun out of galaxylib into its own lib?
3. I think adding an optional top-level version field to CommandLineTool should be enough. Everything else can be extracted from the baseCommand.

P.S.: environment modules aren't typically shared between sites; package names aren't standardized at all; and they rarely work at other sites let alone cross platform :-)

[chapmanb]

This is perfect. I'd love to switch bcbio CWL usage over to use this plus mulled to create custom Docker containers for each step instead of having one giant container with everything. No comments other than I can't wait to use this.

[bgruening]

👍 from me as well. This and the embedded interpreter in the spec is imho the largest weakness of CWL. So I'm really in favour of this change. Especially if we can share code between Galaxy and CWL.

As a side note to mulled: Mulled container names are predictable so we can use the normal name-version spec. For every other container than mulled, we probably need to have an image name.

[tetron]

I'm very much in favor of incorporating a feature like this, and if there is an existing, well developed solution from Galaxy we can base it on, all the better. I'll respond on technical specifics once I've had some time to completely digest the proposal.

[sjackman]

the conda resolver (and to a lesser extent the Homebrew resolver)

I'm a developer on Homebrew-Science and Linuxbrew. What features does Conda have that Homebrew is missing? No animosity intended, I'm just curious. Sounds like a great idea. I usually implement my pipelines in Make and always have a make install-deps rule that will brew install all the necessary dependencies, which works on both Mac and Linux.

[tetron]

My (possibly uninformed) understanding is that conda distributes binary packages, while brew builds from source?

More to the point, this proposal shouldn't try to get the community to settle on a particular packaging system, because the ecosystem is far too diverse.

[jmchilton]

@sjackman The conda dependency resolver implementation here supports options that are not yet implemented with the homebrew resolver implementation (installing tools on demand instead of just using existing installs for instance). Certainly there are nods and hooks in the implementation and pull requests would be very much welcome to bring the functionality on par - the use of this library means such PRs would benefit cwltool, Galaxy, and Homebrew-science - it would be fantastic to see. But ultimately, the increased maturity of the conda resolver here is a reflection of factors that are both technical and social in my opinion.

• A technical aspect, which feeds into a social one, is that Homebrew doesn't support installing old versions software. I put a lot of effort into re-enabling this as Homebrew removed the versions functionality - some of my efforts and concerns are laid out here (https://github.com/Homebrew/homebrew-science/issues/1191) but I can point out other efforts. Ultimately, though the Homebrew science community I think decided they were uninterested in maintaining older versions of software - so there wasn't going to be a cross talk between Galaxy (and other platforms who care about this kind of re-computability over time) the way that bioconda has allowed.
• The social aspect is that the Galaxy community is moving full steam ahead with conda over Homebrew - it has become an indispensable resource in our community and it is going to be the basis for "best practices" dependency management going forward. This, I think, is driven by the versioned "problem" as well as the relative ease of the binary distributions with conda (though your recent work on bottles for Linuxbrew is awesome and I think largely negates this limitation), and finally I think this is because people in the Galaxy community are more comfortable with the base dependencies required to use conda versus Homebrew (no need for a Ruby stack for instance).

I am fully committed to merging and commenting on the Homebrew resolver and seeing its functionality increased if there is community demand and work on it - but I just cannot justify actively developing it myself when most of the Galaxy community is more excited about conda.

[jmchilton]

@mr-c

I think adding an optional top-level version field to CommandLineTool should be enough. Everything else can be extracted from the baseCommand.

I don't think that is true - many tools or at least some tools - if they are going to play well in Galaxy-land at least - are going to depend on multiple underlying packages. For instance, nearly every Galaxy mapper tool uses samtools to sort output BAM files are the initial command. Your own (formerish) khmer tools for instance have multiple dependencies - https://github.com/galaxyproject/tools-iuc/blob/master/tools/khmer/macros.xml. If we say that CWL will never be the implementation for stuff like this - then I cannot really make the argument that CWL will someday replace Galaxy tools.
I have argued from the beginning that our focus needs to be on creating useful "boxes" not modelling the exact CLI commands - I stand by that. I have an agenda that is different than your agenda - but hopefully we can make both work with the same standard.

environment modules aren't typically shared between sites; package names aren't standardized at all; and they rarely work at other sites let alone cross platform :-)

As a... lets call it dependency injection technique it works on CentOS derived things, Debian derived things, FreeBSD derived things. It is far more cross platform than .deb stuffs. Surely, it will take some time and effort to adapt but if you are an institution with environment modules - you probably have the software professionals capable of doing this.

There is work that could be done - and will be done - to make this more easy within galaxy-lib - namely allowing site-specific remapping of requirement names and versions. We have more of a blue-sky scenario to get it right with CWL but even in the Galaxy landscape this technique has proven feasible (ping @pvanheus has deployed tools this way I believe for instance).

Can galaxy.tools.deps.requirements.ToolRequirement be spun out of galaxylib into its own lib?

Discussed on gitter - it can be done if needed - I can't think of a technical reason to do it however.

[sjackman]

Thanks for the explanation, John. I myself would like to see support for installing a older versions of software. You're correct that there's not much interest in that feature by the upstream Homebrew developers, but it would be a really useful feature for Homebrew-Science. The installation of older versions is pretty easy if there'e a binary bottle available for that version:

curl -LO http://linuxbrew.bintray.com/bottles-science/bwa-0.7.13.x86_64_linux.bottle.tar.gz
brew install ./bwa-0.7.13.x86_64_linux.bottle.tar.gz

It would only need a dash of syntactic sugar to expand brew install bwa=0.7.13 into the above command. See all the Linuxbrew Science bottles on Bintray. If no binary bottle is available, it's not quite as easy. I have thoughts on how to handle that case as well. My idea was to tag each GitHub commit with the formula name and version. See https://github.com/Homebrew/homebrew-science/issues/1191#issuecomment-56240595

[pvanheus]

Yep @jmchilton we have a historic environment modules installation that is slowly being migrated to Galaxy dependency manager naming compatibility. Its really not a hard thing to do. We have a newer conda environment and have experimented with manually installing conda packages so that Galaxy detects them. Our cluster is a multi-interface environment: sometimes things are run via Galaxy, other times via traditional scripts. If CWL can emulate Galaxy's dependency manager setup it would make deploying the CWL runner SO much easier!

[mr-c]

This is a great discussion and I am very excited about the broad agreement on the design and usefulness of this proposal; thanks all!

Here are some specific thoughts from me:

1. Where to put the name of the tool dependency?
   In CWL we only execute a single tool within each tool description [Edit: this is wrong, see this]. Therefore in the interests of "Don't Repeat Yourself" I think extracting the tool name from the baseCommand is the kindest choice to make for those writing CWL descriptions. For the case of naming conflicts we can set the id to an IRI to disambiguate. This could be the bio.tools unique identifier, a Debian identifer, or any other namespace.

   Example: snap, in Debian we had to rename it to snap-aligner due to name conflicts which makes it a good candidate for explicit identification.

   cwlVersion: cwl:draft-3
   class: CommandLineTool
   id: http://packages.qa.debian.org/snap-aligner
   baseCommand: [snap-aligner]
   inputs: []
   outputs: []

2. Wait, what about tools with co-dependencies? Fair enough, some programs run differently or have expanded functionality when other programs or libraries are co-installed.

   In this instance only I agree that additional information in the CWL tool description is warranted.

   This should be stronger than a hint, so I propose that we add an optional field with to CommandLineTool that contains an array of IRIs to resources that enhance the program specified by the baseCommand or id fields.

   Example: The Mira assembler has an optional 12s rRNA database that enhances its ability to detect rRNA & rDNA. While the name mira is fairly unique, we specify an ID out of an abundance of caution.

   cwlVersion: cwl:draft-3
   class: CommandLineTool
   id: http://packages.qa.debian.org/mira
   with: http://packages.debian.org/mira-rfam-12s-rrna
   baseCommand: [mira]
   inputs: []
   outputs: []
   

   The direct dependencies of a tool, like the Python screed library for k-h-mer, have no business being listed in the tool description. That information is already known to packaging systems.

3. What about versions?

   Versioning is hard, there is not an agreed upon way to evaluate version numbers, despite semver.org.

   For example, Debian has a nice standard for comparing version number but if upstream does their own thing Debian maintainers rewrite upstream's numbers to Debian's standard with an embedded copy of the original version. Obviously other packaging systems take different approaches.

   Example: The current version of snap is 1.0beta.18. Debian's version is 0.18~1.0~beta.18+dfsg1-3: this sorts before 1.0, contains all the components of the original version, and notes that Debian modified the source tarball once (+dfsg1) and this is the third release of this particular version in Debian (-3).

   This is too Debian specific to include in the CWL standard.

   For runtime control, the job input object requirements should be expanded to specify required version numbers or other platform specific hints.

   Example job input document syntax:

   cwl:requirements:
   - ToolVersionRequirements: { https://bio.tools/tool/ucdavis.edu/khmer: 2.0 }

   For repeatability purposes the output of the execution on a CWL tool or CWL workflow should contain the exact versions run as part of provenance reporting. This will be standardized post-1.0 as part of the RO+CWL bundle project. In the meantime, implementors are encouraged and allowed to add any metadata using any vocabulary to the output object.

[chapmanb]

Michael -- I use multiple independent tools within each CWL step for bcbio so would appreciate a way to be able to manually specify what is used in a step instead of/in addition to trying to define it from baseCommand. I agnostic about using hints versus explicit definitions but wanted to chime up for the non-just-one-tool-per-step crowd.

[mr-c]

@chapmanb Thank you for sharing your user story. My understanding is that any installation of bcbio would by necessity pull in all dependencies. Is that correct?

[bgruening]

@mr-c package names are often prefixed, suffixed or heavily modified so I think we need to annotate it independently from the baseCommand.
Moreover, I second Brads comment. From experience I can say that it is unrealistic to only have one tool per wrapper if you care about user-friendly tools. For example a very common idiom is to modify the output of a tool to stick to standards.

[mr-c]

@bgruening Yep, in case of ambiguity or unparsability the ID field can be used.

A key difference between CWL and Galaxy tool definitions is that we only run one command within a single CWL tool description.

The common Galaxy idiom of many commands is not possible in CWL without a separate wrapper. This was done on purpose and I think it is the source of some confusion in this discussion.

That command may be a script with its own dependencies, but you can't bowtie-build foo; bowtie bar without wrapping that in a separate script which would need to be packaged in some way (Docker, conda, ...). At that point you're already doing packaging so I see no reason to include dependency management beyond tool ID and any optional enhancements.

[bgruening]

@bgruening Yep, in case of ambiguity or unparsability the ID field can be used.

So why not have a more explicit annotation in the first place. I'm still a fan of explicit over implicit and CWL is already so verbose that this probably does not matter, isn't it?
I expect that more tools will have a different package name than the baseCommand as vice versa.

That command may be a script with its own dependencies, but you can't bowtie-build foo; bowtie bar without wrapping that in a separate script which would need to be packaged in some way (Docker, conda, ...). At that point you're already doing packaging so I see no reason to include dependency management beyond tool ID and any optional enhancements.

So if I write such a script to call bowtie-build foo; bowtie bar; magic.py baz I need to create a separate conda package instead of simply specifying bowite=2.2 and python-magic=0.1? This would be possible, conda supports meta-packages but means more work.

[mr-c]

@bgruening Yep, in case of ambiguity or unparsability the ID field can be used.

So why not have a more explicit annotation in the first place. I'm still a fan of explicit over implicit and CWL is already so verbose that this probably does not matter, isn't it?
I expect that more tools will have a different package name than the baseCommand as vice versa.

I agree that explicit is better, and I bet that specifying a tool ID will be apart of various best practice guides, planemo lint, etcetera :-)

So if I write such a script to call bowtie-build foo; bowtie bar; magic.py baz I need to create a separate conda package instead of simply specifying bowite=2.2 and python-magic=0.1? This would be possible, conda supports meta-packages but means more work.

Right, CWL doesn't aim to solve tool delivery. Either the program is already installed, the user installs it themselves (and tells the CWL implementation about it by specifying cwl:requirements: EnvVarRequirement: { PATH: /home/username/bin/ } in the input job object) or the CWL implementation uses the information available (baseCommand, id, with, DockerRequirement) to setup a suitable runtime environment.

Here is a summary of my current thinking. Thank you @bgruening @chapmanb @jmchilton for challenging me :-)

1. Use the existing id field to specify tool identity using bio.tools, or another namespace. Platforms can then use those IDs to install a conda package, setup a new docker image, et cetera. Note: if bio.tools URLs don't contain the information you need please talk to me and them: they are quite happy to add extra fields! Alternatively we can annotate out packages with the unique IDs. I imagine that there will be a bit of both.

2. Add a with field to the CommandLineTool and Workflow CWL spec for tools & libraries that aren't strictly implied by the id field.

3. Add ToolVersionRequirements to the CWL spec for use in job input documents and elsewhere. Behavior beyond pure string matching of version numbers is undefined.

Thoughts?

[sjackman]

How is id and baseCommand mapped to the name of the Debian/Bioconda/Homebrew package name that provides that tool?

[mr-c]

Hello all,

@jmchilton and I had a phone chat, here is a summary:

1. This is a great proposal and everyone is welcome to assist @jmchilton in implementing it, but until it becomes part of the standard it needs to live behind a flag as a hint that this behavior isn't portable yet. So no need to fork cwltool, development can happen here.
2. Speaking of becoming part of the CWL standard, the deadline for 1.0 is tomorrow, which means this is the perfect time to mature this feature for the 1.1 release!
3. CWL absolutely supports "thick" tools; I had completely forgotten about the ShellCommandRequirement! Mea culpa. I edited my reply above to reflect that.
4. Now that we have more breathing room, the community can decide: do we assign one ID to a tool and map that to the different packaging systems or do we have a long, long list of IDs in every tool definition?

@sjackman If the packaging system has its own mappings for identifiers to package names then those can be used. For example, the Debian Med team is integrating the EDAM & bio.tools identifiers into their infrastructure. Alternatively the CWL implementation can maintain their own mappings, or users could provide these mappings in their job input objects. This is likely useful for systems that have no standardization (environment modules). Alternatively the packaging system can cross reference its own index of "binary" application names against the baseCommand

[jmchilton]

do we have a long, long list of IDs in every tool definition?

This remains a false dichotomy and a misstatement of my position. The proposed solution works fine in 95% of cases - it should surprise no one to learn that Homebrew, Conda, MSI's enviornment modules, Debian, and Nix all call bwa... bwa. Small variations will of course pop up - but those can be solved with site or package-manager specific mappings on the backend - small tweaks - there is no need for long lists of IDs in tools. If you are using environment modules - you may need to map snap to snap-aligner locally - for Debian we can even build community mappings.

I'm arguing for one ID per component not long lists. If you feel it needs a namespace fine - if you feel some service needs to do the mapping - I'm fine with that too - go work with bio.tools on that and this can leverage that work. But it should be a goodie, an add-on, a best practice - I don't think CWL tools should require registration of their components in biotools to work - it should just work. This just works.

You are imaging a potential system for these things - Galaxy is a real and existing system that supports multiple package managers today and this problem hasn't really materialized - and we have concrete ideas for how to address it in a lightweight way.

[tetron]

Obligitory xkcd https://xkcd.com/1654/

[jmchilton]

I have prefixed the two command-line options with beta- to make it clear they are beta flags and may change in the future. I have also suppressed them from cwltool's help - I'm fine strictly making use of them as a library through this summer conference season.