Skip to content

Commit

Permalink
Merge branch 'master' into galaxy_deps
Browse files Browse the repository at this point in the history
mr-c authored Jul 7, 2017

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
2 parents 5522e88 + 6e01be1 commit fd2ac01
Showing 16 changed files with 182 additions and 15 deletions.
12 changes: 5 additions & 7 deletions cwltool/schemas/README.md
Original file line number Diff line number Diff line change
@@ -3,18 +3,20 @@ Common Workflow Language

[![GitHub stars](https://img.shields.io/github/stars/common-workflow-language/common-workflow-language.svg)](https://github.com/common-workflow-language/common-workflow-language/stargazers) [![Gitter](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/common-workflow-language/common-workflow-language?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [**Support**](#Support)

<a href="https://www.youtube.com/watch?v=86eY8xs-Vo8"><img align="right" src="https://github.com/common-workflow-language/logo/raw/master/intro_video_screenshot_413x193.png" alt="[Video] Common Workflow Language explained in 64 seconds"></a>
The Common Workflow Language (CWL) is a specification for describing analysis
workflows and tools in a way that makes them portable and scalable across a
variety of software and hardware environments, from workstations to cluster,
cloud, and high performance computing (HPC) environments. CWL is designed to
meet the needs of data-intensive science, such as Bioinformatics, Medical
Imaging, Astronomy, Physics, and Chemistry.

<a href="https://open-stand.org/about-us/principles"><img align="left" src="http://standards.ieee.org/images/openstand/128x128-blue2.png" alt="Open Stand badge"></a>
CWL is developed by an informal, multi-vendor working group consisting of
organizations and individuals aiming to enable scientists to share data
analysis workflows. [The CWL project is on Github](https://github.com/common-workflow-language/common-workflow-language) and we follow the [Open-Stand.org principles for collaborative open standards development](https://open-stand.org/about-us/principles/)

[![Open Stand badge](http://standards.ieee.org/images/openstand/128x128-blue2.png)](https://open-stand.org/about-us/principles/).


CWL builds on technologies such as [JSON-LD](http://json-ld.org)
for data modeling and [Docker](http://docker.com) for portable runtime
@@ -30,12 +32,7 @@ descriptions.

For developers and advanced users, the current
[CWL specification (v1.0)](http://www.commonwl.org/v1.0/) provides
authoritative documentation of the execution of CWL documents. Links older
drafts:
[draft-1](https://github.com/common-workflow-language/common-workflow-language/tree/master/draft-1),
[draft-2](http://www.commonwl.org/draft-2/),
[draft-3](http://www.commonwl.org/draft-3/)

authoritative documentation of the execution of CWL documents.
### Citation

Please cite https://dx.doi.org/10.6084/m9.figshare.3115156.v2
@@ -137,6 +134,7 @@ Conduct](https://github.com/common-workflow-language/common-workflow-language/bl
* [Institute for Systems Biology](https://www.systemsbiology.org/)
* [ELIXIR Europe](https://www.elixir-europe.org/)
* [BioExcel CoE](http://bioexcel.eu/)
* [BD2K](https://commonfund.nih.gov/bd2k)

## Individual Contributors

53 changes: 53 additions & 0 deletions cwltool/schemas/doc/cli-description-languages-comparison.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Comparison of Command Line Interface description languages

## Compared languages

* Galaxy v17.05: https://github.com/galaxyproject/galaxy/blob/release_17.05/lib/galaxy/tools/xsd/galaxy.xsd
* CTD v1.7.0 : https://github.com/WorkflowConversion/CTDSchema/blob/Version_1_7_0/CTD.xsd
* CWL v1.0: http://www.commonwl.org/v1.0/CommandLineTool.html / https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/CommandLineTool.yml

## Summary

### Tool description
CTD reflects the perspective of a tool author targetting the KNIME processing unit approach ("nodes"), whereas
Galaxy reflects the perspective of a platform centered around non-developer users interacting with a graphical interface.

### Parameters description

### Data/parameter types
The typing system for parameters in CTD itself is simpler than the other two, providing mainly support for simple types and input/output files, but excluding arrays/lists and complex types. Galaxy adds GUI specific features (`select`, `drill_down`), domain specific features (`color`, `genome_build`, `data_column`), configuration of web services (`base_url`), and tighter integration with workbench internal data management (`library_data`).

### Command line generation
All of the specifications include a flexible set of possibilities for the generation of command lines, reflecting the effective heterogeneity of the bioinformatics tools ecosystem ;)

## Comparison table

### Tool level

|Galaxy information |Optional |CTD information |Optional |CWL information |Optional |
|---|---|---|---|---|---|
|*description* |X |description |X |label |X |
|*help* |X |*manual* |X |*doc* |X |
|*citation* |X |*citation* (for the underlying tool, DOI or doc URL) |X |*SoftwareRequirement.name.specs* (URI) |X |
| | |*ExecutableName* (overrides the name attribute) |X |*SoftwareRequirement.name.baseCommand[0]* |X |
| | |*ExecutablePath* (specific path to the executable) |X | | |
|*command*+*argument* under parameter | |*CLItype* list of input commands with elements mapping |X | *arguments* and/or *inputBindings* |X |
| | |*logs* (retrospective on execution information - if so probably out of scope)|X | | |
|*tool/outputs/data/@from_work_dir* (name of file to consume) |X |*relocators* (list of parameters, path pairs)|X |*outputBinding* |X |
|*@version* (version of wrapper) |X |*@version* (apprarently version of the tool itself)| |*SoftwareRequirements.name.version* (list of known compatible versions of the underlying tool |X |
|*@name* (just an identifier, not used to build the command itself)||*@name* (also used to build the command itself, unlesss *ExecutableName* is provided)| |baseCommand[0]|X|
|often part of the *help* text|X|*docURL* (URL to documentation)||||
|*edam_topics* and *edam_operations* - categories are external to tool definition, in per-server config (toolconf.xml)|X|*category* (any string)|X|can use EDAM or other 3rd party annotation or derive via identifier from external registry or datatype|
|(often part of help text)|X|*tags*|X|||

### Parameter level

**Important remark** The structure of the CTD file is slightly different, because it explicitely allows the definition of "sub-tools" through a nested structure of NODE/ITEM elements. A node is a subgroup of parameters, and ITEM is a parameter.
Therefore, to focus on Parameter-level information, we consider only the ITEM element here for parameter description in CTD.

|Galaxy information |Optional |CTD information |Optional |CWL information |Optional |
|---|---|---|---|---|---|
| *param* or *data* | | *ITEM* | | *inputs[n]* or *outputs[n]* | |
|*@name*||*@name*||key of *inp|uts* or *outputs* entry||
|*@type*||*@type* (Defines the possible types available in the type attribute of ITEM and ITEMLIST.)||type||

5 changes: 3 additions & 2 deletions cwltool/schemas/v1.0/CommandLineTool.yml
Original file line number Diff line number Diff line change
@@ -36,10 +36,11 @@ $graph:
- {$include: intro.md}

- |
## Introduction to v1.0
## Introduction to CWL Command Line Tool standard v1.0
This specification represents the first full release from the CWL group.
Since draft-3, version 1.0 introduces the following changes and additions:
Since draft-3, version 1.0 introduces the following changes and additions
to the CWL Command Line Tool standard:
* The [Directory](#Directory) type.
* Syntax simplifcations: denoted by the `map<>` syntax. Example: inputs
4 changes: 3 additions & 1 deletion cwltool/schemas/v1.0/Process.yml
Original file line number Diff line number Diff line change
@@ -563,7 +563,9 @@ $graph:
- name: default
type: Any?
jsonldPredicate: "cwl:default"
jsonldPredicate:
_id: cwl:default
noLinkCheck: true
doc: |
The default value for this parameter if not provided in the input
object.
9 changes: 6 additions & 3 deletions cwltool/schemas/v1.0/Workflow.yml
Original file line number Diff line number Diff line change
@@ -34,10 +34,11 @@ $graph:

- |
## Introduction to v1.0
## Introduction to CWL Workflow standard v1.0
This specification represents the first full release from the CWL group.
Since draft-3, this draft introduces the following changes and additions:
Since draft-3, this draft introduces the following changes and additions
to the CWL Workflow standard:
* The `inputs` and `outputs` fields have been renamed `in` and `out`.
* Syntax simplifcations: denoted by the `map<>` syntax. Example: `in`
@@ -260,7 +261,9 @@ $graph:
doc: |
The default value for this parameter if there is no `source`
field.
jsonldPredicate: "cwl:default"
jsonldPredicate:
_id: "cwl:default"
noLinkCheck: true
- name: valueFrom
type:
- "null"
11 changes: 11 additions & 0 deletions cwltool/schemas/v1.0/conformance_test_v1.0.yaml
Original file line number Diff line number Diff line change
@@ -981,3 +981,14 @@
size: 15
tool: v1.0/imported-hint.cwl
doc: Test hints with $import

- output: {}
job: v1.0/default_path_job.yml
tool: v1.0/default_path.cwl
doc: Test warning instead of error when default path is not found

- output:
args: [-A,'2',-B,baz,-C,'10','9','8','7','6','5','4','3','2','1']
job: v1.0/empty.json
tool: v1.0/inline-js.cwl
doc: Test InlineJavascriptRequirement with multiple expressions in the same tool
10 changes: 10 additions & 0 deletions cwltool/schemas/v1.0/v1.0/default_path.cwl
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
cwlVersion: v1.0
class: CommandLineTool
inputs:
- id: "file1"
type: File
default:
class: File
path: default.txt
outputs: []
arguments: [cat,$(inputs.file1.path)]
3 changes: 3 additions & 0 deletions cwltool/schemas/v1.0/v1.0/default_path_job.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
file1:
class: File
path: hello.txt
38 changes: 38 additions & 0 deletions cwltool/schemas/v1.0/v1.0/inline-js.cwl
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo

requirements:
- class: InlineJavascriptRequirement

inputs:
- id: args.py
type: File
default:
class: File
location: args.py
inputBinding:
position: -1

outputs:
- id: args
type:
type: array
items: string

baseCommand: python

arguments:
- prefix: -A
valueFrom: $(1+1)
- prefix: -B
valueFrom: $("/foo/bar/baz".split('/').slice(-1)[0])
- prefix: -C
valueFrom: |
${
var r = [];
for (var i = 10; i >= 1; i--) {
r.push(i);
}
return r;
}
4 changes: 3 additions & 1 deletion cwltool/schemas/v1.1.0-dev1/Process.yml
Original file line number Diff line number Diff line change
@@ -564,7 +564,9 @@ $graph:
- name: default
type: Any?
jsonldPredicate: "cwl:default"
jsonldPredicate:
_id: cwl:default
noLinkCheck: true
doc: |
The default value for this parameter if not provided in the input
object.
4 changes: 3 additions & 1 deletion cwltool/schemas/v1.1.0-dev1/Workflow.yml
Original file line number Diff line number Diff line change
@@ -267,7 +267,9 @@ $graph:
doc: |
The default value for this parameter if there is no `source`
field.
jsonldPredicate: "cwl:default"
jsonldPredicate:
_id: "cwl:default"
noLinkCheck: true
- name: valueFrom
type:
- "null"
5 changes: 5 additions & 0 deletions cwltool/schemas/v1.1.0-dev1/conformance_test_v1.1.0-dev1.yaml
Original file line number Diff line number Diff line change
@@ -992,3 +992,8 @@
size: 15
tool: v1.1.0-dev1/imported-hint.cwl
doc: Test hints with $import

- output: {}
job: v1.0/default_path_job.yml
tool: v1.0/default_path.cwl
doc: Test warning instead of error when default path is not found
10 changes: 10 additions & 0 deletions cwltool/schemas/v1.1.0-dev1/v1.1.0-dev1/default_path.cwl
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
cwlVersion: v1.0
class: CommandLineTool
inputs:
- id: "file1"
type: File
default:
class: File
path: default.txt
outputs: []
arguments: [cat,$(inputs.file1.path)]
3 changes: 3 additions & 0 deletions cwltool/schemas/v1.1.0-dev1/v1.1.0-dev1/default_path_job.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
file1:
class: File
path: hello.txt
15 changes: 15 additions & 0 deletions tests/test_default_path.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import unittest
from cwltool.load_tool import fetch_document, validate_document
from .util import get_data
from schema_salad.ref_resolver import Loader

class TestDefaultPath(unittest.TestCase):
# Testing that error is not raised when default path is not present
def test_default_path(self):
document_loader, workflowobj, uri = fetch_document(
get_data("tests/wf/default_path.cwl"))
document_loader, avsc_names, processobj, metadata, uri = validate_document(
document_loader, workflowobj, uri)

self.assertIsInstance(document_loader,Loader)
self.assertIn("cwlVersion",processobj)
11 changes: 11 additions & 0 deletions tests/wf/default_path.cwl
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
cwlVersion: v1.0
class: CommandLineTool
inputs:
- id: "file1"
type: File
default:
class: File
path: default.txt
outputs: []
arguments: [cat, $(inputs.file1.path)]

0 comments on commit fd2ac01

Please sign in to comment.