Problem in cwl/schema (minimal pip install)

[jxtx]

Installing from pip in a minimal environment, planemo command will not run.

~ ❯❯❯ docker run -i -t python:2.7-slim /bin/bash
root@f9f25348797a:/# pip install planemo
Collecting planemo
  Downloading planemo-0.25.0-py2.py3-none-any.whl (228kB)
    100% |████████████████████████████████| 235kB 3.4MB/s 
Collecting six>=1.7.0 (from planemo)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting Click (from planemo)
  Downloading click-6.6.tar.gz (283kB)
    100% |████████████████████████████████| 286kB 3.0MB/s 
Collecting aenum (from planemo)
  Downloading aenum-1.4.4-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 10.1MB/s 
Collecting virtualenv (from planemo)
  Downloading virtualenv-15.0.1-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 863kB/s 
Collecting jinja2 (from planemo)
  Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB)
    100% |████████████████████████████████| 266kB 3.2MB/s 
Collecting galaxy-lib>=16.7.2 (from planemo)
  Downloading galaxy_lib-16.7.5-py2.py3-none-any.whl (205kB)
    100% |████████████████████████████████| 215kB 216kB/s 
Collecting docutils (from planemo)
  Downloading docutils-0.12.tar.gz (1.6MB)
    100% |████████████████████████████████| 1.6MB 733kB/s 
Collecting glob2 (from planemo)
  Downloading glob2-0.4.1.tar.gz
Collecting pyyaml (from planemo)
  Downloading PyYAML-3.11.zip (371kB)
    100% |████████████████████████████████| 378kB 950kB/s 
Collecting bioblend (from planemo)
  Downloading bioblend-0.7.0.tar.gz (858kB)
    100% |████████████████████████████████| 860kB 934kB/s 
Collecting MarkupSafe (from jinja2->planemo)
  Downloading MarkupSafe-0.23.tar.gz
Collecting requests>=2.4.3 (from bioblend->planemo)
  Downloading requests-2.10.0-py2.py3-none-any.whl (506kB)
    100% |████████████████████████████████| 512kB 1.1MB/s 
Collecting requests-toolbelt (from bioblend->planemo)
  Downloading requests_toolbelt-0.6.2-py2.py3-none-any.whl (49kB)
    100% |████████████████████████████████| 51kB 2.4MB/s 
Collecting boto>=2.9.7 (from bioblend->planemo)
  Downloading boto-2.40.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.4MB 970kB/s 
Building wheels for collected packages: Click, docutils, glob2, pyyaml, bioblend, MarkupSafe
  Running setup.py bdist_wheel for Click ... done
  Stored in directory: /root/.cache/pip/wheels/b0/6d/8c/cf5ca1146e48bc7914748bfb1dbf3a40a440b8b4f4f0d952dd
  Running setup.py bdist_wheel for docutils ... done
  Stored in directory: /root/.cache/pip/wheels/db/de/bd/b99b1e12d321fbc950766c58894c6576b1a73ae3131b29a151
  Running setup.py bdist_wheel for glob2 ... done
  Stored in directory: /root/.cache/pip/wheels/25/5e/d1/742f1a34931b3353d26688b43dfdc017b0d853fe910ef6bae8
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /root/.cache/pip/wheels/4a/bf/14/d79994d19a59d4f73efdafb8682961f582d45ed6b459420346
  Running setup.py bdist_wheel for bioblend ... done
  Stored in directory: /root/.cache/pip/wheels/7a/db/f8/c1be24304ed4114a49a627aa9b0ffc8b1fe45e1f45168701e4
  Running setup.py bdist_wheel for MarkupSafe ... done
  Stored in directory: /root/.cache/pip/wheels/a3/fa/dc/0198eed9ad95489b8a4f45d14dd5d2aee3f8984e46862c5748
Successfully built Click docutils glob2 pyyaml bioblend MarkupSafe
Installing collected packages: six, Click, aenum, virtualenv, MarkupSafe, jinja2, docutils, pyyaml, galaxy-lib, glob2, requests, requests-toolbelt, boto, bioblend, planemo
Successfully installed Click-6.6 MarkupSafe-0.23 aenum-1.4.4 bioblend-0.7.0 boto-2.40.0 docutils-0.12 galaxy-lib-16.7.5 glob2-0.4.1 jinja2-2.8 planemo-0.25.0 pyyaml-3.11 requests-2.10.0 requests-toolbelt-0.6.2 six-1.10.0 virtualenv-15.0.1
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@f9f25348797a:/# planemo 
Traceback (most recent call last):
  File "/usr/local/bin/planemo", line 7, in <module>
    from planemo.cli import planemo
  File "/usr/local/lib/python2.7/site-packages/planemo/cli.py", line 13, in <module>
    from planemo.galaxy import profiles
  File "/usr/local/lib/python2.7/site-packages/planemo/galaxy/__init__.py", line 5, in <module>
    from .config import galaxy_config
  File "/usr/local/lib/python2.7/site-packages/planemo/galaxy/config.py", line 26, in <module>
    from planemo.conda import build_conda_context
  File "/usr/local/lib/python2.7/site-packages/planemo/conda.py", line 10, in <module>
    from galaxy.tools.loader_directory import load_tool_elements_from_path
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/loader_directory.py", line 12, in <module>
    from .parser import get_tool_source
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/parser/__init__.py", line 4, in <module>
    from .factory import get_tool_source
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/parser/factory.py", line 9, in <module>
    from .cwl import CwlToolSource
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/parser/cwl.py", line 14, in <module>
    from galaxy.tools.cwl import tool_proxy
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/cwl/__init__.py", line 1, in <module>
    from .parser import tool_proxy
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/cwl/parser.py", line 23, in <module>
    from .schema import schema_loader
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/cwl/schema.py", line 63, in <module>
    schema_loader = SchemaLoader()
  File "/usr/local/lib/python2.7/site-packages/galaxy/tools/cwl/schema.py", line 21, in __init__
    self._raw_document_loader = schema_salad.ref_resolver.Loader({"cwl": "https://w3id.org/cwl/cwl#", "id": "@id"})
AttributeError: 'NoneType' object has no attribute 'ref_resolver'
root@f9f25348797a:/#

[jmchilton]

A series of problems here -

• I've messed up and galaxy/tools/cwl/schema.py doesn't properly treat cwltool as optional. I'll push a new galaxy-lib out tonight to fix that.
• Currently cwltool requires gcc to install via pip (I've been fighting about that on gitter with @mr-c today) so it cannot even be installed in that container). This is now the third planemo-related problem related to introducing ruamel.yaml as a dependency in schema salad (see also Installation of planemo 0.25.0 via Homebrew fails #473 and this hack https://github.com/galaxyproject/homebrew-tap/blob/master/planemo.rb#L214)
• The syntax extras_requires={':python_version == "2.7"': ['cwltool>=1.0.20160511162129', ],}, seems to have broken - I swear this worked in earlier versions of pip but it doesn't in that container. http://stackoverflow.com/questions/21082091/install-requires-based-on-python-version.

[jmchilton]

The first of my three problems has been resolved - a minimal planemo installs properly in that container now (with newly released planemo 0.25.1). I'm still trying to figure out how to handle cwltool as a dependency. Thanks for the report @jxtx.

[mr-c]

John, shall I contact the ruamel.YAML people about making a manylinux1 wheel?

[mr-c]

Relevant issues https://bitbucket.org/ruamel/yaml/issues/32/cannot-run-setup-script-on-python-2710
https://bitbucket.org/ruamel/yaml/issues/28/consider-making-ruamelyaml-available-as-a

I've sent them a PR to add manylinux1 wheels https://bitbucket.org/ruamel/yaml/issues/33/request-manylinux1-wheel