dada2 #2483

bernt-matthias · 2019-07-05T12:58:44Z

moved over from here https://github.com/bernt-matthias/mb-galaxy-tools/pulls

TODOs:

generate yml based workflow with tests
- workflow moved to IWC

FOR CONTRIBUTOR:

- I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
- License permits unrestricted use (educational + commercial)
- This PR adds a new tool or tool collection
- This PR updates an existing tool or tool collection
- This PR does something else (explain below)

bernt-matthias · 2019-07-05T13:00:24Z

ping @shiltemann @yvanlebras

bernt-matthias · 2019-07-06T13:15:14Z

Seems that the datatypes have are not yet in the Galaxy version used by planemo. They are alreadey in the Galaxy sources: galaxyproject/galaxy#7895 but I don't know if I was in time for 19.05.

I could backport them if it would help.

The data manager tests should actually work. Are they invoked correctly?

tools/dada2/macros.xml

loraine-gueguen · 2019-10-14T15:36:23Z

hi @bernt-matthias (cc @lecorguille)
Thanks a lot for you great job on dada2! We are very interested to add these tools in our galaxy instance.
Can I help in any way to solve the issues and complete the development ?

bernt-matthias · 2019-10-15T11:24:55Z

Hi @loraine-gueguen. At the moment you can install the tools from the test toolshed (I just made sure that they are at the latest version). I guess a review and eventually approval of this PR would be the best.

The errors only affect the data manager (aka dadamanager) .. and I did not understand them .. just tried to rebase to the latest master .. maybe this helps. My local tests are successful. Otherwise I will ask the IUC for help.

loraine-gueguen · 2019-10-15T15:41:16Z

Thanks a lot for the testtoolshed update !
I am completely ignorant about dada2, but I can have a look at the "tool wrapper syntax" stuff of the PR if you want (except data manager unfortunately, I know nothing about data manager :-|)

bernt-matthias · 2019-10-15T15:56:47Z

This would already help a lot.

bernt-matthias · 2019-10-29T14:58:32Z

Surprise: v1.12 (not yet in the TTS)

would be really happy to bring this to an end soon. reviews would be great.

loraine-gueguen · 2019-10-29T15:04:37Z

Sorry I didn't have time to review last 2 weeks (urgent projects and days off)... But I'll do it as soon as possible

gregvonkuster · 2019-11-05T21:58:26Z

@bernt-matthias @bgruening Sorry I'm getting into this late, I have a new customer that needs this tool in Galaxy. Is there anything I can do to help move this PR along?

yvanlebras · 2019-11-05T22:51:45Z

+1 I can try to test last version "as a biologist"

gregvonkuster · 2019-11-05T23:57:51Z

@bernt-matthias This code is beautiful! Just a couple of nit-picky comments, but I don't feel that these necessarily warrant changes unless you agree.

Could these lines in the 3 tools become a macro? https://github.com/galaxyproject/tools-iuc/pull/2483/files#diff-246b665596cf9b58d86b16d961ebf249R18-R19
Maybe remove commented code in some places just to make the code as clean as possible.

I've restricted my review to the tool configs, including data managers, and R code. If nothing is missing form this PR to make the tools all work within Galaxy (I'm late to the game here), then I'm +1 on this when tests pass.

bernt-matthias · 2019-11-06T07:19:16Z

hey @gregvonkuster thanks for the review. I removed the comments (in one case I decided to leave it, but format it properly).

I decided against a macro for the 2 lines, I had the feeling that it does not help with the readability of the code.

bernt-matthias · 2019-11-06T07:21:01Z

Anyone problems with profile="19.09" (which is not even released yet)? Needed because of the new data types. If one would vote for an older Galaxy release I would need to backport the corresponding commit.

bernt-matthias · 2019-11-06T08:12:56Z

The most important question to me is how well the output of the pipeline (i.e. of assignTaxonomy / makeSequenctable) as implemented in the wrapppers

can be fed to connect to possible downstream tools
are consistent with other amplicon approaches in Galaxy (qiime / mothur)

Where the first seems more important to me.

gregvonkuster · 2019-11-06T12:38:44Z

Anyone problems with profile="19.09" (which is not even released yet)? Needed because of the new data types. If one would vote for an older Galaxy release I would need to backport the corresponding commit.

I'll need this on Galaxy main which states that it is currently running 19.09, but not sure since it is not yet released. ;)

mvdbeek

Nice, thanks for finding the data manager test workaround. The fastq.gz files are relatively large, any chance that you could downsample them a bit more ?

tools/dada2/datatypes_conf.xml

bernt-matthias · 2019-11-06T16:43:37Z

The fastq.gz files are relatively large, any chance that you could downsample them a bit more ?

Consider it done. Do I need to rebase and squash the commits to actually affect the size of the git repo?

nsoranzo · 2019-11-06T16:58:30Z

Do I need to rebase and squash the commits to actually affect the size of the git repo?

That would be better, otherwise whoever merges this PR will need to remember to select "Squash and merge".

data_managers/data_manager_dada2/.shed.yml

data_managers/data_manager_dada2/data_manager/dada2_fetcher.xml

bgruening · 2019-11-06T17:25:35Z

Just send my two last comments. Please check the tool owners ... I can squash during merge.

bernt-matthias · 2019-11-06T22:06:04Z

Will be a few days offline and implement the changes afterwards.

I think squashing is not necessary anymore - I rebased already.

loraine-gueguen

My 2 cents, but the tool configurations look good to me.

data_managers/data_manager_dada2/.shed.yml

tools/dada2/.shed.yml

tools/dada2/dada2_filterAndTrim.xml

loraine-gueguen · 2019-11-07T20:39:37Z

We have a few interested users that will test the suite. Is the code strictly limited to installation and use with release 19.09 ?

bernt-matthias · 2019-11-08T08:39:32Z

Is the code strictly limited to installation and use with release 19.09 ?

The only aspect that requires 19.09 are the data types. You can add them manually to your datatypes.xml. I'm not sure if installation from the TS is checking for the version, but manual installation is definitely possible.

Let me know if I should update the version in the TTS.

loraine-gueguen · 2019-11-08T14:52:11Z

It seems not possible to install a tool with a profile higher than the current galaxy profile, neither from the TS or manually (unless editing every tool config to change the profile, what I turned out to do).

From the TS, the tools appear as "invalid" in the admin "manage tools" section with no error messages. Manually, I had the following error messages:

  File "lib/galaxy/tools/__init__.py", line 446, in __init__
    raise e
Exception: The tool dada2_assignTaxonomyAddspecies targets version 19.09 of Galaxy, you should upgrade Galaxy to ensure proper functioning of this tool.

bernt-matthias · 2019-11-08T16:54:39Z

@loraine-gueguen good to know. manually editing the tool xml files should boil down to something like:

find . -name "dada2_*xml" -exec sed -i -e 's/profile="19.09"/profile="19.01"/' {} ;

and adding the data types.

Could be added to the docs.

moved over from here https://github.com/bernt-matthias/mb-galaxy-tools/pulls

if there is more than 1 changed repo (eg 1 tool and 1 data manager) then planemo is called per repo, which fails for data manager because the path to the xml is needed. hence if the repo is in the data_managers directory the path to the xml file is determined with an additional call to `planemo ci_find_tools` and then used in the planemo test call.

bgruening · 2019-11-08T22:41:21Z

Ok, let's get this in, way too long open. Hope that is ok for everyone ...

loraine-gueguen · 2019-11-27T09:59:28Z

A few comments regarding our DADA2 installation in Galaxy from the TS:

For Galaxy releases older than 19.09, it seems not enough to manually change profile in the xml files (restrictions due to a profile parameter registered in the database ?). To make the tools available in our 18.05 instance, I had to add the tools manually in galaxy-dist/tools/dada2/ (in fact, I added symbolic links pointing to the repositories of each tool in shed_tools). The best would be to update our galaxy instance to 19.09, which is now released.

Moreover, a release of conda >4.5.11 is definitely needed to have the [email protected] conda environment installing correctly: I faced the same issue as bioconda/bioconda-recipes#13846 (comment), and it was ok after updating conda to 4.7.12 (and python).

Small stuffs: the image pairpipe.png is not displaying in the help section (text pairpipe.png instead), and the doi citation displays an error message Failed to fetch BibTeX for DOI (although the hyperlinks are ok). Do you want me to open an issue for these ?

A user is currently testing the tool suite.

gregvonkuster · 2019-11-27T12:54:25Z

This tool suite is also currently being tested on Galaxy test https://test.galaxyproject.org, although the reference datasets are not yet installed for the tools that use them.

bernt-matthias · 2019-11-27T13:06:26Z

For Galaxy releases older than 19.09, it seems not enough to manually change profile in the xml files (restrictions due to a profile parameter registered in the database ?). To make the tools available in our 18.05 instance, I had to add the tools manually in galaxy-dist/tools/dada2/ (in fact, I added symbolic links pointing to the repositories of each tool in shed_tools). The best would be to update our galaxy instance to 19.09, which is now released.

Odd. Do the dada tools end up in shed_tools.xml? Or in which of the tool.xml files?

Moreover, a release of conda >4.5.11 is definitely needed to have the [email protected] conda environment installing correctly: I faced the same issue as bioconda/bioconda-recipes#13846 (comment), and it was ok after updating conda to 4.7.12 (and python).

A small comment on the README in the repo would be nice. Could you do this?

Small stuffs: the image pairpipe.png is not displaying in the help section (pairpipe.png instead), and the doi citation displays an error message Failed to fetch BibTeX for DOI (although the hyperlinks are ok). Do you want me to open an issue for these ?

Images seem to work only when installed from the TS.

bernt-matthias · 2019-11-27T13:07:20Z

@gregvonkuster cool :) .. manual tests I guess?

gregvonkuster · 2019-11-27T13:11:42Z

A lab here on the Penn State campus is running some of the tools on their data. If all goes well, I'll try to get the tools installed on Galaxy main before the Christmas break or shortly after the new year.

gregvonkuster · 2019-11-27T13:12:28Z

I need to get in touch with @jennaj to get the reference datasets installed.

gregvonkuster · 2019-11-27T14:24:03Z

This @bernt-matthias I've opened this issue galaxyproject/usegalaxy-playbook#273 requesting dada2 reference databases to be installed on Galaxy test / main, but I'm not quite sure which databases are the most used. So if I've chosen incorrectly, please let me know.

loraine-gueguen · 2019-11-28T09:21:17Z

Odd. Do the dada tools end up in shed_tools.xml? Or in which of the tool.xml files?

The dada tools were correctly added in a shed_tool_conf file dedicated to dada2 tools (shed_tool_conf_dada2.xml) which has been added in the tool_config_file parameter in galaxy.yml. And the installation went fine for dada2 v1.10 from the testtoolshed in the same shed tool_conf file.

A small comment on the README in the repo would be nice. Could you do this?

done: #2699

Images seem to work only when installed from the TS.

ok. For DOI citation as well ?

Thanks

bernt-matthias · 2019-11-30T15:30:10Z

Any immediate requests for dada2 could be added here: #2705

bernt-matthias force-pushed the topic/dada2 branch from d104e6c to eb326e3 Compare July 6, 2019 08:50

bernt-matthias changed the title ~~dada2: initial commit~~ dada2 Jul 8, 2019

nsoranzo force-pushed the master branch from 53d8d20 to 959f4e7 Compare July 18, 2019 10:07

bgruening reviewed Aug 29, 2019

View reviewed changes

tools/dada2/macros.xml Outdated Show resolved Hide resolved

This was referenced Sep 5, 2019

dada2 Helmholtz-UFZ/galaxy-tools#5

Closed

Issues trying to install Dada2 from Test TS Helmholtz-UFZ/galaxy-tools#4

Closed

bernt-matthias force-pushed the topic/dada2 branch from 3769789 to df2dfeb Compare October 15, 2019 11:17

bernt-matthias mentioned this pull request Nov 6, 2019

dada2: add dada2 workflow for paired end data galaxyproject/iwc#14

Merged

mvdbeek reviewed Nov 6, 2019

View reviewed changes

tools/dada2/datatypes_conf.xml Outdated Show resolved Hide resolved

bernt-matthias force-pushed the topic/dada2 branch from 9c1efa7 to 3935640 Compare November 6, 2019 17:03

bgruening reviewed Nov 6, 2019

View reviewed changes

data_managers/data_manager_dada2/.shed.yml Outdated Show resolved Hide resolved

data_managers/data_manager_dada2/data_manager/dada2_fetcher.xml Show resolved Hide resolved

loraine-gueguen reviewed Nov 7, 2019

View reviewed changes

data_managers/data_manager_dada2/.shed.yml Outdated Show resolved Hide resolved

data_managers/data_manager_dada2/.shed.yml Outdated Show resolved Hide resolved

tools/dada2/.shed.yml Outdated Show resolved Hide resolved

tools/dada2/dada2_filterAndTrim.xml Outdated Show resolved Hide resolved

bernt-matthias added 2 commits November 8, 2019 18:09

dada2: initial commit

0042ba1

moved over from here https://github.com/bernt-matthias/mb-galaxy-tools/pulls

bernt-matthias force-pushed the topic/dada2 branch from 3935640 to 15a10d3 Compare November 8, 2019 17:10

bgruening merged commit f8b6b6e into galaxyproject:master Nov 8, 2019

bernt-matthias deleted the topic/dada2 branch March 8, 2020 10:37

dada2 #2483

dada2 #2483

Conversation

bernt-matthias commented Jul 5, 2019 • edited Loading

bernt-matthias commented Jul 5, 2019

bernt-matthias commented Jul 6, 2019

loraine-gueguen commented Oct 14, 2019

bernt-matthias commented Oct 15, 2019

loraine-gueguen commented Oct 15, 2019

bernt-matthias commented Oct 15, 2019

bernt-matthias commented Oct 29, 2019

loraine-gueguen commented Oct 29, 2019

gregvonkuster commented Nov 5, 2019

yvanlebras commented Nov 5, 2019

gregvonkuster commented Nov 5, 2019

bernt-matthias commented Nov 6, 2019

bernt-matthias commented Nov 6, 2019

bernt-matthias commented Nov 6, 2019

gregvonkuster commented Nov 6, 2019

mvdbeek left a comment • edited by nsoranzo Loading

Choose a reason for hiding this comment

bernt-matthias commented Nov 6, 2019

nsoranzo commented Nov 6, 2019

bgruening commented Nov 6, 2019

bernt-matthias commented Nov 6, 2019

loraine-gueguen left a comment

Choose a reason for hiding this comment

loraine-gueguen commented Nov 7, 2019

bernt-matthias commented Nov 8, 2019

loraine-gueguen commented Nov 8, 2019

bernt-matthias commented Nov 8, 2019

bgruening commented Nov 8, 2019

loraine-gueguen commented Nov 27, 2019 • edited Loading

gregvonkuster commented Nov 27, 2019

bernt-matthias commented Nov 27, 2019

bernt-matthias commented Nov 27, 2019

gregvonkuster commented Nov 27, 2019

gregvonkuster commented Nov 27, 2019

gregvonkuster commented Nov 27, 2019

loraine-gueguen commented Nov 28, 2019 • edited Loading

bernt-matthias commented Nov 30, 2019

bernt-matthias commented Jul 5, 2019 •

edited

Loading

mvdbeek left a comment •

edited by nsoranzo

Loading

loraine-gueguen commented Nov 27, 2019 •

edited

Loading

loraine-gueguen commented Nov 28, 2019 •

edited

Loading