migrate in samtools, kraken, vcflib #1148

martenson · 2017-01-24T16:47:41Z

part of galaxyproject/tools-devteam#406
xref: #1060

this is an effort to address part of galaxyproject/tools-devteam#416

martenson · 2017-01-24T16:54:19Z

tool_collections/samtools/bam_to_sam/bam_to_sam.xml

+        <test>
+            <param ftype="bam" name="input1" value="bam_to_sam_in1.bam" />
+            <param name="header" value="-h" />
+            <output file="bam_to_sam_out1.sam" name="output1" sorted="True" />


travis: /tmp/tmp4XWX1b:35:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element 'output', attribute 'sorted': The attribute 'sorted' is not allowed.

should this be just sort?
source https://docs.galaxyproject.org/en/latest/dev/schema.html#id74

martenson · 2017-01-24T18:17:17Z

tool_collections/samtools/samtools_stats/samtools_stats.xml

@@ -0,0 +1,296 @@
+<tool id="samtools_stats" name="Stats" version="2.0">


this file has

.. ERROR: Invalid tmpmXYONH found. Errors [/tmp/tmpmXYONH:218:0:ERROR:SCHEMASV:SCHEMAV_ELEMENT_CONTENT: Element 'filter': This element is not expected.]

problem

@martenson See my comment below.

nsoranzo · 2017-01-24T18:19:46Z

tool_collections/samtools/samtools_stats/samtools_stats.xml

+                    <when value="cached">
+                        <param name="ref_file" type="select" label="Using genome">
+                            <options from_data_table="fasta_indexes" />
+                            <filter type="data_meta" ref="input_file" key="dbkey" column="1" />


This should be:

<options from_data_table="fasta_indexes"> <filter type="data_meta" ref="input_file" key="dbkey" column="1" /> </options>

thought so, thanks for clarification, fixing

martenson · 2017-01-24T20:21:20Z

both samtools_mpileup tests fail with

======================================================================
FAIL: MPileup ( samtools_mpileup ) > Test-2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tmpW4VvWu/galaxy-dev/test/functional/test_toolbox.py", line 302, in test_tool
    self.do_it( td )
  File "/tmp/tmpW4VvWu/galaxy-dev/test/functional/test_toolbox.py", line 78, in do_it
    raise e
JobOutputsError: [Errno 2] No such file or directory: '/home/travis/build/galaxyproject/tools-iuc/tool_collections/samtools/samtools_mpileup/test-data/samtools_mpileup_out_2.log'

bgruening · 2017-01-29T11:53:12Z

@martenson can we remove all tool_dependencies.xml files and make sure all are condafied?

martenson · 2017-01-29T21:27:43Z

@bgruening will do

martenson · 2017-01-30T19:38:11Z

tool_collections/kraken/macros.xml

+    <xml name="requirements">
+        <requirements>
+            <requirement type="package" version="0.10.6_eaf8fb68">kraken</requirement>
+            <requirement type="package" version="0.10.6-eaf8fb68">kraken</requirement>


@davebx Do you know why is the requirement here twice? Once with dash and once with underscore.

I guess it is because of the bioconda vs TS package names.

martenson · 2017-02-06T15:03:48Z

thank you @yhoogstrate !!

bgruening · 2017-02-06T22:32:03Z

it seems there is a wrong samtools version used:

[main] unrecognized command '--version'
[sam_header_read2] 1 sequences loaded.
sort: invalid option -- 'O'
sort: invalid option -- 'T'

yhoogstrate · 2017-02-15T12:40:13Z

This error is coming from samtools 0.1.19, installed by travis: https://github.com/galaxyproject/tools-iuc/blob/master/.travis.yml#L20 . I think Galaxy does not override it with 1.2.* properly and uses 0.1,19 that was/is necessary for the Galaxy backend, instead.

@jmchilton is trying to solve it in this PR: galaxyproject/galaxy#3620 , thanks 👍

Once this works, I would like all tools that make use of samtools, bcftools and ucsc-tools to re-trigger a travis test to make sure they were not tested with wrong dependencies.

@nsoranzo

This has never happened in my testing, but there have been multiple reports of Conda crafting environments for the set metadata tool that include Python. When this happens Galaxy's Python environment is lost and the tool cannot function properly. Fixes #3238. Fixes #3224. Fixes https://biostar.usegalaxy.org/p/20865/. Rebased to add a comment as suggested by @nsoranzo.

nsoranzo · 2017-02-17T12:50:44Z

tool_collections/kraken/kraken/kraken.xml

+            #else:
+                --fasta-input
+            #end if
+            "$forward_input" "$reverse_input"


'${single_paired.forward_input}' '${single_paired.reverse_input}'

nsoranzo · 2017-03-01T10:17:07Z

tool_collections/samtools/cram_to_bam/samtools_cram_to_bam.xml

+    </macros>
+    <expand macro="requirements"/>
+    <expand macro="version_command"/>
+    <expand macro="stdio"/>


<expand macro="stdio"/>

should be moved between requirements and version_command

nsoranzo · 2017-03-01T10:18:16Z

tool_collections/kraken/kraken_filter/kraken-filter.xml

+            <param name="threshold" value="0"/>
+            <param name="kraken_database" value="test_db"/>
+
+            <output name="output" file="kraken_filter_test1_output.tab" ftype="tabular"/>


s/"output"/"filtered_output"/

nsoranzo · 2017-03-01T10:20:23Z

tool_collections/samtools/samtools_rmdup/samtools_rmdup.xml

+
+    <expand macro="stdio"/>
+    <expand macro="version_command"/>
+    <expand macro="requirements" />


<expand macro="requirements" />

should be moved before stdio.

Also, rmdup should stay on 0.1.19 or move to 1.3.*, see galaxyproject/tools-devteam#284 and galaxyproject/tools-devteam#286 .

nsoranzo · 2017-03-01T10:30:17Z

tool_collections/samtools/samtools_flagstat/samtools_flagstat.xml

+    </inputs>
+
+    <outputs>
+    <data name="output1" format="txt" />


Indentation.

nsoranzo · 2017-03-01T10:31:23Z

tool_collections/samtools/samtools_flagstat/samtools_flagstat.xml

+        <test>
+          <param name="input1" value="samtools_flagstat_input1.bam" ftype="bam" />
+
+          <output name="output1" file="samtools_flagstat_out1.txt" />


Indentation of the previous 2 lines.

nsoranzo · 2017-03-01T10:31:41Z

tool_collections/samtools/samtools_flagstat/samtools_flagstat.xml

+  0 + 0 with mate mapped to a different chr (mapQ>=5)
+
+  </help>
+  <expand macro="citations"/>


Indentation of the previous 2 lines.

nsoranzo · 2017-03-01T10:34:10Z

tool_collections/kraken/kraken_filter/kraken-filter.xml

+        kraken-filter
+            @INPUT_DATABASE@
+            --threshold $threshold
+            '${input}' >


I'd move the > on the next line.

nsoranzo · 2017-03-01T10:39:01Z

tool_collections/kraken/kraken_report/kraken-report.xml

+
+        kraken-report
+            @INPUT_DATABASE@
+            '${kraken_output}' >


I'd move the > on the next line.

nsoranzo · 2017-03-01T10:39:09Z

tool_collections/kraken/kraken_translate/kraken-translate.xml

+        kraken-translate
+            @INPUT_DATABASE@
+            $mpa_format
+            '${input}' >


I'd move the > on the next line.

nsoranzo · 2017-03-01T10:39:30Z

tool_collections/samtools/samtools_reheader/samtools_reheader.xml

+    <command><![CDATA[
+        samtools reheader
+            '${input_header}'
+            '${input_file}' >


I'd move the > on the next line.

martenson · 2017-03-07T20:22:56Z

@nsoranzo I addressed your comments

yhoogstrate · 2017-03-08T14:23:26Z

Nice @martenson , planemo test --update_test_data and I think we're there 👍

nsoranzo · 2017-03-08T14:38:40Z

tool_collections/samtools/macros.xml

+<macros>
+    <xml name="requirements">
+        <requirements>
+            <requirement type="package" version="1.3">samtools</requirement>


s/1.3/1.3.1/

1.3.1 is the latest and is already used by tools/htseq_count/htseq-count.xml

should this go together with a version bump?

Yes, all samtools tool versions should be bumped, for bam_to_cram, cram_to_bam and samtools_split the new version should be 1.3.1 (using a macro ideally), all other tools have already a major version 2 so I'd increase the 'patch' version number, hoping one day we can sync again with the upstream version.

nsoranzo · 2017-03-08T14:39:34Z

tool_collections/samtools/samtools_flagstat/samtools_flagstat.xml

+    </inputs>
+
+    <outputs>
+      <data name="output1" format="txt" />


Indentation.

nsoranzo · 2017-03-08T14:53:31Z

tool_collections/samtools/samtools_stats/samtools_stats_tool_test_output.html

@@ -0,0 +1,372 @@
+<!DOCTYPE html>


This file should be removed.

bgruening · 2017-04-25T09:17:44Z

...llections/samtools/samtools_mpileup/test-data/samtools/mpileup/samtools_mpileup_out_1.pileup

@@ -0,0 +1,43 @@
+phiX174	1411	A	1	^P.	$


Is this file correct here?

bgruening · 2017-04-25T09:18:41Z

tool_collections/samtools/samtools_mpileup/test-data/samtools_mpileup_out_1.pileup

@@ -0,0 +1,43 @@
+phiX174	1411	A	0				1
+phiX174	1412	G	0				2,1,1


According to the last failing test, the last column needs to be removed to pass.
Is this correct? @nekrut have you used mpileup and now what the last column is and if it is ok if it get's removed?

This looks like a fixed bug in samtools and results in backwards incompatibility (compared to 0.19).
The option -O returns the base positions of the corresponding reads.
The 4th column indicates zero bases are included in the mpileup, while the last column indicates multiple reads are present. To me the behaviour in 1.3 looks appropriate.

But this file has multiple columns, the end is just a lot of tabulars afaik.

…r addtional reheader-header

nsoranzo · 2017-04-28T13:31:17Z

tool_collections/samtools/samtools_stats/test-data/samtools_stats_out2/gcc.tab

@@ -0,0 +1,252 @@
+# ACGT content per cycle. The columns are: cycle, and A,C,G,T counts (percent)
+1	21.50	29.50	33.50	15.50


This file also needs to be updated.

… updating it

nsoranzo · 2017-04-28T14:01:01Z

It's green 🎉 , thanks @yhoogstrate!

bgruening · 2017-04-28T16:11:30Z

🎉 !!!

bgruening · 2017-04-28T16:12:15Z

Thanks all, this is awesome!

nsoranzo · 2017-04-28T18:03:34Z

😟 samtools tool versions have not been updated for the new samtools 1.3 requirement, see https://github.com/galaxyproject/tools-iuc/pull/1148/files#r104935513

nsoranzo · 2017-04-29T11:27:47Z

Luckily all repo updates have failed because @martenson hasn't given the necessary permissions to iuc yet, so we can still fix the tool versions.

martenson · 2017-05-01T14:11:08Z

@nsoranzo yay! :)

martenson · 2017-05-05T17:27:45Z

all repos from this PR have been added to IUC-permitted

martenson added 2 commits January 24, 2017 11:23

migrate in kraken, vflib, samtools

4d079c8

change shed.yml: remote repo url to iuc

a0eb535

martenson added new software wip labels Jan 24, 2017

martenson mentioned this pull request Jan 24, 2017

migrate out kraken, samtools, vcflib; move to legacy galaxyproject/tools-devteam#442

Merged

martenson commented Jan 24, 2017

View reviewed changes

martenson added 2 commits January 24, 2017 12:58

update the rest of the remote repo urls

d9c1bf7

set chunk size to 4 temporarily

c345e50

martenson commented Jan 24, 2017

View reviewed changes

nsoranzo reviewed Jan 24, 2017

View reviewed changes

fix samtools stats schema lint

2a298cb

martenson commented Jan 30, 2017

View reviewed changes

martenson and others added 5 commits January 30, 2017 14:47

remove tool_dependencies.xml for samtools, let conda do it

70de720

samtools linting errors

b392dc3

samtools mpileup - attempt to get testing and linting working

c5da921

samtools mpileup - attempt 2 to get testing and linting working

ae39c24

samtools stats+sam_to_bam - small changes to get tests working

b637cef

yhoogstrate added 5 commits February 15, 2017 12:02

Samtools: update to version_command

eee57a1

Samtools: update to version_command - attempt 2

840024e

kraken: minor changes

eaf496a

Kraken: removal of tool-dependencies

d99fc25

samtools bam_to_cram small changes

9c88880

nsoranzo reviewed Feb 17, 2017

View reviewed changes

nsoranzo reviewed Mar 1, 2017

View reviewed changes

update rmdup to samtools 1.3 and tiny improvements, all thanks to 🦅 👀

f8fb9de

nsoranzo reviewed Mar 8, 2017

View reviewed changes

nsoranzo mentioned this pull request Apr 7, 2017

Update samtools to 1.4 and migrate to iuc. #1240

Closed

yhoogstrate and others added 4 commits April 25, 2017 09:09

Make test of cram/bam tools pass

9bb0b6d

Fix more tests

5ce99d2

test fixes

96d9878

Change test data

b758595

bgruening reviewed Apr 25, 2017

View reviewed changes

yhoogstrate added 4 commits April 26, 2017 10:06

removal of not used test files

2f64c9d

upgraded mpileup tests

c339ded

samtools reheader: allow testing with 1 line difference to correct fo…

8e28b30

…r addtional reheader-header

samtools stats also reports stats on "N" and "0"

9cf113c

nsoranzo reviewed Apr 28, 2017

View reviewed changes

samtools stats: moving test-data from sub-directory to allow properly…

46b0568

… updating it

yhoogstrate removed the wip label Apr 28, 2017

bgruening merged commit 0b2d726 into galaxyproject:master Apr 28, 2017

bgruening mentioned this pull request May 1, 2017

Samtools version bump, fixes and linting #1283

Merged

		@@ -0,0 +1,296 @@
		<tool id="samtools_stats" name="Stats" version="2.0">

		@@ -0,0 +1,252 @@
		# ACGT content per cycle. The columns are: cycle, and A,C,G,T counts (percent)
		1 21.50 29.50 33.50 15.50

migrate in samtools, kraken, vcflib #1148

migrate in samtools, kraken, vcflib #1148

Conversation

martenson commented Jan 24, 2017

martenson Jan 24, 2017 • edited Loading

Choose a reason for hiding this comment

martenson Jan 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martenson commented Jan 24, 2017

bgruening commented Jan 29, 2017

martenson commented Jan 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martenson commented Feb 6, 2017

bgruening commented Feb 6, 2017

yhoogstrate commented Feb 15, 2017 • edited Loading

nsoranzo Feb 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martenson commented Mar 7, 2017

yhoogstrate commented Mar 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhoogstrate Apr 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nsoranzo commented Apr 28, 2017

bgruening commented Apr 28, 2017

bgruening commented Apr 28, 2017

nsoranzo commented Apr 28, 2017

nsoranzo commented Apr 29, 2017

martenson commented May 1, 2017

martenson commented May 5, 2017

martenson Jan 24, 2017 •

edited

Loading

martenson Jan 24, 2017 •

edited

Loading

yhoogstrate commented Feb 15, 2017 •

edited

Loading

nsoranzo Feb 17, 2017 •

edited

Loading

yhoogstrate Apr 26, 2017 •

edited

Loading