Shell error in antismash_overview_gather rule: Argument list too long

[OmkarSaMo]

In a large project with close to 70 k BGCs the following issue appeared as the argument list for shell command was too long.

/datadrive/data2/bgcflow/.snakemake/shell_tmp.pvmw0yn0/script.sh: line 2: /datadrive/data2/bgcflow/.snakemake/conda/18ddcd54edd3f107fecb89d7a50d830f_/bin/python: Argument list too long

[OmkarSaMo]

Temporary solution that may work is to increase the argument size limit

https://www.veritech.net/argument-list-long-simple-solution/

ulimit -s 665536

However this solution did not work in my example

[OmkarSaMo]

@matinnuhamunada - is this issue easy to resolve? It's very important to find some solution in large-scale run. Do you have temporary solution as well?

[matinnuhamunada]

The solution is to use either a .txt file as input or pure python using the run instead of shell

[matinnuhamunada]

Can you check if 15879d9 fixes the problem?

Bash scripts written "safely" avoid any command $LONG_ARGUMENT_LIST, and instead use pipes when the argument list is unbounded, e.g. with constructs such as echo $LONG_ARGUMENT_LIST | xargs command. The latter works safely because echo is a bash "builtin", so no external command is run and the ARG_MAX limit is not relevant.

Inspired from:
snakemake/snakemake#854

[OmkarSaMo]

@matinnuhamunada - Still got an error on the large dataset with v0.7.0.

However this time the error was with the rule antismash_summary. The rule antismash_overview_gather worked without any issues.

/datadrive/bgcflow/.snakemake/shell_tmp.41z66epq/script.sh: line 2: /datadrive/bgcflow/.snakemake/conda/5278299a2f85ccdaa9384b194969e4bd_/bin/python: Argument list too long

[matinnuhamunada]

Will be fixed in 0.7.1

[matinnuhamunada]

Fix in 144ad89

[OmkarSaMo]

Thanks a lot for the fix