Pileup variant process stuck on specific chunks

[antoniognmk]

Operating System

Other Linux (please specify below)

Other Linux

Amazon Linux on AWS Batch

Workflow Version

v2.4.1

Workflow Execution

Command line (Cluster)

Other workflow execution

No response

EPI2ME Version

No response

CLI command run

No response

Workflow Execution - CLI Execution Profile

None

What happened?

Hello, I am currently testing the wf-human-variation Nextflow pipeline on AWS Batch. I aligned a human sample with mean depth coverage 29X with wf-alignment workflow, and it worked well.

Anyway, during my first try with wf-human-variation, using the [email protected] for the override_basecaller_cfg parameter, the snp:pileup_variants step triggered 632 processes, one for each chunk. The majority of chunks were processed within minutes. However, chunks 26, 29, 497, and 498 ran for over 24 hours and did not finish. Therefore, I canceled the execution.

The next day, thinking that perhaps I had chosen the wrong model, I tried again, this time with [email protected]. To my surprise, the same chunks got stuck again.

I am trying to execute the workflow with all variant call options active (sv, cnv, str, mod, snp). All other parameters are with their default values (except bam_min_coverage, which I am using 15).

Does anybody have any insights on the matter? Thanks for your attention.

Relevant log output

Unfortunately, I could not retrieve the logs because I canceled the execution and AWS CloudWatch did not produce the Nextflow engine.log.

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

[maggieconrad624]

Hello! Did you solve this problem? I have the same issue.

[alanfwilliams]

Just bumping this as it is affecting our workflows. Unfortunately, the logs are unhelpful because all there is is a bunch of Submitted process > snp:pileup_variants and then nothing.

[SiLu72]

same issue here, run through GUI on win11

[pgonzale60]

Hello all,

Thanks for reaching out.

I suspect those chunks correspond to high coverage regions. Could you try --downsample_coverage?

Relevant documentation

downsample_coverage	boolean	Downsample the coverage to along the genome.	This options will trigger a downsampling of the read alignments to the target coverage specified by --downsample_coverage_target. Downsampling will make the workflow run faster but could lead to non-deterministic variant calls.	False
downsample_coverage_target	number	Average coverage or reads to use for the analyses.	This options will set the target coverage for the downsampling stage, if downsampling has been enabled.	60