Process terminated while processing chromosomes

[kwglam]

Hi,

I was running the following command with 100g memory and 2 threads:

isoquant.py --reference /gencode/human/fasta/GRCh38.primary_assembly.genome.fa --genedb /gencode/human/gtf/gencode.v38.annotation.gtf --complete_genedb --bam file1_GRCh38_aligned.bam --data_type nanopore -o /isoquant_outputs

The program terminated in the middle of processing chromosomes (please see the log below). Would you please kindly advise how to fix it? Thanks!

Error in the log:

2025-01-23 22:42:23,084 - INFO - Collecting read alignments
2025-01-23 22:42:23,731 - INFO - Processing chromosome chr1
2025-01-23 22:42:23,990 - INFO - Processing chromosome chr2
2025-01-23 22:42:24,040 - INFO - Processing chromosome chr13
2025-01-23 22:42:24,124 - INFO - Processing chromosome chr4
2025-01-23 22:42:24,141 - INFO - Processing chromosome chr5
2025-01-23 22:42:24,150 - INFO - Processing chromosome chr3
2025-01-23 22:42:24,181 - INFO - Processing chromosome chrX
2025-01-23 22:42:24,182 - INFO - Processing chromosome chr10
2025-01-23 22:42:24,217 - INFO - Processing chromosome chr7
2025-01-23 22:42:24,225 - INFO - Processing chromosome chr12
2025-01-23 22:42:24,243 - INFO - Processing chromosome chr11
2025-01-23 22:42:24,273 - INFO - Processing chromosome chr14
2025-01-23 22:42:24,298 - INFO - Processing chromosome chr6
2025-01-23 22:42:24,299 - INFO - Processing chromosome chr15
2025-01-23 22:42:24,300 - INFO - Processing chromosome chr9
2025-01-23 22:42:24,314 - INFO - Processing chromosome chr8
2025-01-24 04:36:23,811 - CRITICAL - IsoQuant failed with the following error, please, submit this issue to https://github.com/ablab/IsoQuant/issuesTraceback (most recent call last):
File "/opt/conda/bin/isoquant.py", line 819, in
main(sys.argv[1:])
File "/opt/conda/bin/isoquant.py", line 813, in main
run_pipeline(args)
File "/opt/conda/bin/isoquant.py", line 763, in run_pipeline
dataset_processor.process_all_samples(args.input_data)
File "/opt/conda/share/isoquant-3.6.2-0/src/dataset_processor.py", line 458, in process_all_samples
self.process_sample(sample)
File "/opt/conda/share/isoquant-3.6.2-0/src/dataset_processor.py", line 482, in process_sample
self.collect_reads(sample)
File "/opt/conda/share/isoquant-3.6.2-0/src/dataset_processor.py", line 560, in collect_reads
for read_groups, alignment_stats, processed_reads in results:
^^^^^^^
File "/opt/conda/lib/python3.12/concurrent/futures/process.py", line 642, in _chain_from_iterable_of_lists
for element in iterable:
^^^^^^^^
File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/conda/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

[savytskanatalia]

I have a similar error, but at the stage of loading read assignments after processing:

2025-01-30 13:23:59,309 - INFO - Loading read assignments from /data/isoquant/aux/isoquant.save_chrM
File "/opt/conda/envs/isoquant/bin/isoquant.py", line 819, in
main(sys.argv[1:])
File "/opt/conda/envs/isoquant/bin/isoquant.py", line 813, in main
run_pipeline(args)
File "/opt/conda/envs/isoquant/bin/isoquant.py", line 763, in run_pipeline
dataset_processor.process_all_samples(args.input_data)
File "/opt/conda/envs/isoquant/share/isoquant-3.6.3-0/src/dataset_processor.py", line 456, in process_all_samples
self.process_sample(sample)
File "/opt/conda/envs/isoquant/share/isoquant-3.6.3-0/src/dataset_processor.py", line 509, in process_sample
self.process_assigned_reads(sample, saves_file)
File "/opt/conda/envs/isoquant/share/isoquant-3.6.3-0/src/dataset_processor.py", line 698, in process_assigned_reads
for read_stat_counter, tsc in results:
File "/opt/conda/envs/isoquant/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
for element in iterable:
File "/opt/conda/envs/isoquant/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/opt/conda/envs/isoquant/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/conda/envs/isoquant/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

[kwglam]

How many samples are you running? I've got the problem fixed by increasing the memory to 1TB. But don't know if it's the exact reason. Thx!

[andrewprzh]

Dear @savytskanatalia @kwglam

Both errors are not related to the IsoQuant code itself. Could be a memory error. Decreasing the number of threads (even down to 1) may indeed solve the problem.
Could you send me the log files just in case?

Best
Andrey