Impossible to analyse multiple contig files on Python>=3.8

[misialq]

Hello,

I have been trying to use QUAST to analyse multiple assemblies - I followed the instructions given in the documentation and used the test data from this repo. When I try to run the command as-is (i.e., passing both contig files) but also in its "minimal" version:
quast.py test_data/contigs_1.fasta test_data/contigs_2.fasta --debug -o test_output2
in an environment with Python 3.8 the command fails with the following error (see the attached file for the full log):

  File "/Users/mziemski/miniconda3/envs/quast-test-38/lib/python3.8/site-packages/quast_libs/qutils.py", line 181, in correct_contigs
    old_fpaths, corr_fpaths, broken_scaffold_fpaths, logs, is_fatal_error = run_parallel(parallel_correct_contigs, parallel_run_args, n_jobs)
  File "/Users/mziemski/miniconda3/envs/quast-test-38/lib/python3.8/site-packages/quast_libs/qutils.py", line 1066, in run_parallel
    results_tuples = Parallel(**parallel_args)(delayed(_fn)(*args) for args in fn_args)
  File "/Users/mziemski/miniconda3/envs/quast-test-38/lib/python3.8/site-packages/joblib/parallel.py", line 1054, in __call__
    self.retrieve()
  File "/Users/mziemski/miniconda3/envs/quast-test-38/lib/python3.8/site-packages/joblib/parallel.py", line 933, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/Users/mziemski/miniconda3/envs/quast-test-38/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
TypeError: '>=' not supported between instances of 'int' and 'NoneType'
ERROR! exception caught!

There is no error and everything runs and finishes correctly when I run the very same command in a Python 3.7 environment. Also, it does not fail when I set the threads parameter to 1 (i.e., no parallelisation).
In contrast, it fails regardless of QUAST version: tried with 5.0.2 (log attached) but also latest master with the same result.

After some poking, I noticed that the min_contig parameter gets reset to None at some point when the parallelisation is being applied in the correct_contigs function.

Run log:
py38_quast_502.log

[mikolmogorov]

Confirming this for MacOs + Python 3.9.7

[alexeigurevich]

Well, I was able to reproduce this on macOS with Python 3.8 and 3.9. But it works fine on Linux with e.g., Python 3.9.5.
So, it seems that here we see a crucial combination of a specific platform (macOS), specific Python/joblib versions (3.8, 3.9, probably above) and ugly handling of global variables in parallel functions in Quast :)

As an intermediate solution I added handling of this particular crash and suggesting users to use single thread (-t 1).
A better solution with Quast refactoring follows (but it takes time).

[LilyYe98]

I met the same problem with MacOS + Python 3.9.7.
Then I set thread to 1 (-t 1) and it worked.
Thanks so much for your information!

Well, I was able to reproduce this on macOS with Python 3.8 and 3.9. But it works fine on Linux with e.g., Python 3.9.5. So, it seems that here we see a crucial combination of a specific platform (macOS), specific Python/joblib versions (3.8, 3.9, probably above) and ugly handling of global variables in parallel functions in Quast :)

As an intermediate solution I added handling of this particular crash and suggesting users to use single thread (-t 1). A better solution with Quast refactoring follows (but it takes time).