Refactor parallelism utilities for public API

[jakelishman]

Summary

should_run_in_parallel was added in a stable manner to enable backport to the 1.1 series, but from 1.3 onwards, we want this to be part of the public interface so that others can rely on it too.

As part of this, the parallelisation configuration was made more robust and controllable with context managers. This is convenient beyond just for users - it makes it far easier to control the parallelism during the test suite runs. Several instances where different parts of Qiskit and its test suite reached into deep internals of the parallelism utilities and made significant assumptions about the internal logic are refactored to use public interfaces to achieve what they wanted to.

The multiprocessing detection is changed from making OS-based assumptions about what Python does to simply querying the module for its configuration. This makes it more robust to changes in Python's handling (especially important since 3.14 will change the default start method on Unix). In the future, we may want to change to making these assumptions only if the user hasn't configured the multiprocessing start method themselves.

Details and comments

This was elided from #12410 to make that PR backwards compatible. This PR exposes the feature as part of the public API, so will be new for 1.2.

Depends on #12410.

[qiskit-bot]

One or more of the the following people are requested to review this:

• @Qiskit/terra-core

[coveralls]

Pull Request Test Coverage Report for Build 11633906056

Details

• 81 of 93 (87.1%) changed or added relevant lines in 8 files are covered.
• 8 unchanged lines in 3 files lost coverage.
• Overall coverage increased (+0.01%) to 88.737%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
qiskit/circuit/quantumcircuit.py	2	3	66.67%
qiskit/transpiler/preset_passmanagers/builtin_plugins.py	1	2	50.0%
qiskit/utils/parallel.py	68	78	87.18%

Files with Coverage Reduction	New Missed Lines	%
crates/qasm2/src/expr.rs	1	94.02%
qiskit/user_config.py	1	86.87%
crates/qasm2/src/lex.rs	6	91.98%

Totals
Change from base Build 11632855301:	0.01%
Covered Lines:	76394
Relevant Lines:	86090

---

💛 - Coveralls

[jakelishman]

Now rebased over #12410.

[mtreinish]

I mean sort of, we don't explicitly check what multiprocessing is set to (ie we ignore if the user explicitly sets this to spawn), but the value for PARALLEL_DEFAULT is based on what the OS default start method is.

[jakelishman]

Yeah, I think I was being more aspirational than correct here. We should do the check based on the multiprocessing start method if that's what we care about, but we don't. I can change the wording.

[mtreinish]

Maybe we should just do this, it seems totally in scope to update the logic in the function to call multiprocessing.get_start_method() and get rid of the OS based logic here for 1.2.

[kevinhartman]

Maybe we should just do this, it seems totally in scope to update the logic in the function to call multiprocessing.get_start_method() and get rid of the OS based logic here for 1.2.

@mtreinish is this still something we ought to do prior to merging this for 1.3?

[jakelishman]

Yeah, we still intended to do this

[jakelishman]

I did this, which then of course broke a bunch of weird other assumptions we were making thirty miles away in the library, so I ended up really rejigging the public API of the parallelism utilities so we didn't need to make internal assumptions across file boundaries.

I'm not entirely sold on the way I wrote the multiprocessing test (see the commit message) - I was trying to match the spirit of the previous assumption-heavy code, but I'm not certain we couldn't do a shade better.

[kevinhartman]

Maybe we should just do this, it seems totally in scope to update the logic in the function to call multiprocessing.get_start_method() and get rid of the OS based logic here for 1.2.

@mtreinish is this still something we ought to do prior to merging this for 1.3?

[jakelishman]

I've force-pushed a major new commit that properly refactored a bunch of the parallelism utilities to better support should_run_in_parallel at the level of robustness we'd expect from the public interface. This then removes a bunch of our own library code and test code that was reaching in the belly of the internals of the parallel code and layering assumptions on top of assumptions, and wraps the necessary components in safe interfaces.

I also added a QISKIT_IGNORE_USER_SETTINGS environment variable and configured it to be used in all tox and CI runs by default, which lets us isolate the test suite from the environment - that should make the new tests of what I added reliable, but it's probably something we should have had before anyway.

[jakelishman]

Given how major the changes I made here were, and that we're after the feature freeze deadline, I'm fine if we choose to leave this for Qiskit 2.0.

[mtreinish]

I agree, given the scope of the changes and that we're only one day out from rc1 lets defer this to 2.0. We can move forward with it pretty soon after rc1 though.