Basis translation can fail if run in parallel

[Cryoris]

Environment

• Qiskit version: Qiskit 1.3.0 and main
• Python version: 3.11.9
• Operating system: macOS

What is happening?

This is the problem underlying Qiskit/qiskit-addon-cutting#714.

Running basis translation in parallel (see snippet below) causes the following failure

[...]
  File "/Users/jul/Qiskit/qiskit/qiskit/transpiler/passes/basis/basis_translator.py", line 129, in run
    return base_run(
           ^^^^^^^^^
ValueError: An invalid parameter was provided. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jul/IBM/snippets/repro.py", line 25, in <module>
    pm.run([qc_a, qc_b]) # This fails with the error
    ^^^^^^^^^^^^^^^^^^^^
[...]

  File "/opt/homebrew/Cellar/[email protected]/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
ValueError: An invalid parameter was provided.

This seems to be due to the recent port of the basis translator to Rust (see #12246 for an overview). Dissecting the code, the error seems to be triggered by compose_transforms, so it might be worth checking if this already existed after #13137 and before the rest of the basis translator was moved to Rust.

How can we reproduce the issue?

From @mtreinish:

I just came up with a minimal reproduce:

from qiskit.circuit import QuantumCircuit, Parameter
from qiskit import generate_preset_pass_manager
from qiskit.providers.fake_provider import GenericBackendV2
from qiskit.circuit.library import RXGate

backend = GenericBackendV2(5)
pm = generate_preset_pass_manager(1, backend)

param = Parameter('a')
gate = RXGate(param)

qc_a = QuantumCircuit(1)
qc_a.append(gate, [0])
qc_b = QuantumCircuit(1)
qc_b.append(gate, [0])

pm.run(qc_a)
pm.run(qc_b)
pm.run([qc_a, qc_b]) # This fails with the error

Note that this only happens with multiprocessing turned on. On Mac this is turned off per default but can be enabled with export QISKIT_PARALLEL=TRUE.

What should happen?

The above should run, as it did in 1.2.4.

Any suggestions?

One possible solution is to delete the cached py_op upon parameter assignment, i.e. replace

qiskit/crates/circuit/src/circuit_data.rs

Lines 1385 to 1389 in 6979d91

	borrowed
	.bind(py)
	.getattr(params_attr)?
	.set_item(parameter, new_param)?;
	borrowed.bind(py).setattr("_definition", py.None())?

by

previous.py_op = OnceLock::new();  // or OnceCell on Qiskit 1.3.0

But this comes at a ~5% performance overhead on my machine, benchmarked on some utility scale circuits. Maybe there's another way to avoid this overhead?

[mtreinish]

The specific issue here is around having the pass manager reused between multiple executions and then having the multiprocessing dispatching for the second run. My guess is that is most likely the round trip pickling of the equivalence library in the pass manager after we've bound parameters once during the basis translator messing up shared state in the cached pygates. This wasn't caught before #13482 because weren't updating the cache properly before but now we are.

[cosenal]

We noticed this as well in Mitiq. We have a jupyter notebook failing in the Github actions build, where platform=linux and execution is in parallel.

The tutorial is cumbersome, but we extracted a minimum non-working example from there:

from qiskit import transpile, QuantumCircuit
from qiskit_aer import QasmSimulator
from qiskit_aer.noise import NoiseModel

backend = QasmSimulator(noise_model=NoiseModel())

qc = QuantumCircuit(1)
qc.rx(1, 0)

exec_circuits = transpile(
    qc,
    backend=backend,
)
exec_circuits = transpile(
    [qc, qc],
    backend=backend,
)

I hope this helps.