Improve the ergonomics of the TranspileLayout class

[mtreinish]

Summary

This commit updates the TranspileLayout class to make it easier to work with. The TranspileLayout class was orginally just a container class that took the information returned from the transpiler needed to reverse the transpiler layout permutation (in the absence of ancillas) so that QuantumCircuit.from_circuit() could compare Operator instances across a transpile(). However, the internal data structures used by the transpiler are hard to work with, and don't map well to the normal reasoning around how the transpiler transforms the circuit. To improve the usability of the class this commit adds 4 new methods to the class:

• initial_layout_list() and final_layout_list() which compute a list form of the initial_layout and final_layout respectively
• full_layout() which generates a list that maps the input circuits qubit positions in the input circuit to the final position at the end of the circuit (which is what most people think of when they hear "final layout")
• permute_sparse_pauli_op() which takes in a SparsePauliOp object and permutes it based on the full layout. This is especially useful when combined with the Estimator primitive This was removed and will be added in a follow-up PR as SparsePauliOp.apply_layout(): Add apply_layout method to SparsePauliOp #10947

To implement these functions a few extra pieces of information are needed to fully implement them. The first is we need to know the number of original circuit qubits. This is used to account for any ancillas that are added in the circuit, once we have the input circuit count we can use the original_qubit_indices attribute to discern which qubits in the initial layout are from the original circuit and which are ancillas. The second piece of information needed is the list of qubits in the output circuit. as this is needed to look up the position of the virtual qubit in the final_layout. These are both added as optional private attributes to the TranspileLayout class and there are some small changes to the pass manager and QPY to accomodate them.

Similarly the change in the TranspileLayout class requires a new QPY version to include the missing details that were not being serialized in QPY and weren't representable in the previous payload format.

Details and comments

Fixes #10826
Fixes #10818

[qiskit-bot]

One or more of the the following people are requested to review this:

• @Qiskit/terra-core
• @mtreinish
• @nkanazawa1989

[nonhermitian]

This looks like it is more or less what is needed here. However, I am not a big fan of calling the "final layout" the "full layout" as it seems off. Namely in the initial_layout is what goes in, irrespective of the need for swap mapping. The final_layout should be the actual "final layout" (here called "full"), with the post-routing layout called something like post_routing_layout or the like. If most people think this way, myself included, why not try to reach that end state?

[nonhermitian]

One question too is what happens to a layout if the swap mapper decides to use an ancilla qubit for routing? The active number of qubits is no longer the same as the original circuit. Does this just track the permutations of the original set?

[mtreinish]

This looks like it is more or less what is needed here. However, I am not a big fan of calling the "final layout" the "full layout" as it seems off. Namely in the initial_layout is what goes in, irrespective of the need for swap mapping. The final_layout should be the actual "final layout" (here called "full"), with the post-routing layout called something like post_routing_layout or the like. If most people think this way, myself included, why not try to reach that end state?

Well the path to changing final_layout to being the equivalent of full_layout in this PR is pretty lengthy because to change the old behavior of the final_layout symbol and migrate the existing functionality to a new name it would be a multi-release process:

1. add the alternative in this release
2. deprecate final_layout to warn users in the next release it's going away
3. remove the attribute after the deprecation period
4. add a new final_layout() method that matches full_layout() here.

I don't have an issue changing the names of anything added in this PR, full_layout was just the best name I could come up with. We can also add a post_routing_layout alias to start that migration process here. But at least in the medium term we'll need to have a name like full_layout (it can be anything except for final_layout, initial_layout, or input_qubit_mapping) for the method.

[mtreinish]

One question too is what happens to a layout if the swap mapper decides to use an ancilla qubit for routing? The active number of qubits is no longer the same as the original circuit. Does this just track the permutations of the original set?

For full_layout and permute_sparse_pauli_op it only tracks the permutations of the qubits in the original circuit. I realized I didn't add an explicit test case of this for full_layout to point to (only via permute_sparse_pauli_op).

initial_layout_list has a filter_ancilla option so you can either get the full width or just for the qubits in the original circuit. The final_layout_list method is only full width though, although I guess I could add a filter_ancilla option to it too it just wasn't as natural a fit to me for that given the permutation it tracks.

EDIT: I added a test case to show this in: 8546928

[jakelishman]

Building on what's in this PR, maybe something like this works as a full set of coherent operations? I've judiciously used the word "virtual" to avoid naming clashes and distinguish bi-maps Qubit: int from List[int], but mostly these are just the first names that came to mind:

class TranspileLayout:
    # Bi-map of initial positions (post-layout, pre-routing) of virtual `Qubit`
    # to physical `int`
    def initial_virtual_layout(self, filter_ancillas: bool) -> Layout: ...
    # Map of initial positions virtual list indices to physical `int`
    def initial_index_layout(self, filter_ancillas: bool) -> List[int]: ...
    # Bi-map of final positions (post-routing) of virtual `Qubit` to physical `int`.
    def final_virtual_layout(self, filter_ancillas: bool) -> Layout: ...
    # Map of final positions virtual list indices to physical `int`.
    def final_index_layout(self, filter_ancillas: bool) -> List[int]: ...
    
    # Potentially this too (I can't see much meaning for `filter_ancillas` here.)
    def routing_permutation(self) -> List[int]: ...

    # ... and now a bunch of helper methods that could also live elsewhere.
    def permute_to_initial(self, operator: OperatorLike) -> OperatorLike: ...
    def permute_to_final(self, operator: OperatorLike) -> OperatorLike: ...

For comparison:

• initial_virtual_layout is like the current initial_layout attribute, except it gains a filter_ancillas argument
• initial_index_layout is the PR's initial_layout_list
• final_virtual_layout has no counterpart
• final_index_layout is like the PR's full_layout except it gains a filter_ancillas argument
• routing_permutation is like the current final_layout attribute, except it more accurately uses a List[int] rather than mapping.
• permute_to_initial has no counterpart
• permute_to_final is the PR's permute_sparse_pauli_op

I know the argument for full_layout is to avoid having a name too close to the current final_layout attribute, but here I'm with Paul - I think the current attribute has the wrong name, and it'd be better not to constrain the new API based on it, especially if we document that people should avoid using the attributes and should instead always use the methods.

edit: and just for clarity, it hardly matters much what we actually store as the internal properties of the object if they're not publicly accessible - all the methods are calculable from knowing three pieces of information:

• the initial virtual-to-physical mapping given to routing including ancillas
• the indices of virtual qubits in the original circuit (without ancillas)
• the permutation applied to the physical qubits by routing

The PR already ensures we have access to all those three things, so there's no trouble calculating everything we need from there. There's many combinations of bits of the information we could store to recalculate everything from.

[jakelishman]

I should also say: I like the general direction of the PR, and I think this will be a very nice ergonomic improvement for users of the transpiler across the board - it's not exposing any information they couldn't already calculate themselves, but it's definitely making it easier to work with.

[coveralls]

Pull Request Test Coverage Report for Build 6228373802

• 71 of 75 (94.67%) changed or added relevant lines in 6 files are covered.
• 13 unchanged lines in 3 files lost coverage.
• Overall coverage increased (+0.001%) to 87.277%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
qiskit/qpy/binary_io/circuits.py	15	17	88.24%
qiskit/transpiler/layout.py	48	50	96.0%

Files with Coverage Reduction	New Missed Lines	%
qiskit/qpy/binary_io/circuits.py	3	92.46%
crates/qasm2/src/lex.rs	4	91.41%
crates/qasm2/src/parse.rs	6	97.6%

Totals
Change from base Build 6209701566:	0.001%
Covered Lines:	74243
Relevant Lines:	85066

---

💛 - Coveralls

[mtreinish]

Building on what's in this PR, maybe something like this works as a full set of coherent operations? I've judiciously used the word "virtual" to avoid naming clashes and distinguish bi-maps Qubit: int from List[int], but mostly these are just the first names that came to mind:

class TranspileLayout:
    # Bi-map of initial positions (post-layout, pre-routing) of virtual `Qubit`
    # to physical `int`
    def initial_virtual_layout(self, filter_ancillas: bool) -> Layout: ...
    # Map of initial positions virtual list indices to physical `int`
    def initial_index_layout(self, filter_ancillas: bool) -> List[int]: ...
    # Bi-map of final positions (post-routing) of virtual `Qubit` to physical `int`.
    def final_virtual_layout(self, filter_ancillas: bool) -> Layout: ...
    # Map of final positions virtual list indices to physical `int`.
    def final_index_layout(self, filter_ancillas: bool) -> List[int]: ...
    
    # Potentially this too (I can't see much meaning for `filter_ancillas` here.)
    def routing_permutation(self) -> List[int]: ...

    # ... and now a bunch of helper methods that could also live elsewhere.
    def permute_to_initial(self, operator: OperatorLike) -> OperatorLike: ...
    def permute_to_final(self, operator: OperatorLike) -> OperatorLike: ...

For comparison:

* `initial_virtual_layout` is like the current `initial_layout` attribute, except it gains a `filter_ancillas` argument

* `initial_index_layout` is the PR's `initial_layout_list` 

* `final_virtual_layout` has no counterpart

* `final_index_layout` is like the PR's `full_layout` except it gains a `filter_ancillas` argument

* `routing_permutation` is like the current `final_layout` attribute, except it more accurately uses a `List[int]` rather than mapping.

* `permute_to_initial` has no counterpart

* `permute_to_final` is the PR's `permute_sparse_pauli_op`

I'm fine with this as the API we go with here, it makes sense to me and is a lot clearer for users than what I came up with. I agree that full_layout probably wasn't a great name and I think the doc updates I added already probably cover the potential confusion around the .final_layout attribute being different than the final_layout_* methods. It's not hard to update the state of the PR to align with this proposal, I'll update it in a bit.

The only thing I think missing here is the equivalent of final_layout as an attribute routing_virtual_layout (I think that would be it following the naming convention here), just so we have consistent access patterns. I agree there is probably little value in having a filter_ancillas argument for the routing* pieces. For the routing_permutation() a method makes sense because it has potential overhead depending on the number of qubits, but routing_virtual_layout is just an alias to the final_layout attribute. Should it be a method or an attribute?

[jakelishman]

I think my issue with the existing final_layout attribute is that, to me, it's not a layout at all. Imo (by the typing/terminology of the class) Layouts must be a mapping of the virtual space to the physical space, while this is a mapping of the physical space to itself linked by which hidden virtual qubit is assigned to the physicals at the start/end. That's why I had it only as routing_permutation, and then only using integer indices; I think that particular object is quite different to the returns from the methods that I called *_layout.

I think the closest coherent object we could put out for the current final_layout which makes the virtual Qubit instances explicit would be a mapping of something like Qubit: (int, int), which be computed as

initial = transpile_layout.initial_virtual_layout(filter_ancillas=True)
final = transpile_layout.final_virtual_layout(filter_ancillas=True)
{
   virtual: (initial[virtual], final[virtual])
   for virtual, initial_physical in initial.get_virtual_bits().items()
}

I'm not super sure what use such an object could have though, which is why I didn't include it.

In terms of methods vs attributes: personally, I'd just make everything public here a method, even if (as an internal detail), it just returns a copy of an attribute. The cost of copying a layout is small compared to any operation using layouts, and users shouldn't really be able to mutate a TranspileLayout anyway; it wouldn't mean anything and would just make the circuit/layout correspondence invalid.

[mtreinish]

I reworked the API per the suggestions in 3f2542b . The only thing I haven't touched is the permute_sparse_pauli_op method. Mostly because I'm not sure which operator types I should support. I didn't see a common interface that would let me do the permutation in the same manner on something like BaseOperator in a consistent efficient manner. So we should figure out which set of operator classes we want to support for the new methods (and optionally where we should put the functionality)

[jakelishman]

I think the layout-related parts of the API look good and far more ergonomic in this new form, but I'm obviously biased so it would be good to have review on that from other potential users. I'm happy to approve those parts.

I think we should get some input from others on other teams / other downstream consumers of Qiskit about things like permute_sparse_pauli_op with regard to what methods would be useful to them, and where it would make most sense for them to live, but I think the principle of them is certainly something that's going to be part of a full UI for users.

[jakelishman]

Technically this struct format string doesn't match the definition of the C struct up top - exists is a char in the C definition and _Bool here - but in practice it would be pretty wild to have a system where _Bool wasn't ABI compatible with char.

[mtreinish]

I think I probably just copy and pasted both from V1 as this doesn't change between V1 and V2. IIRC we added this discrepancy between docs and the struct format string as part of your review in #10148: #10148 (comment) . I can update it to be _Bool in the docs if you prefer though.

[jakelishman]

Nah, it's fine, let's just leave it.

[jakelishman]

Strictly I guess this would go to n - 1?

[mrossinek]

I just gave this a review because I am also interested in this topic. I recall having looked into this some months ago when I was using skip_transpilation with the Estimator and ran into Qiskit/qiskit-ibm-runtime#338.

Unfortunately, I no longer have the code snippets that I was using at the time. That said, I think the proposed API here makes sense to me and would be reasonable to use (definitely better than how one would need to go about this right now) 👍

As per the permute_sparse_pauli_op method: I suppose the method here is fine. From experience I can say that I always work with SparsePauliOp and no other operator classes. Whether you need to support permuting other objects is up to you and might be better discussed in terms of API consistency within the quantum_info module.

FWIW a while ago I implemented a similar index permutation utility in the SparseLabelOp classes in Qiskit Nature: qiskit-community/qiskit-nature@d9599dd
Rather than relying on a composition with a permuted identity operator (as you do here) I chose to act directly on the underlying labels. Not sure if this is "better" but it might be easier to achieve especially thinking about the underlying PauliList which are just some numpy arrays that natively support axis permutations.

[ikkoham]

I will improve BaseEstimator after this PR. Some operator classes beyond SparsePauliOp are being discussed internally, so it can be a future issue.

[nonhermitian]

I played around with this and it works quite well.

[mtreinish]

I've updated this PR to not have the permute_sparse_pauli_op method anymore. I pushed a follow-up PR #10947 that adds an apply_layout method to SparsePauliOp that has the same functionality. It is a better fit on the operator class, and it also enables us to add similar methods to the other operators in the future if needed.

[jakelishman]

I just saw a couple of things in the documentation, but otherwise this looks good to me.

It makes more sense to me too to have the permute method on the operators, though we can always revisit it in the future; it's of course easier to add methods than to remove ones we don't like.