Add stage plugin interface for transpile

[mtreinish]

Summary

This commit adds a new plugin interface to qiskit for enabling external
packages to write plugins that will replace a stage in the transpilation
pipeline. For example, if an external package had a custom layout pass
that they wanted to integrate into transpile() they could export a
plugin using this new interface and then users would just need to run

transpile(.., layout_method=foo)

This adds long asked for extensibility to the transpiler so that to
cleanly integrate new transpiler passes we're no longer required to
merge the features into terra. This should hopefully make it easier for
downstream pass authors to integrate their passes into terra and make it
easier for the terra maintainers to evaluate new transpiler passes.

Details and comments

TODO:

• Add release notes
• Fix docs build
• Add support to opt level 0 and 3
• Add test coverage for plugin functions
• Rework some stages as built-in plugins?

[qiskit-bot]

Thank you for opening a new pull request.

Before your PR can be merged it will first need to pass continuous integration tests and be reviewed. Sometimes the review process can be slow, so please be patient.

While you're waiting, please feel free to review other open PRs. While only a subset of people are authorized to approve pull requests for merging, everyone is encouraged to review open pull requests. Doing reviews helps reduce the burden on the core team and helps make the project's code better for everyone.

One or more of the the following people are requested to review this:

• @Qiskit/terra-core

[coveralls]

Pull Request Test Coverage Report for Build 2958142026

• 269 of 302 (89.07%) changed or added relevant lines in 9 files are covered.
• 3 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.01%) to 84.139%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
qiskit/transpiler/preset_passmanagers/builtin_plugins.py	87	91	95.6%
qiskit/transpiler/preset_passmanagers/level0.py	21	26	80.77%
qiskit/transpiler/preset_passmanagers/level1.py	30	35	85.71%
qiskit/transpiler/preset_passmanagers/level2.py	28	33	84.85%
qiskit/transpiler/preset_passmanagers/plugin.py	49	54	90.74%
qiskit/transpiler/preset_passmanagers/level3.py	38	47	80.85%

Files with Coverage Reduction	New Missed Lines	%
qiskit/pulse/library/waveform.py	3	89.36%

Totals
Change from base Build 2958133494:	0.01%
Covered Lines:	56980
Relevant Lines:	67721

---

💛 - Coveralls

[alexanderivrii]

@mtreinish. This looks very useful. I am trying to better understand what we would or would not be able to do, so a few questions, if I may:

• Suppose we want to run transpile with optimization_level=2 but replace the layout stage by a layout method that is already in Qiskit (e.g., "noise_adaptive"). In this case, we would not need to write any plugin, and would simply pass layout_method = 'noise_adaptive' as an argument to transpile. This is correct so far?
• Suppose we want to run transpile with optimization_level=2 but replace the layout stage by a layout method that is not part of Qiskit. Then how to do it is precisely your example in plugin.py, correct?
• The above is the same for the optimization stage (instead of the layout stage), for instance if we want to replace the optimization stage by an algorithm that is not part of Qiskit. But suppose we don't want to fully replace the optimization stage, but simply add the external algorithm to the current set of optimization algorithms. Can we do that?
• We already have plugins for unitary synthesis algorithms, and unitary synthesis is used both in the init and optimization stages, I think. Should we specify unitary synthesis plugins independently, or as part of the stages?
• Suppose that we want to integrate an external Clifford synthesis algorithm in Qiskit. In some sense, this would be similar to how we do unitary synthesis. What would be the steps to do that? If this derails the discussion too much, we can take this offline or to a different thread.

[mtreinish]

@mtreinish. This looks very useful. I am trying to better understand what we would or would not be able to do, so a few questions, if I may:

* Suppose we want to run `transpile` with `optimization_level=2` but replace the `layout` stage by a layout method that is already in Qiskit (e.g., "noise_adaptive"). In this case, we would not need to write any plugin, and would simply pass `layout_method = 'noise_adaptive'` as an argument to `transpile`. This is correct so far?

Yes, that's correct. One thing I'm debating is whether I want to bundle things like noise_adaptive layout as a plugin (so we're using our own plugin interface) or just leave it as a standalone thing. But that's an implementation detail and doesn't effect the end user interface or what we include with terra.

* Suppose we want to run `transpile` with `optimization_level=2` but replace the `layout` stage by a layout method that is not part of Qiskit. Then how to do it is precisely your example in `plugin.py`, correct?

Yes exactly, the idea is exactly that. A custom layout method (or any stage) that isn't part of qiskit can cleanly integrate into transpile() by creating a plugin. Then the end user can just do transpile(layout_method='custom') (or whatever the plugin name is) and it will work just as if it were one of the built in layout methods.

* The above is the same for the optimization stage (instead of the layout stage), for instance if we want to replace the optimization stage by an algorithm that is not part of Qiskit. But suppose we don't want to fully replace the optimization stage, but simply add the external algorithm to the current set of optimization algorithms. Can we do that?

Not with this, the unit of a plugin is a stage. This enables an external package to advertise that they have a custom pass manager which can be used for an optimization stage. So using this model you'd have to have your plugin return the full pass manager including the existing default stage contents. That being said in the last release we made this sort of modification easy to do. For example something like:

from qiskit.transpiler.preset_passmanager import generate_preset_pass_manager
from qiskit.providers.fake_provider import FakeLagosV2

backend = FakeLagosV2()
pass_manager = generate_preset_pass_manager(3, backend)
pass_manager.optimization.append(CustomPass())
pass_manager.run(qc)

You could embed something similar in your plugin but optimization is a weird case because there isn't a handy generator function like the other stages as each optimization level is custom. I guess you could have your plugin call generate_preset_pass_manager() and return a modified .optimization from it.

* We already have plugins for unitary synthesis algorithms, and unitary synthesis is used both in the `init` and `optimization` stages, I think. Should we specify unitary synthesis plugins independently, or as part of the stages?

Unitary synthesis plugins are already handled separately with an existing (for about a year now) plugin interface. https://qiskit.org/documentation/apidoc/transpiler_plugins.html?highlight=plugin#module-qiskit.transpiler.passes.synthesis.plugin this enables synthesis method authors to hook into the existing UnitarySynthesis pass at a lower level and replace how the circuits are synthesized for a unitary. I guess I'm saying this doesn't supersede this already existing plugin interface, it just gives people another hook point to swap out a larger chunk of functionality from the transpiler to an external package.

Since the unit of the plugins added in this PR are stages the specific passes used as part of the returned PassManager is up to the plugin author. We pass the full context the preset pass managers get (the PassManagerConfig) to the plugin so if the plugin stage is using the UnitarySynthesis pass the burden is on the plugin author to ensure the user specified unitary_synthesis_method argument is passed to the constructor for UnitarySynthesis in the returned stage PassManager.

* Suppose that we want to integrate an external Clifford synthesis algorithm in Qiskit. In some sense, this would be similar to how we do unitary synthesis. What would be the steps to do that? If this derails the discussion too much, we can take this offline or to a different thread.

My plan for clifford synthesis was to probably add a specific plugin interface for this once the dedicated clifford synthesis pass exists. We have this mechanism specifically for unitary synthesis already so having a dual for clifford synthesis makes sense to me. You could leverage this using this interface though and write a custom plugin for the init or translation stage that used a custom synthesis pass if you wanted, but that would add a lot of boiler plate if all you're really looking for is an interface that gets a black box plugin/function that takes a clifford in and returns a dag circuit.

[mtreinish]

This is ready for review now, I've added the pieces I thought were missing.

The only open question is whether I should massage the additional built-in stages for the preset pass managers (init, layout, translation, optimization, and scheduling) into plugins. I started with just routing since it was fairly self contained. Although given the size of this PR already we can save the plugin refactor for the remaining stages into follow up PRs.

[alexanderivrii]

@mtreinish, can I volunteer to review this (partly for self-educational purposes)?

[1ucian0]

Is there any plan to replace *_method with the plugin interface?

[1ucian0]

I think this looks good. I would add some more testing of the included plugins (therefore, the request to change). And I'm still uncomfortable with the level being part of the PassManagerConfig (non-blocking). The rest is good to merge for me.

[mtreinish]

I think this looks good. I would add some more testing of the included plugins (therefore, the request to change). And I'm still uncomfortable with the level being part of the PassManagerConfig (non-blocking). The rest is good to merge for me.

@1ucian0 I moved the optimization level out of the PassManagerConfig in d731d06 and add test coverage for the built-in plugins in f185c59

[kdk]

Is there somewhere we've documented how to implement a transpiler stage from the perspective of a pass developer or backend vendor?

[kevinhartman]

It's probably worth documenting this as something for plugin authors to watch out for. Alternatively, I guess we can detect the duplicates right after we build the extension manager object and raise. What do you think?

Documenting would be nice. Is there a way we could log a warning when the transpilation is run to log which one was selected? I think raising would be troublesome since it'd force users to uninstall something to continue. Packages could contain multiple packages, so forcing the user to uninstall might leave them in a situation where it's impossible to set up the pipeline they want.

[kdk]

Is there somewhere we've documented how to implement a transpiler stage from the perspective of a pass developer or backend vendor?

@mtreinish pointed me to https://github.com/Qiskit/qiskit-terra/blob/6d344c068c7e3605d735f3efa654b9f1e406a0cb/qiskit/transpiler/preset_passmanagers/plugin.py#L89 .

[1ucian0]

Thanks!