Incorrect Properties in FakeBackends #3735
Labels
Comments
How do you image this? Something like |
|
I was thinking about something similar to https://github.com/luisg5/qiskit-ibmq-provider/blob/4e85e660ed3fb869b08242cc803bb5c7d4e10910/qiskit/providers/ibmq/api_v2/rest/backend.py#L40 where a datetime object can be passed to the properties function to filter the returned property data. I see two options how this can be implemented for fake backends:
One challenge is to get the historical property data in the first place. Should it be randomly generated or dumped from IBM Q backends? |
mtreinish
added a commit
to mtreinish/qiskit-core
that referenced
this issue
Mar 25, 2020
Right now there is nothing in validating that we're able to use all the fake backends defined in qiskit.test.mock.backends. So when we make changes to the fake backends (or backends in general) we don't know if they work until someone goes to use them. This was especially needed for Qiskit#4016. During the development of that PR all the tests pass but the docs builds failed because a fake backend is used in a jupyter-execute section. This commit addresses the situation by adding a simple test that just runs a simple deterministic circuit that just has some easy 1q gate run optimizations on the fake backend at each optimization level (to ensure we use all the backend properties needed for execution). This will provide us coverage over the fake backends to ensure that they at least are functional as we make changes to terra. As part of the development of these tests, 2 missing properties from the new qobj classes were added because the tests caught that they were missing. Additionally the properties json file for fake yorktown had to be updated, the properties there were incorrect when run with aer (and thus a noise model) all the results were invalid. Fixes Qiskit#3735
ajavadia
pushed a commit
that referenced
this issue
Mar 25, 2020
* Add tests executing a circuit on each fake backend Right now there is nothing in validating that we're able to use all the fake backends defined in qiskit.test.mock.backends. So when we make changes to the fake backends (or backends in general) we don't know if they work until someone goes to use them. This was especially needed for #4016. During the development of that PR all the tests pass but the docs builds failed because a fake backend is used in a jupyter-execute section. This commit addresses the situation by adding a simple test that just runs a simple deterministic circuit that just has some easy 1q gate run optimizations on the fake backend at each optimization level (to ensure we use all the backend properties needed for execution). This will provide us coverage over the fake backends to ensure that they at least are functional as we make changes to terra. As part of the development of these tests, 2 missing properties from the new qobj classes were added because the tests caught that they were missing. Additionally the properties json file for fake yorktown had to be updated, the properties there were incorrect when run with aer (and thus a noise model) all the results were invalid. Fixes #3735 * Add qiskit-aer to ci jobs To fully test the fake backend properties we need to use aer to build a noise model and run the simulation with that. This commit adds qiskit-aer to the ci configuration so that it is a vailable for these new tests. In general we don't want to rely on aer (because of the potential circular dependency) but for this case it's seems worth the tradeoff. * Reduce error rates on fake_openpulse_2q backend When running simulations with a noise model the tests were failing on fake_openpulse_2q because it was set as being too noisy. This commit drops the error rates set in the properties so we can run a circuit with a single x gate and get the expected result. * Handle simulators properly for fake backends The fake_qasm_simulator is set as a simulator, but the base fake_provider class was not setup to handle a backend which was a simulator. This commit handles this by having the fake_qasm_simulator return no properties, and also updates the run() method to not try and build a noise model for a simulator. * Adjust tests to use fake provider This commit adjusts the tests to use the fake provider to get a list of backends instead of scanning the backends module for Fake* objects. * Change fake_qasm_simulator basis to basic aer's For fake_qasm_simulator's run() method top work when aer is not installed it needs to be usable with the basic aer simulator. However, right now the basis set was closer to aer's. This commit adjusts it to be basic aer's which is LCD for running in a simulator. * Cleanup arg name for test
faisaldebouni
pushed a commit
to faisaldebouni/qiskit-terra
that referenced
this issue
Aug 5, 2020
* Add tests executing a circuit on each fake backend Right now there is nothing in validating that we're able to use all the fake backends defined in qiskit.test.mock.backends. So when we make changes to the fake backends (or backends in general) we don't know if they work until someone goes to use them. This was especially needed for Qiskit#4016. During the development of that PR all the tests pass but the docs builds failed because a fake backend is used in a jupyter-execute section. This commit addresses the situation by adding a simple test that just runs a simple deterministic circuit that just has some easy 1q gate run optimizations on the fake backend at each optimization level (to ensure we use all the backend properties needed for execution). This will provide us coverage over the fake backends to ensure that they at least are functional as we make changes to terra. As part of the development of these tests, 2 missing properties from the new qobj classes were added because the tests caught that they were missing. Additionally the properties json file for fake yorktown had to be updated, the properties there were incorrect when run with aer (and thus a noise model) all the results were invalid. Fixes Qiskit#3735 * Add qiskit-aer to ci jobs To fully test the fake backend properties we need to use aer to build a noise model and run the simulation with that. This commit adds qiskit-aer to the ci configuration so that it is a vailable for these new tests. In general we don't want to rely on aer (because of the potential circular dependency) but for this case it's seems worth the tradeoff. * Reduce error rates on fake_openpulse_2q backend When running simulations with a noise model the tests were failing on fake_openpulse_2q because it was set as being too noisy. This commit drops the error rates set in the properties so we can run a circuit with a single x gate and get the expected result. * Handle simulators properly for fake backends The fake_qasm_simulator is set as a simulator, but the base fake_provider class was not setup to handle a backend which was a simulator. This commit handles this by having the fake_qasm_simulator return no properties, and also updates the run() method to not try and build a noise model for a simulator. * Adjust tests to use fake provider This commit adjusts the tests to use the fake provider to get a list of backends instead of scanning the backends module for Fake* objects. * Change fake_qasm_simulator basis to basic aer's For fake_qasm_simulator's run() method top work when aer is not installed it needs to be usable with the basic aer simulator. However, right now the basis set was closer to aer's. This commit adjusts it to be basic aer's which is LCD for running in a simulator. * Cleanup arg name for test
ElePT
pushed a commit
to ElePT/qiskit-ibm-provider
that referenced
this issue
Oct 9, 2023
* Add tests executing a circuit on each fake backend Right now there is nothing in validating that we're able to use all the fake backends defined in qiskit.test.mock.backends. So when we make changes to the fake backends (or backends in general) we don't know if they work until someone goes to use them. This was especially needed for Qiskit/qiskit#4016. During the development of that PR all the tests pass but the docs builds failed because a fake backend is used in a jupyter-execute section. This commit addresses the situation by adding a simple test that just runs a simple deterministic circuit that just has some easy 1q gate run optimizations on the fake backend at each optimization level (to ensure we use all the backend properties needed for execution). This will provide us coverage over the fake backends to ensure that they at least are functional as we make changes to terra. As part of the development of these tests, 2 missing properties from the new qobj classes were added because the tests caught that they were missing. Additionally the properties json file for fake yorktown had to be updated, the properties there were incorrect when run with aer (and thus a noise model) all the results were invalid. Fixes Qiskit/qiskit#3735 * Add qiskit-aer to ci jobs To fully test the fake backend properties we need to use aer to build a noise model and run the simulation with that. This commit adds qiskit-aer to the ci configuration so that it is a vailable for these new tests. In general we don't want to rely on aer (because of the potential circular dependency) but for this case it's seems worth the tradeoff. * Reduce error rates on fake_openpulse_2q backend When running simulations with a noise model the tests were failing on fake_openpulse_2q because it was set as being too noisy. This commit drops the error rates set in the properties so we can run a circuit with a single x gate and get the expected result. * Handle simulators properly for fake backends The fake_qasm_simulator is set as a simulator, but the base fake_provider class was not setup to handle a backend which was a simulator. This commit handles this by having the fake_qasm_simulator return no properties, and also updates the run() method to not try and build a noise model for a simulator. * Adjust tests to use fake provider This commit adjusts the tests to use the fake provider to get a list of backends instead of scanning the backends module for Fake* objects. * Change fake_qasm_simulator basis to basic aer's For fake_qasm_simulator's run() method top work when aer is not installed it needs to be usable with the basic aer simulator. However, right now the basis set was closer to aer's. This commit adjusts it to be basic aer's which is LCD for running in a simulator. * Cleanup arg name for test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Information
What is the current behavior?
All u2 and u3 gates in FakeYorktown have an error rate of 1.0 or 2.0. This leads to a complete depolarizing channel when used in basic_device_noise_model.
FakeTokyo has error rates for some cx gate, that are not defined in the coupling map (and vice versa).
FakeRueschlikon has no property file.
Steps to reproduce the problem
Simulate with basic_device_noise_model() or have a look at the .json files in https://github.com/Qiskit/qiskit-terra/blob/master/qiskit/test/mock/backends
What is the expected behavior?
FakeYorktown should not define a complete depolarization after each single-qubit gate. FakeTokyo should have missing cx gates and cx error rates added to the coupling map/property file and/or unused cx gates should be consistently removed from the .json file. FakeRueschlikon should have a property file.
Suggested solutions
Regenerate the .json files.
For FakeYorktown, I assume either an unit error (0.0002 becoming 2.0 or similar) or a snapshot of an incorrect calibration. FakeTokyo might have been build from different snapshots for the coupling map and the gate errors, thereby setting error rates for cx gates that are not defined in the coupling map and setting cx gates in the coupling map that have no error rate set.
As a side note, historical calibration data for FakeBackends would be nice. :-)
The text was updated successfully, but these errors were encountered: