more useful output for update_fake_backends.py #8336

1ucian0 · 2022-07-13T10:16:01Z

This PR extends tools/update_fake_backends.py to warning the user when the snapshot has errors of 1.

@nonhermitian noticed that some calibration data was bad when I was creating FakeGeneva in #8322 (comment)

Indeed, many property json files contain gates with gate_error==1

$ rgrep '"gate_error", "unit": "", "value": 1}' qiskit -l
qiskit/providers/fake_provider/backends/rochester/props_rochester.json
qiskit/providers/fake_provider/backends/washington/props_washington.json
qiskit/providers/fake_provider/backends/cambridge/props_cambridge_alt.json
qiskit/providers/fake_provider/backends/cambridge/props_cambridge.json
qiskit/providers/fake_provider/backends/manhattan/props_manhattan.json

Moreover, many exploratory backends have errors==1 somewhere, but some calibrations are better than others. So it is important to show the relative amount of gates with error==1.

Additionally, the script tools/update_fake_backends.py might silently fail and it is hard to see why is not doing what it suppose to do. Take the situation of trying to update a backend you dont have access to. It runs without any output, but does not generate the expected files.

In this PR, the script outputs when some of the args.backends did not get updated or if some of the json files were skipped.

This is how it looks:

$ ./tools/update_fake_backends.py --hub <redacted> ibm_washington

ibm_washington
==============
Readout errors == 1:
 * qubit 3
Gate errors == 1:
 * cx 96, 109
 * cx 109, 96
 * cx 118, 119
 * cx 3, 4
 * cx 10, 9
 * cx 9, 10
 * cx 119, 118
 * cx 4, 3
╒═══════════════╤═════════════════════════════════╤═════════════════════════════════════╕
│    error == 1 │  new 2022-07-24 09:02:23+02:00  │  current 2022-04-12 23:42:47+09:00  │
╞═══════════════╪═════════════════════════════════╪═════════════════════════════════════╡
│ readout error │          1/127 (0.79%)          │            0/127 (0.00%)            │
├───────────────┼─────────────────────────────────┼─────────────────────────────────────┤
│      cx error │          8/284 (2.82%)          │            6/284 (2.11%)            │
├───────────────┼─────────────────────────────────┼─────────────────────────────────────┤
│     all gates │          8/919 (0.87%)          │            6/919 (0.65%)            │
╘═══════════════╧═════════════════════════════════╧═════════════════════════════════════╛

In addition, I added a parameter --datetime to be able to fetch specific snapshots, like suggested here.

qiskit-bot · 2022-07-13T10:16:04Z

Thank you for opening a new pull request.

Before your PR can be merged it will first need to pass continuous integration tests and be reviewed. Sometimes the review process can be slow, so please be patient.

While you're waiting, please feel free to review other open PRs. While only a subset of people are authorized to approve pull requests for merging, everyone is encouraged to review open pull requests. Doing reviews helps reduce the burden on the core team and helps make the project's code better for everyone.

One or more of the the following people are requested to review this:

@Qiskit/terra-core

coveralls · 2022-07-13T10:43:55Z

Pull Request Test Coverage Report for Build 3146527252

0 of 0 changed or added relevant lines in 0 files are covered.
11 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.003%) to 84.492%

Files with Coverage Reduction	New Missed Lines	%
qiskit/compiler/transpiler.py	11	89.71%

Totals
Change from base Build 3146373335:	0.003%
Covered Lines:	60199
Relevant Lines:	71248

💛 - Coveralls

…e_backends/no_props

jakelishman

The additional information coming out of the script certainly looks useful to me to help catch broken fake backends as they're generated. Now that it's got rather more complicated, I think it wouldn't hurt if we could write a bit more documentation into the script about what it outputs, why those statistics are important, and a mention of when it's ok to accept a calibration that's got some broken elements in it.

jakelishman · 2022-08-16T21:56:48Z

tools/update_fake_backends.py

            if config:
                config_path = os.path.join(args.dir, name, "conf_%s.json" % name)
                config_dict = config.to_dict()

                with open(config_path, "w") as fd:
                    fd.write(json.dumps(config_dict, cls=BackendEncoder))
+                summary.conf_msg = f"conf_{name}.json okey"


These lines seem a little pointless, since the only way they can not get set is if the script throws some exception and exits without printing anything. If you do want to keep them, a minor spelling note: "ok" or "okay", but not "okey".

I changed it to "ok" and clarified the alternative path in a60833b

jakelishman · 2022-08-16T21:59:02Z

tools/update_fake_backends.py

+            output += [f"{line}" for line in self.readout_summary]
+
+        if self.gate_summary:
+            output.append("Gate errors == 1:")
+            output += [f"{line}" for line in self.gate_summary]


Largely irrelevant, but f"{line}" is a bit redundant since line is already a string! output.extend(self.gate_summary) ought to work fine, I think? (And similar for the others.)

ups! good catch. It was from the time I was doing some formatting in each line. Fixed in a2a1828

tools/update_fake_backends.py

…e_backends/no_props

1ucian0 · 2022-09-16T16:14:00Z

I added some extra explanation and follow the advice in #8336 (comment)

ibm_washington
==============
Broken operations (gates with errors == 1)
 * cx 96, 109
 * cx 109, 96
 * cx 118, 119
 * cx 3, 4
 * cx 10, 9
 * cx 9, 10
 * cx 119, 118
 * cx 4, 3
╒═════════════════════════════════════╤═════════════════════════════════╤═════════════════════════════════════╕
│                                     │  new 2022-07-23 23:10:17+02:00  │  current 2022-04-11 14:24:11+02:00  │
╞═════════════════════════════════════╪═════════════════════════════════╪═════════════════════════════════════╡
│                broken cx / total cx │          8/284 (2.82%)          │            6/284 (2.11%)            │
├─────────────────────────────────────┼─────────────────────────────────┼─────────────────────────────────────┤
│ broken operations / total operation │          8/919 (0.87%)          │            6/919 (0.65%)            │
╘═════════════════════════════════════╧═════════════════════════════════╧═════════════════════════════════════╛
The calibration you are pulling (in the "new" column) contain some broken error values (when the error is 1). However, if the rate of broken errors is better than the current one, you might want to update the files.

HuangJunye · 2022-11-24T11:23:05Z

tools/update_fake_backends.py

+
+            sys.path.append(args.dir)
+            backend_mod = import_module(name)
+            fake_backend = getattr(backend_mod, "Fake" + name.capitalize())()


I got this error for ibm_oslo which is related to what I commented in the other PR.

File "/Users/junye/Documents/GitHub/qiskit-dev/qiskit-terra/./tools/update_fake_backends.py", line 300, in _main summary = _Summary(props, fake_backend.properties()) AttributeError: 'FakeOslo' object has no attribute 'properties'

Here you assume that Fake{Backend} is a V1 backend and it has properties attribute. But FakeOslo is actually a V2 backend.

This script should probably be updated for V2 backend

Let's leave that for another PR, since it'll be a bit more complicated, most likely. Right now, we can just skip the backend if the one we're given is V2.

----- Co-authored-by: "Lev S. Bishop" <[email protected]> Co-authored-by: Junye Huang <[email protected]> Co-authored-by: Luciano Bello <[email protected]> Co-authored-by: Matthew Treinish <[email protected]> Co-authored-by: Thomas Alexander <[email protected]>

1ucian0 · 2023-12-06T12:18:00Z

Closing in favor of Qiskit/qiskit-ibm-runtime#1262 as IBM specific fake backends are moving to https://github.com/Qiskit/qiskit-ibm-runtime

1ucian0 requested a review from a team as a code owner July 13, 2022 10:16

1ucian0 added Changelog: None Do not include in changelog mod: fake_provider Related to the fake_provider module and fake backends labels Jul 13, 2022

1ucian0 mentioned this pull request Jul 17, 2022

warning in update_fake_backends.py when the snapshot has errors of 1 #8331

Closed

1ucian0 marked this pull request as draft July 17, 2022 13:55

extended output

b0ccdf6

1ucian0 force-pushed the update_fake_backends/no_props branch from 5c2ee92 to b0ccdf6 Compare July 17, 2022 14:28

1ucian0 marked this pull request as ready for review July 17, 2022 14:54

Merge branch 'main' of github.com:Qiskit/qiskit-terra into update_fak…

2591c43

…e_backends/no_props

1ucian0 marked this pull request as draft July 25, 2022 12:21

1ucian0 added 6 commits July 25, 2022 22:55

even better output

f116f98

when new is not as good as curent

5978ba9

Merge branch 'main' of github.com:Qiskit/qiskit-terra into update_fak…

082e68b

…e_backends/no_props

new parameter datetime

efabb5c

portable import

c192c1c

Merge branch 'main' into update_fake_backends/no_props

ced4b10

1ucian0 marked this pull request as ready for review July 26, 2022 12:23

Merge branch 'main' into update_fake_backends/no_props

9bc0ef9

HuangJunye self-assigned this Aug 5, 2022

jakelishman reviewed Aug 16, 2022

View reviewed changes

HuangJunye assigned jakelishman and unassigned HuangJunye Aug 17, 2022

1ucian0 added 3 commits September 16, 2022 17:46

Merge branch 'main' of github.com:Qiskit/qiskit-terra into update_fak…

a26ae85

…e_backends/no_props

okey

a60833b

extend

a2a1828

1ucian0 added 2 commits September 16, 2022 18:15

better output

f80d1ee

Merge branch 'main' into update_fake_backends/no_props

ce337d4

HuangJunye reviewed Nov 24, 2022

View reviewed changes

1ucian0 mentioned this pull request Dec 6, 2023

Moving update_fake_backends.py from Qiskit Qiskit/qiskit-ibm-runtime#1262

Closed

1ucian0 closed this Dec 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

more useful output for update_fake_backends.py #8336

more useful output for update_fake_backends.py #8336

1ucian0 commented Jul 13, 2022 •

edited

Loading

qiskit-bot commented Jul 13, 2022

coveralls commented Jul 13, 2022 •

edited

Loading

jakelishman left a comment

jakelishman Aug 16, 2022

1ucian0 Sep 16, 2022

jakelishman Aug 16, 2022

1ucian0 Sep 16, 2022

1ucian0 commented Sep 16, 2022

HuangJunye Nov 24, 2022

HuangJunye Nov 24, 2022

jakelishman Nov 24, 2022

1ucian0 commented Dec 6, 2023 •

edited

Loading

more useful output for update_fake_backends.py #8336

more useful output for update_fake_backends.py #8336

Conversation

1ucian0 commented Jul 13, 2022 • edited Loading

qiskit-bot commented Jul 13, 2022

coveralls commented Jul 13, 2022 • edited Loading

Pull Request Test Coverage Report for Build 3146527252

💛 - Coveralls

jakelishman left a comment

Choose a reason for hiding this comment

jakelishman Aug 16, 2022

Choose a reason for hiding this comment

1ucian0 Sep 16, 2022

Choose a reason for hiding this comment

jakelishman Aug 16, 2022

Choose a reason for hiding this comment

1ucian0 Sep 16, 2022

Choose a reason for hiding this comment

1ucian0 commented Sep 16, 2022

HuangJunye Nov 24, 2022

Choose a reason for hiding this comment

HuangJunye Nov 24, 2022

Choose a reason for hiding this comment

jakelishman Nov 24, 2022

Choose a reason for hiding this comment

1ucian0 commented Dec 6, 2023 • edited Loading

1ucian0 commented Jul 13, 2022 •

edited

Loading

coveralls commented Jul 13, 2022 •

edited

Loading

1ucian0 commented Dec 6, 2023 •

edited

Loading