ExperimentData tests assume that .data() is ordered
#1044
Labels
bug
Something isn't working
Milestone
Comments
wshanks
added a commit
to wshanks/qiskit-experiments
that referenced
this issue
Sep 7, 2023
Previously, the test `test_add_data_job` called `add_jobs` with two jobs and then tested that the results from `ExperimentData.data()` matched the results of the first job concatenated with the second. `ExperimentData` uses threads to handle tracking jobs. Even though these jobs are dummy jobs that take no time, `ExperimentData` would occasionally add the results from the second job before the first and then fail the results comparison. With this change, a label is attached to each result and then the results are sorted based on the label before testing against the expected results. Closes qiskit-community#1044
github-merge-queue bot
pushed a commit
that referenced
this issue
Sep 7, 2023
Previously, the test `test_add_data_job` called `add_jobs` with two jobs and then tested that the results from `ExperimentData.data()` matched the results of the first job concatenated with the second. `ExperimentData` uses threads to handle tracking jobs. Even though these jobs are dummy jobs that take no time, `ExperimentData` would occasionally add the results from the second job before the first and then fail the results comparison. With this change, a label is attached to each result and then the results are sorted based on the label before testing against the expected results. Closes #1044
nkanazawa1989
pushed a commit
to nkanazawa1989/qiskit-experiments
that referenced
this issue
Jan 10, 2024
…munity#1267) Previously, the test `test_add_data_job` called `add_jobs` with two jobs and then tested that the results from `ExperimentData.data()` matched the results of the first job concatenated with the second. `ExperimentData` uses threads to handle tracking jobs. Even though these jobs are dummy jobs that take no time, `ExperimentData` would occasionally add the results from the second job before the first and then fail the results comparison. With this change, a label is attached to each result and then the results are sorted based on the label before testing against the expected results. Closes qiskit-community#1044
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Informations
What is the current behavior?
Several of the ExperimentData tests assume that
ExperimentData.data()returns data in a reliable order. This is not the case.Steps to reproduce the problem
One example is this test:
https://github.com/Qiskit/qiskit-experiments/blob/c0cc88c244bc4e41fd3355d76ff2438a10dbac43/test/database_service/test_db_experiment_data.py#L119-L140
It can fail as was the case here.
The problem is that
ExperimentDataputs jobs that are added to it into a thread pool that waits for their results to be available and then adds them to an internal list that holds all the results. The test adds jobs very quickly and apparently sometimes the second job can come out of the thread pool before the first.What is the expected behavior?
The tests should not rely on the order of data returned by
ExperimentData.data().Suggested solutions
In practice, the analysis code usually attaches labels to the metadata of each circuit which is then attached to each result. The data is then analyzed using these labels, so the order of the data returned by
data()does not matter. One solution would be to add labels to the fake results used in the tests and then sort by those labels before comparing to the expected results.The text was updated successfully, but these errors were encountered: