Adding test suite for testing IBMQJob in isolation. #515

delapuente · 2018-05-28T14:20:25Z

The test suite does not require a network connection. It fakes the QJOB object and the API behaviour.

This commit partially refactors the IBMQJob#status() method to avoid raising exceptions while reading a property.

Motivation and Context

Current tests behave as integration tests so, in case of failure, it is not guaranteed the problem arises from IBMJob-related abstractions malfunctioning.

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

ajavadia · 2018-05-29T15:46:27Z

test/python/test_ibmqjob_states.py

+
+    def run_job(self, *_args, **_kwargs):
+        time.sleep(0.2)
+        return {'error': 'ivalid test qobj'}


typo here "invalid"

ewinston · 2018-05-29T17:54:40Z

qiskit/backends/ibmq/ibmqjob.py

+    # pylint: disable=arguments-differ
+    def result(self, timeout=60, wait=5):
+        self._wait_for_initialization(timeout=timeout)
+        result = self._wait_for_job(timeout=timeout, wait=wait)


I think the expectation for timeout will be the maximum time for the result call to return. Perhaps we can instead start a clock here (timeout_time = time.time() + timeout) and pass timeout_time to _wait_for_initialization and _wait_for_job.

ewinston · 2018-05-29T21:10:34Z

qiskit/backends/ibmq/ibmqjob.py

-            if self._future_submit.exception():
-                self._exception = self._future_submit.exception()
-            self._status_msg = str(self.exception)
+


How does this PR handle exceptions raised by self._future_submit which is done async and may occur later?

Notice that reading status will be stuck in the INITIALIZING state until the initialization routine _submit changes the internal job status.

All the errors happening inside the _submit() are silenced (except for one case I just realize, thanks!) and stored inside the _exception member, the internal status progress to JobStatus.ERROR in these cases.

This causes all future reads to status to skip querying the API since the job is in one of the final states.

Both tests test_status_flow_for_invalid_job and test_status_flow_for_throwing_api test the job in the cases when the job is invalid (and the API rejects it by raising an ApiError) or the network side fails somehow (and the API returns an errored result) respectively. Anyway, I'll reinforce the tests in the next commits, checking the exceptions to get more confidence in the paths we are following.

Hope it helps.

There is an error in my assumptions regarding _submit() progressing the job to a final state. This final assignment happening at the end of the method invalidates de second paragraph of the explanation. I'll fix in the next commit. Thanks for the heads-up.

delapuente · 2018-06-05T08:33:37Z

Everything is fixed now. What about a second review?

diego-plan9 · 2018-06-08T13:08:35Z

test/python/test_ibmqjob_states.py

+
+"""IBMQJob states test-suite."""
+
+import unittest


For the new files, can you adjust the license header so it follows the format recently introduced by #550 ?

# -*- coding: utf-8 -*- # Copyright 2018, IBM. # # This source code is licensed under the Apache License, Version 2.0 found in # the LICENSE.txt file in the root directory of this source tree. # pylint: disable=something-only-if-there-is-no-other-way """ Description of the module ... """ from qiskit import ...

@delapuente

This commit incorporates improvements from PR Qiskit#515 by @delapuente to handling job status such that it makes fewer api calls. Co-authored-by: [email protected]

atilag · 2018-06-18T13:37:47Z

@delapuente @ewinston not sure how to handle this PR. As this is going to test Jobs, and Ericks PR #501 (jobs from Backend) is hughly changing IBMQJob... we would need to have them sync.
My proposal is: ... as this is a PR about testing, I would merge this one fist and then rebase and solve conflicts on #501. Does it sound good?

ewinston · 2018-06-18T17:57:40Z

Since #501 is tagged as "high priority" and it's an earlier PR perhaps it should be the other way around.

ajavadia · 2018-06-21T01:41:18Z

@delapuente is this PR rebased after #501 got merged?

ajavadia · 2018-06-21T02:51:12Z

Does this solve #234?

delapuente · 2018-06-21T07:19:42Z

@delapuente is this PR rebased after #501 got merged?

Yes. Since the implementation of IBMQJob changed noticeably, I've discarded most of my changes from the original PR but that's the only difference.

Does this solve #234?

Not yet. It contributes to that but it is not replacing the original IBMQJob suite and more work is needed in other parts of the library.

atilag · 2018-06-21T08:52:44Z

qiskit/backends/ibmq/ibmqjob.py

+            this_result = self._wait_for_job(timeout=timeout, wait=wait)
+        except TimeoutError as err:
+            # A timeout error retrieving the results does not imply the job
+            # is failing. The job can be still running.


Good point, but the Result has an ERROR status, so for the user something wrong happened with this Job. Should we state in any way, that this may not be a real Job error?... like having another status for "TIMEOUT" or something similar.

Let's mark this for future improvements... we need to merge this asap.

See #588 for the follow-up.

When job cancellation works on all IBM Q systems timeout could use that to cancel the job, which could be considered an error.

The test suite does not require network connection. It fakes the QJOB object and the API behaviour.

diego-plan9

Thanks everyone for the effort!

* Adding test suite for testing IBMQJob in isolation. The test suite does not require network connection. It fakes the QJOB object and the API behaviour. * Removing QuantumJob * Adding expected failure when cancelling during initialization

delapuente requested review from ajavadia, atilag, diego-plan9 and ewinston as code owners May 28, 2018 14:20

ajavadia reviewed May 29, 2018

View reviewed changes

delapuente force-pushed the test-ibmqjob-in-isolation branch from 0becd31 to a105d5b Compare May 29, 2018 15:57

ewinston reviewed May 29, 2018

View reviewed changes

delapuente force-pushed the test-ibmqjob-in-isolation branch 2 times, most recently from ffd3ba0 to acdfd57 Compare June 4, 2018 15:22

diego-plan9 mentioned this pull request Jun 6, 2018

Complete the Jobs implementation #539

Closed

5 tasks

diego-plan9 added this to the 0.5.x milestone Jun 6, 2018

delapuente force-pushed the test-ibmqjob-in-isolation branch 4 times, most recently from a3fd9e7 to e13056d Compare June 8, 2018 07:47

diego-plan9 reviewed Jun 8, 2018

View reviewed changes

delapuente force-pushed the test-ibmqjob-in-isolation branch from e13056d to bf0efec Compare June 8, 2018 17:35

ewinston added a commit to ewinston/qiskit that referenced this pull request Jun 15, 2018

improvements to ibmqjob.status

10d0bc2

This commit incorporates improvements from PR Qiskit#515 by @delapuente to handling job status such that it makes fewer api calls. Co-authored-by: [email protected]

delapuente force-pushed the test-ibmqjob-in-isolation branch from bf0efec to 9eaf078 Compare June 18, 2018 10:30

atilag added the priority: high label Jun 19, 2018

delapuente force-pushed the test-ibmqjob-in-isolation branch from f5daa11 to 06c3a10 Compare June 20, 2018 15:19

atilag previously approved these changes Jun 21, 2018

View reviewed changes

Adding test suite for testing IBMQJob in isolation.

3eea61c

The test suite does not require network connection. It fakes the QJOB object and the API behaviour.

Removing QuantumJob

290e28d

delapuente dismissed atilag’s stale review via 290e28d June 21, 2018 15:22

delapuente force-pushed the test-ibmqjob-in-isolation branch from 06c3a10 to 290e28d Compare June 21, 2018 15:22

delapuente and others added 2 commits June 21, 2018 17:31

Adding expected failure when cancelling during initialization

a6864da

Merge branch 'master' into test-ibmqjob-in-isolation

4edb3fd

diego-plan9 approved these changes Jun 21, 2018

View reviewed changes

diego-plan9 merged commit 2cbbf78 into Qiskit:master Jun 21, 2018

delapuente deleted the test-ibmqjob-in-isolation branch November 8, 2018 15:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding test suite for testing IBMQJob in isolation. #515

Adding test suite for testing IBMQJob in isolation. #515

delapuente commented May 28, 2018 •

edited

Loading

ajavadia May 29, 2018

ewinston May 29, 2018

ewinston May 29, 2018

delapuente May 30, 2018

delapuente May 30, 2018

delapuente commented Jun 5, 2018

diego-plan9 Jun 8, 2018

atilag commented Jun 18, 2018

ewinston commented Jun 18, 2018

ajavadia commented Jun 21, 2018

ajavadia commented Jun 21, 2018

delapuente commented Jun 21, 2018 •

edited

Loading

atilag Jun 21, 2018

atilag Jun 21, 2018

delapuente Jun 21, 2018

ewinston Jun 21, 2018

diego-plan9 left a comment

Adding test suite for testing IBMQJob in isolation. #515

Adding test suite for testing IBMQJob in isolation. #515

Conversation

delapuente commented May 28, 2018 • edited Loading

Motivation and Context

Checklist:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

delapuente commented Jun 5, 2018

Choose a reason for hiding this comment

atilag commented Jun 18, 2018

ewinston commented Jun 18, 2018

ajavadia commented Jun 21, 2018

ajavadia commented Jun 21, 2018

delapuente commented Jun 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diego-plan9 left a comment

Choose a reason for hiding this comment

delapuente commented May 28, 2018 •

edited

Loading

delapuente commented Jun 21, 2018 •

edited

Loading