Workflow scheduling race condition

[jmchilton]

This piece of bioblend test code:

        self.gi.workflows.cancel_invocation(workflow_id, invocation_id)
        invocation = self.gi.workflows.show_invocation(workflow_id, invocation_id)
        self.assertEqual(invocation['state'], 'cancelled')

Causes this error transiently:

======================================================================
FAIL: test_cancelling_workflow_scheduling (TestGalaxyWorkflows.TestGalaxyWorkflows)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/galaxyproject/bioblend/tests/test_util.py", line 56, in wrapped_method
    return method(has_gi, *args, **kwargs)
  File "/home/travis/build/galaxyproject/bioblend/tests/test_util.py", line 56, in wrapped_method
    return method(has_gi, *args, **kwargs)
  File "/home/travis/build/galaxyproject/bioblend/tests/TestGalaxyWorkflows.py", line 98, in test_cancelling_workflow_scheduling
    self.assertEqual(invocation['state'], 'cancelled')
AssertionError: u'ready' != 'cancelled'

Which I believe is demonstrating the race condition described as follows:

17:17 < jmchilto1> this looks like a bug in galaxy - a race condition
17:17 < jmchilto1> one thread marks it as canceled and another as ready
17:17 < jmchilto1> nsoranzo: if you rerun this test does it usually pass?
17:18 < nsoranzo> Yes
17:18 < jmchilto1> that is a bug - we have a test that found a bug 
17:18 < jmchilto1> project test stuff is a success
17:19 < dave_b> awesome, the tests have served their purpose. We can delete them now.
17:19 < jmchilto1> nsoranzo: how about we just throw a skip if that is ready instead of 
                   cancelled
17:20 < nsoranzo> I thought it was a delay in getting the state to the database before the 
                  new call to show_invocation()
17:21 < jmchilto1> I think it is two threads wirting the state in galaxy. I don't think a 
                   delay would help
17:21 < jmchilto1> when that cancel call is complete - the db is almost certainly flushed
17:21 < jmchilto1> so show should be callable right away
17:22 < jmchilto1> on the other hand - one can easily imagine two galaxy threads - one 
                   responding to the cancel and the workflow scheduler - each with a copy 
                   of the invocation
17:23 < jmchilto1> they each geet it in state 'new' - cancel changed it to 'cancelled' and 
                   the scheduler changes it to 'ready' - two seconds later it could be 
                   flushed as either

[mvdbeek]

If this was still a problem I'd assume 2a11420 is going to fix it