Extracting a workflow from a history results in a failure

[jennaj]

Might impact more than main

test history I tried to extract from: https://usegalaxy.org/u/jen/h/16s-microbial-analysis-with-mothur---proccess

If error result are included, this error: GURU MEDITATION: #c18051a783074ee2ac82b0c21fd5d680

If error results are not included (deselected): GURU MEDITATION: #90d8f12fe45e4166a9b84e58fd61ae7a

Not sure what else could be triggering the problem. Possibly updated tools?

ping @natefoo @jmchilton @bgruening

[mvdbeek]

This is a galaxy bug:

Exception: Failed to extract job.
  File "galaxy/web/framework/middleware/sentry.py", line 43, in __call__
    iterable = self.application(environ, start_response)
  File "/cvmfs/main.galaxyproject.org/venv/lib/python2.7/site-packages/paste/recursive.py", line 85, in __call__
    return self.application(environ, start_response)
  File "galaxy/web/framework/middleware/statsd.py", line 35, in __call__
    req = self.application(environ, start_response)
  File "/cvmfs/main.galaxyproject.org/venv/lib/python2.7/site-packages/paste/httpexceptions.py", line 640, in __call__
    return self.application(environ, start_response)
  File "galaxy/web/framework/base.py", line 136, in __call__
    return self.handle_request(environ, start_response)
  File "galaxy/web/framework/base.py", line 215, in handle_request
    body = method(trans, **kwargs)
  File "galaxy/webapps/galaxy/controllers/workflow.py", line 952, in build_from_current_history
    dataset_collection_names=dataset_collection_names
  File "galaxy/workflow/extract.py", line 29, in extract_workflow
    steps = extract_steps(trans, history=history, job_ids=job_ids, dataset_ids=dataset_ids, dataset_collection_ids=dataset_collection_ids, dataset_names=dataset_names, dataset_collection_names=None)
  File "galaxy/workflow/extract.py", line 150, in extract_steps
    raise Exception("Failed to extract job.")

FWIW, this warning message is logged just before raising the exception:

u'Failed to find matching implicit job - job id is 17866205, implicit pairs are [(u'logfile', <galaxy.model.HistoryDatasetCollectionAssociation object at 0x7f2ae06b19d0>), (u'biomfiles', <galaxy.model.HistoryDatasetCollectionAssociation object at 0x7f2ae06b1790>)], assoc_name is __new_primary_file_0.03|0.03__.'

Looks like something related to the collection status improvements in 18.01, any idea what this could be @jmchilton ?

[jmchilton]

It looks to me like the history has manually "unhidden" datasets that are attempting to be included the workflow extraction - I have never tested this but it seems problematic. My guess is that never worked but might be related to 18.01? Can anyone confirm this is a regression?

[jennaj]

I haven't tested in an earlier version -- did you want me to or someone else?

Another workflow extraction issue came up when the history contains the tool Quast. The general consensus is that it is not the tool but the extraction process itself. Seems related or at least might be part of the fix for hidden/unhidden dataset extraction. Details here include sentry logs: #109

And one final related item - in the GUI after extracting a workflow successfully - the "run" link in blue box points to a dead link. Seems like a bug but not sure if server-specific or general. @guerler is this a known or is putting the issue here ok or do you want a distinct ticket or should we add this as a fix/enhancement for the "New" GUI? (multiple choice, ha :)

Tiny test history that shows the behavior: https://usegalaxy.org/u/jen/h/test-history-import-export

To reproduce: Exact a workflow, then click on the run link. The edit link works fine.

After extracting:

After clicking on run:

[jennaj]

Retesting

[jennaj]

@guerler @jmchilton @dannon The "run" link from an extracted workflow is still broken. Can use test history https://usegalaxy.org/u/jen/h/test-history-import-export to test this on main (very small, no collections). Is this fix going to be included in 18.05?

[jennaj]

@jmchilton @mvdbeek @dannon This history still produces an error upon extraction: https://usegalaxy.org/u/jen/h/16s-microbial-analysis-with-mothur---proccess

GURU MEDITATION: #e4ea722ec1954f0b9c1aafc3af2bcc30

Was the fix for manual unhidden datasets included in 18.05? Has this been updated on main yet to integration test?

[jennaj]

Including the tool "Quast" specificially triggers a wf extraction problem, see #109

So, all three problems are still present in 18.05 as loaded on main

[jennaj]

The "run" link works now.

The extract workflow also now works, but the order of tools is wrong and one is not connected with a noodle (when it should be by default). Not sure if this is just Main or something is still wrong with that particular tool (Quast) yet so leaving this ticket open.

Ticket for main #109

[jennaj]

Fixed, closing, see #109 for details