Tracking ticket for workflow extraction problems on main

[jennaj]

If the reference genome option is not used (is optional), Quast cannot be extracted into a workflow from the history. Or, that is my best guess - leaving it out works.

Quast Genome assembly Quality (Galaxy Version 4.6.3) and the errors were first reported mid-March to Sentry.

https://sentry.galaxyproject.org/sentry/main/issues/290895/events/

Is this a https://usegalaxy.org problem, a Galaxy problem, or a tool problem? Hard to tell (for me). Looks like a parameter is not being parsed correctly during workflow extraction. The bug report I sent in today is from a purposefully failed run, just to see what happens, and can be ignored (the tool is failing correctly when this option is used but given an inappropriate genome input, which leads me to believe this is is not a tool/option problem, or not an obvious one).

@dannon or @natefoo Can you tell what is going on? If a tool or /galaxy problem, we can open a ticket against the appropriate repo and change this ticket to an upgrade tracking ticket for main.

For users - the workaround is to either not use Quast in extracted workflows or to wait for the correction. If another solution comes up, we will post an update to this ticket.

Reported by: https://biostar.usegalaxy.org/p/27449/ (thank you!)

[jennaj]

Update - the problem is with the extraction function itself. Added to an existing ticket with a few related workflow issues: #99

Leaving this open so we can track the fixes once they are made and promoted to main.

[jennaj]

Retesting

[jennaj]

Workflow extraction when the tool Quast is in the history still fails.

If you omit it during extraction, the workflow extracts fine (but "run" is still problematic #99).

@Delphine-L @jmchilton @natefoo @dannon Is this a tool problem with Quast (only)?

test history: https://usegalaxy.org/u/jen/h/test-history-quast-wf-extract

GURU MEDITATION: #2c54a661378e4b28a2a4f4efb0d1f015

[natefoo]

This is a different traceback from the one in #99:

KeyError: 'input_ref'
  File "galaxy/web/framework/middleware/sentry.py", line 43, in __call__
    iterable = self.application(environ, start_response)
  File "/cvmfs/main.galaxyproject.org/venv/lib/python2.7/site-packages/paste/recursive.py", line 85, in __call__
    return self.application(environ, start_response)
  File "galaxy/web/framework/middleware/statsd.py", line 40, in __call__
    req = self.application(environ, start_response)
  File "/cvmfs/main.galaxyproject.org/venv/lib/python2.7/site-packages/paste/httpexceptions.py", line 640, in __call__
    return self.application(environ, start_response)
  File "galaxy/web/framework/base.py", line 142, in __call__
    return self.handle_request(environ, start_response)
  File "galaxy/web/framework/base.py", line 221, in handle_request
    body = method(trans, **kwargs)
  File "galaxy/webapps/galaxy/controllers/workflow.py", line 812, in build_from_current_history
    dataset_collection_names=dataset_collection_names
  File "galaxy/workflow/extract.py", line 29, in extract_workflow
    steps = extract_steps(trans, history=history, job_ids=job_ids, dataset_ids=dataset_ids, dataset_collection_ids=dataset_collection_ids, dataset_names=dataset_names, dataset_collection_names=None)
  File "galaxy/workflow/extract.py", line 109, in extract_steps
    tool_inputs, associations = step_inputs(trans, job)
  File "galaxy/workflow/extract.py", line 310, in step_inputs
    associations = __cleanup_param_values(tool.inputs, param_values)
  File "galaxy/workflow/extract.py", line 371, in __cleanup_param_values
    cleanup("", inputs, values)
  File "galaxy/workflow/extract.py", line 330, in cleanup
    tmp = values[key]

[jennaj]

Just retested this -- the workflow can now be extracted, and that preview looks correct, but the actual workflow created has problems.

test history (same as above): https://usegalaxy.org/u/jen/h/test-history-quast-wf-extract

extracted workflow: https://usegalaxy.org/u/jen/w/workflow-constructed-from-history-test-history-quast-wf-extract

• Tools are in the wrong order in the extracted workflow (does not match the preview or the source history
• Quast is not connected by a noodle to the Upstream job. This can be seen in the workflow editor.

One of the tools in the workflow (unicycler) has an updated version. I'm rerunning a test using that next. I'm not sure why this would be a factor but seems worth doing, just in case.

test history with updated unicycler tool:
https://usegalaxy.org:/u/jen/h/copy-of-test-history-quast-wf-extract

extracted workflow: not created yet (jobs still running) -- unfortunately includes Unicycler and that won't be available till after 10/26 (pcs bridges cluster down for maintenance). I might have another run/history I can use, checking.... nope, none have the updated tool with successful jobs (that other test is queued, too)

[jennaj]

This looks fixed now. Noodles are connected correctly in the wf editor and runs correctly without edits.

Jobs finished, Extracted workflow: https://usegalaxy.org/u/jen/w/workflow-constructed-from-history-copy-of-test-history-quast-wf-extract-renamed-in-saved-histories-list

Run and results sent to a new history: https://usegalaxy.org:/u/jen/h/workflow-constructed-from-history-copy-of-test-history-quast-wf-extract-renamed-in-saved-histories-list

Not sure what fixed this, but closing!