Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run history export setup via celery, if configured #11789

Merged
merged 3 commits into from
Apr 21, 2021

Conversation

mvdbeek
Copy link
Member

@mvdbeek mvdbeek commented Apr 5, 2021

What did you do?

  • Add a history export task

This builds on #11792, which is needed so that celery workers can enqueue jobs. The workers could also do the full export, but actually creating the archive seems like it should be done by a job.

There are additional things that should be done in a MQ worker, but for that we need to be able to chain tasks, and that requires a result backend in celery (typically redis or memcached)

Why did you make this change?

(Cite Issue number OR provide rationalization of changes if no issue exists)
(If fixing a bug, please add any relevant error or traceback)

How to test the changes?

(select the most appropriate option; if the latter, provide steps for testing below)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

For UI Components

  • I've included a screenshot of the changes

@mvdbeek mvdbeek force-pushed the history_export_task branch from ad60471 to 7199184 Compare April 5, 2021 17:22
@mvdbeek mvdbeek force-pushed the history_export_task branch from 7199184 to 6906a32 Compare April 13, 2021 12:35
@mvdbeek mvdbeek changed the title History export task - WIP Run history export setup via celery, if configured Apr 13, 2021
@mvdbeek mvdbeek force-pushed the history_export_task branch from 007a895 to 1023a26 Compare April 14, 2021 15:33
@mvdbeek mvdbeek force-pushed the history_export_task branch from 1023a26 to 961d66d Compare April 14, 2021 15:57
@mvdbeek mvdbeek force-pushed the history_export_task branch from f8897f0 to 08d4a51 Compare April 14, 2021 16:55
@mvdbeek mvdbeek marked this pull request as ready for review April 14, 2021 19:20
@github-actions github-actions bot added this to the 21.05 milestone Apr 14, 2021
@mvdbeek
Copy link
Member Author

mvdbeek commented Apr 14, 2021

And it's green! I will work on starting celery with/for tests next, but i think that's gonna be a separate piece of work and the current tests already exercise the Celery task itself.

@dannon
Copy link
Member

dannon commented Apr 21, 2021

This is exciting stuff. I especially really like the approach of using the same task function, simply sync, for this migration period.

@dannon dannon merged commit a277c24 into galaxyproject:dev Apr 21, 2021
@github-actions
Copy link

This PR was merged without a "kind/" label, please correct.

@nsoranzo nsoranzo deleted the history_export_task branch April 21, 2021 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants