Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prototype New Tool Submission/Job Creation Endpoint Based on Celery #11820

Open
Tracked by #12399
jmchilton opened this issue Apr 12, 2021 · 0 comments
Open
Tracked by #12399

Prototype New Tool Submission/Job Creation Endpoint Based on Celery #11820

jmchilton opened this issue Apr 12, 2021 · 0 comments

Comments

@jmchilton
Copy link
Member

Part of switching to celery tasks for this will mean the API endpoint will have to return different things. We probably want something truly asynchronous and that doesn't depend on all output dataset objects existing, jobs created, etc...

I'm not sure what that looks like exactly yet, but if we had an async, task-driven endpoint for this I think it would be more feasible to add functionality like passing in S3 URIs, DRS URIs (#11819), etc... during tool submission and resolving them immediately before tool submission via task chaining. We could do that first in the celery task worker but ultimately plan to stash them in the job definition to run on the worker (#10873). Designing this API though would let us decompose this task into pieces and write a GUI that can be agnostic about where and when URIs are resolved.

This is related to what is outlined in #11787 toward the end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants