Add FastAPI framework as next step for API framework

[mvdbeek]

FastAPI does a lot of things (websockets, graphql, Oauth2 + JWT tokens, backend tasks, celery integration, sync + async routes, openapi + typing, good performance, file serving, active community and nice documentation) and can be integrated nicely route-by-route without having to toss out everything at once.

This is a quick experiment where I replaced /api/jobs, IFF you start galaxy with python scripts/fapi.py -c config/galaxy.yml (need to activate the virtualenv and install uvicorn). The rest of routes continues to work as normal, so Galaxy is fully functional.

Here's the openAPI doc for the new route:

In terms of challenges:

• I don't think uwsgi can serve ASGI apps
• We'd have to replace the interactive tool routing
• We'd have to get buy-in from admins to abandon uwsgi

TODO for this PR:

• test API with paste and uvicorn until we drop (u)wsgi-only support
• fix job lock location

[hexylena]

1. that docs picture is gorgeous, it would make life so much better.
2. No one who deploys ITs in prod uses that routing anyway. And we should really switch to path based proxying (i.e. GIE style) which would address the local routing issues, and then we could toss out this uwsgi routing completely.
3. (as an admin) I've accrued a lot of knowledge and scripting/ansibling to handle uwsgi/deployment. It would be sad to throw out completely, but, sounds like some positive improvements for devs?

[mvdbeek]

3. (as an admin) I've accrued a lot of knowledge and scripting/ansibling to handle uwsgi/deployment. It would be sad to throw out completely, but, sounds like some positive improvements for devs?

I'd also be curious about what you think of gunicorn as a process manager, which seems like it's the recommended route for production. https://docs.gunicorn.org/en/latest/signals.html#master-process lists some signals it can handle. Reading this naively it sounds like it can also do zero-downtime restarts, as a sort of replacement for zerglings ?

[hexylena]

That potentially looks OK, that there's a master/worker split. I'm not as big of a fan of signals as Nate is, but it looks potentially nice?

[Nerdinacan]

+1 for websocket support. I know there's a lot more stuff we'd have to do server side to actually generate a subscription channel with changes that we could consume, but I love that it would be possible.

[mvdbeek]

This is by no means complete, but I think it is mergeable. That would make distributing work on various parts easier (xref: https://github.com/galaxyproject/galaxy/projects/20)

[mvdbeek]

@nsoranzo do you know why make update-dependencies didn't respect pygithub pinning requests to <2.25 ?

[nsoranzo]

@nsoranzo do you know why make update-dependencies didn't respect pygithub pinning requests to <2.25 ?

I don't have time to investigate in deep at the moment, you can try to regenerate the Docker image with: make -C lib/galaxy/dependencies/pipfiles/docker/ clean and re-run make update-dependencies

[mvdbeek]

Thanks, no luck there, but I think the manual pin shouldn't be a big deal for now.

[jmchilton]

Can you integrate some of the idea from:

• jmchilton/galaxy@0f9279f (add option to integrate testing using uvicorn)
• jmchilton/galaxy@e094706 (clearer transition path for older code, less duplication, and pushing more logic out of controllers that are tied to particular frameworks)

I know it is a little more work to rework the old APIs as we migrate the new ones - but I think there is value. It pushes more logic out of the controller layer if nothing else, and then it puts us in a better position if we're not able to fully make the transition within one or two releases.

[jmchilton]

I didn't mean to insist that all the tests pass under uvicorn, just wanted to have the option to run them so I could play with the branch.

[jmchilton]

Brilliant work!