Executive summary of 2021 Q2 Backend Goals

[jmchilton]

This is a draft/proposal until the backend team and the steering committee have reached an agreement.

Storage

• Assist deployment group in developing an end-to-end remote data example architecture - demonstration should include cloud object storage and Pulsar job handling with remote and extended metadata (so data stays remote). @mvdbeek @jmchilton
  • Fix Pulsar to work with remote data. Fixes for extended metadata working with Pulsar. #11788
  • Fix various Pulsar issues with extended metadata for certain classes of tools Problems with Pulsar Remote, Extended Metadata #11813
• Implement async tool execution API based on celery (to allow for deferred file resolution) @mvdbeek Prototype New Tool Submission/Job Creation Endpoint Based on Celery #11820
• Implement exporting individual files to file source plugins (for AnVIL) @jmchilton @mvdbeek Add tool for exporting individual files to galaxy file source plugins. #11613
• Finish objectstore abstractions started in Q1 to enable scratch storage in Galaxy @jmchilton
  • Implementing quotas per objectstore [WIP] Implement quota tracking options per ObjectStore. #10977
  • Implemented private objectstores to disable sharing, etc.. [WIP] Implement abstractions to annotate non-sharable datasets & objectstores. #10840
• Make substantial progress toward a prototype that migrates major portions of job setup to workers (allowing resolution of remote file URIs to be done only on workers and prevent extra copies of workflow/tool inputs, etc..). @mvdbeek, @jmchilton Containerized, Database-Free Job Creation #10873

Security

• Enable IdP-based login on Main (@dannon) (Custos Issues #10075 ?)
• Prototype a mechanism for connecting tools to an external secretes API (e.g. custos). (@dannon ?)
• Sanitize Galaxy logs for AnVIL project (@jmchilton and possibly @assuntad23)
  • Implement Test Cases Around GDPR Compliance #11807

Architecture

• New history panel support - upgrade APIs, model layer, tests, etc.. to support new history panel as the default. @mvdbeek, @dannon Fix collection state updates by setting collection update time #11849, Restore triggers dropped in #10821 #11843
• Upgrade our SQL Alchemy (SA) dependency to 1.4. (@ic4f) Migrate to SQLAlchemy 1.4 #11737
• Disable legacy SA behaviors to ensure we're ready for 2.0 and better support for asynchronous endpoints. (@ic4f) Sqlalchemy 1.4 compatibility #10892
• Roadmap to FastAPI #12257
  • Provide FastAPI apps for Tool Shed and Reports so we can drop paster/uwsgi. Provide FastAPI apps (entry points) for Tool Shed and Reports #11822
  • Provide example circus setup that starts the nodejs proxy we need for interactive tools. Provide example circus config for deploying nodejs proxy for ITs. #11823
  • Drop support for uWSGI mules - provide alternatives Switch default handler assignment to skip locked or transaction isolation if possible #11792
  • Migrate ansible roles to support gunicorn, etc.. during 21.05 deployment cycle (@mvdbeek).
  • Organize a documentation collaboration event with admin group on migrating all uWSGI documentation on docs and training material to newer stack (@hexylena , @mvdbeek)
• Use Celery to improve history export performance @mvdbeek Run history export setup via celery, if configured #11789
• Assemble a tutorial on contributing to Galaxy including a backend component @ic4f @jmchilton @ Go-To Developer Examples #11298
• Implement performance tests and continue to track down Postgres slowdowns and deadlock issues. @ic4f @mvdbeek

Stretch Goals:

• Prototype redis integration for more robust Celery task decomposition.
• Adopt Alembic for database migrations Move from sqlalchemy-migrate to alembic #10369
• Migrate more API routes to FastAPI - especially once the stack is the default and can be assumed.
• Various other ideas for utilizing remote data more efficiently from Directions to Improve Distributed Data Handling in Galaxy #11787
  • Allow file inputs to tools to improve potential for using remote data Allow Simple "File" Inputs to Tools #11785.
  • Two-pod-like Pulsar without Kubernetes (for clusters and main) Batch-mode Pulsar pulsar#250
• Per-user objectstore User-based ObjectStore #4840
• History update subscriptions over websockets
• SQLAlchemy: switch from classical to declarative mapping; Switch from SQLAlchemy classical mapping to declarative #10901
• Support import and exporting workflow invocations [WIP] Model Store Workflow Invocation Import/Export #9024
• Implement support for GA4GH DRS @davelopez, @jmchilton Initial Support for Consuming GA4GH DRS URIs #11819