Empower users to bring their own storage and file sources

[jmchilton]

Overview

This PR introduces plugin frameworks that allows admin to describe templates for object stores and file sources that users can setup themselves - this effectively allows users to define custom sources for getting files into Galaxy, exporting files out of Galaxy, and storing datasets in Galaxy. This work includes extensive documentation on the framework itself and on a wide variety of tested, production quality, and ready-to-use examples for major cloud services as well as a wide variety of examples that could be used to tailor the framework to local infrastructure.

Background

Pull Requests

This is very heavily based on #14073, #12940, #18117, and #15875.

#14073 is...

• used to give users a way to selecting an object store they define for histories, tool runs, workflow executions, etc...
• provides a visual framework for describing aspects of object stores data is defined in and exposes this
• used to make these data sources both private and untethered from Galaxy's default quota mechanism

#12940 is used to store secrets and in a structured way with multiple potential backends.

#18117 overhauled the object stores themselves so we have a solid foundation to build these plugins up from.

#15875 is these ideas but only applied to object stores and lacking the depth of typing, testing, documentation.

Issues

Implements #13816.
Fixes  #8790

Datasets vs Files, Object Stores vs File Sources

(This section of the PR description is ripped from the new data documentation included in this PR).

File sources in Galaxy are a sprawling concept but essentially they provide users access to simple files (stored hierarchically into folders) that can be navigated and imported into Galaxy. Importing a "file" into Galaxy generally creates a copy of that file into a Galaxy "object store". Once these files are stored in Galaxy,
they become "datasets". A Galaxy dataset is much more than a simple file - Galaxy datasets include various generic metadata a datatype, datatype specific metadata, and ownership and sharing rules managed by Galaxy.

Galaxy object stores (called "storage locations" in the UI) store datasets and global (accessible to all users) object stores are configured with the galaxy.yml property object_store_config_file (or object_store_config for a configuration embedded right in galaxy.yml) that defaults to object_store_conf.xml or object_store_conf.yml if either is present in Galaxy's configuration directory. Galaxy file sources provide users access to raw files and global files sources are configured with the galaxy.yml property file_sources_config_file (or file_sources for embedded configurations) that defaults to file_sources_conf.yml if that file is present in Galaxy's configuration directory.

Some of Galaxy's most updated and complete administrator documentation can be found in configuration sample files - this is definitely the case for object stores and file sources. The relevant sample configuration files include file_sources_conf.yml.sample and object_store_conf.sample.yml.

File sources and object stores configured with the above files essentially are available to all users of your Galaxy instance - hence this document describes them as "global" file sources and object stores. File source configurations do allow some templating that does allow the a global file source to be materialized differently for different users. For instance, you as an admin may setup a Dropbox file source and may explicitly add custom user properties that allow that single Dropbox file source to read from a user's preferences. Since there is just one Dropbox service and most people only have a single Dropbox account, this use case can be somewhat adequately addressed by the global file source and the global user preferences file. For a use case like Amazon S3 buckets though for instance, a single bucket file source that is parameterized one way is probably more clearly inadequate. For instance, users would very likely want to attach different buckets for different projects. Additionally, the Galaxy user interface doesn't tie the user preferences to the particular file source and so this method introduces a huge education burden on your Galaxy instance. Finally, the templating available to file sources are not available for object stores - and allowing users to describe how they would like datasets stored and to pay for their own dataset storage are important use cases.

This implements Galaxy configuration template libraries that allow the administrator to setup templates for file sources and object stores that users may instantiate as they see fit. User's can instantiate multiple instances of any template, the template concept can apply to both file source and object store plugins, and the user interface is unified from the template configuration file.

Implementation

File Source and Object Store Templates

Admins can define a set of object store "templates" and a set of file source “templates” - for either type, the configured set of these templates is called the "catalog" of (either object store or file source) templates. These are currently defined in object_store_templates.yml and file_source_templates.yml in the config directory or can be directly embedded into Galaxy’s main configuration (galaxy.yml) with object_store_templates and file_source_templates. The documentation includes extensive details on how to define the templates, what the syntax looks like, a description of use cases and examples of many such templates, and a set of production-ready templates that are completely generic and would feel at home in any general purpose Galaxy instance.

Very strict Pydantic models are included for the templates and for the resulting object store and file source configurations that they would yield when bound to user supplied "variables" and "secrets". The models are the source of truth about the documentation and we generate various images from them to be explicit about this and stick them in the documentation included here. We are going to store configurations of object stores and file sources in the database so the JSON blobs we define should be extremely well defined and well tested so we can have old blobs continue to work as the interface to object stores and files sources evolve over time. Proposed configurations for disk, boto3, azure_blob object storage have been included - in addition to aws_s3 and generic_s3 legacy configurations that target the older object store. These expose the relevant knobs available in our object store configurations currently and should be adapted as we migrate the object store code. Proposed configurations for posix, s3fs, ftp, and azure file sources have been included as well. The documentation includes details on all these object store and file source types, syntax descriptions, and various examples when they make sense.

The templates in the catalog can be hidden and new versions can be appended and the old ones will be automatically hidden - but admins should be warned that older definitions should remain for existing defined plugins. The UI has the ability to let users “upgrade” their instances of plugins as new template versions become available. The semantics are similar to tool re-running with newer versions - but much more structured is applied at every level.

The templates are parameterized with variables and secrets - and can include admin supplied fields and admin injected secrets that can be injected via global Vault values or environment variables.

I'm using Jinja templating as opposed to Python string templating or mako templating. Various plugins to Galaxy have used all three approaches, I've gone back and forth on this but Jinja seems the best fit because it preserves type information (in this implementation) - which seems to be where Galaxy is heading and dovetails well with the level structure and typing we're using throughout this implementation (from the database to typescript schema consumed by the frontend).

The documentation contains reference details on everything injected into the Jinja environment, lots of examples for admins to develop against, and a whole section on how to manage Jinja templates with Ansible - in case admins want to attempt to template the templates (there is no recommendation they do that - but I suspect they may want to and I’ve thought through and documented how to do that).

Database Models

The templates can be used by users to create UserObjectStore and UserFileSource model instances. I've used a prototype to separate the implementation from the object store code so an object store library consumer could store these on disk or in some other persistence store but for the purposes of the Galaxy application - instantiations of templates created by users are stored in the database in UserObjectStore instances and are called "object store instances" in the API. Likewise instantiations of UserFileSource objects are called “file source instances”. (I think not calling them User Object Stores in the API makes sense because one can easily imagine group or role implementations of these things in the database and one would expect the API to work with all of those).

The UserObjectStore model is:

class UserObjectStore(Base, HasConfigTemplate):
    __tablename__ = "user_object_store"
    secret_config_type = "object_store_config"

    id: Mapped[int] = mapped_column(Integer, primary_key=True)
    user_id: Mapped[Optional[int]] = mapped_column(Integer, ForeignKey("galaxy_user.id"), index=True)
    uuid: Mapped[Union[UUID, str]] = mapped_column(UUIDType(), index=True)
    create_time: Mapped[datetime] = mapped_column(DateTime, default=now)
    update_time: Mapped[datetime] = mapped_column(DateTime, default=now, onupdate=now, index=True)
    # user specified name of the instance they've created
    name: Mapped[str] = mapped_column(String(255), index=True)
    # user specified description of the instance they've created
    description: Mapped[Optional[str]] = mapped_column(Text)
    # active but doesn't appear in user selection
    hidden: Mapped[bool] = mapped_column(default=False)
    # set to False to deactive the source
    active: Mapped[bool] = mapped_column(default=True)
    # a purged file source cannot be re-activated because secrets have been purged
    purged: Mapped[bool] = mapped_column(default=False)
    # the template store id
    template_id: Mapped[str] = mapped_column(String(255), index=True)
    # the template store version (0, 1, ...)
    template_version: Mapped[int] = mapped_column(Integer, index=True)
    # Full template from object_store_templates.yml catalog.
    # For tools we just store references, so here we could easily just use
    # the id/version and not record the definition... as the templates change
    # over time this choice has some big consequences despite being easy to swap
    # implementations.
    template_definition: Mapped[Optional[CONFIGURATION_TEMPLATE_DEFINITION_TYPE]] = mapped_column(JSONType)
    # Big JSON blob of the variable name -> value mapping defined for the store's
    # variables by the user.
    template_variables: Mapped[Optional[CONFIGURATION_TEMPLATE_CONFIGURATION_VARIABLES_TYPE]] = mapped_column(JSONType)
    # Track a list of secrets that were defined for this object store at creation
    template_secrets: Mapped[Optional[CONFIGURATION_TEMPLATE_CONFIGURATION_SECRET_NAMES_TYPE]] = mapped_column(JSONType)

The UserFileSource model can be found in galaxy.models and looks very similar.

Backend Plumbing

The catalog and related models all so far are decoupled from the rest of Galaxy outside the object store. The layer above that is in lib/galaxy/managers/object_store_instances.py that ties together the database objects, the vault, templates, and object store factory methods to implement most of the target functionality. Code for creating, updating, and upgrading user object stores from one template version to the next are all defined in this file as well as some relevant CRUD code. Likewise, file_source_instances.py plays a similar role to establish the decoupling and isolation of user file sources.

API Endpoints have been added to:

• Fetch the templates of both types.
• Create instances of either type of object from supplied variables and secrets.
• Update the variables of a specified type.
• Update a secret associated with an instance.
• Upgrade an instance to a newer template target after the admin has added a new one.

Alternatives

The PR write up of #14073 describes in detail how it provided several abstractions that would be needed to address limitations of the work proposed in #14073. In additions to the description of the limitations described there - this work will be implemented with a keen eye toward implementation efficiency and will be usable with essentially any concrete object store implementation as opposed to tightly coupling to the cloud object store. I am confident the result of this will allow admins to address a greater number of potential scenarios.

The Docs

Here are some screenshots of the docs rendered. Here is the table of contents:

This demonstrates examples and docs of particular plugins:

along with this:

Hands-On Production Examples

Open config/file_source_templates.yml and place this in this contents to load up production-quality examples:

- include: ./lib/galaxy/files/templates/examples/production_azure.yml
- include: ./lib/galaxy/files/templates/examples/production_ftp.yml
- include: ./lib/galaxy/files/templates/examples/production_s3fs.yml
- include: ./lib/galaxy/files/templates/examples/production_aws_private_bucket.yml
- include: ./lib/galaxy/files/templates/examples/production_aws_public_bucket.yml

Open config/object_store_templates.yml and place this in the contents to load up production-quality examples:

- include: ./lib/galaxy/objectstore/templates/examples/production_azure_blob.yml
- include: ./lib/galaxy/objectstore/templates/examples/production_aws_s3.yml
- include: ./lib/galaxy/objectstore/templates/examples/production_aws_s3_legacy.yml
- include: ./lib/galaxy/objectstore/templates/examples/production_generic_s3.yml
- include: ./lib/galaxy/objectstore/templates/examples/production_generic_s3_legacy.yml
- include: ./lib/galaxy/objectstore/templates/examples/production_gcp_s3.yml

These production quality templates are described in a bit more detail in the documentation and have all been tested (I think). Many have screenshots in the documentation added as part of this PR as well.

After updating those configuration files and restarting Galaxy, you can now go to User Preferences and see the new options for creating personal file sources and object stores.

Warning: You will need to have a distributed object store for any of this to work and nearly every example also requires a Vault (the public AWS file source is a good example that does not require a Vault).

Clicking the file source option should show the available options for file sources templates. Hovering over an option will show the detailed Markdown description for that template.

The same is possible for

Some screenshots generated from Selenium test cases that exercise these examples include:

Public AWS Bucket File Source

Azure Blob File Source

FTP File Source

Azure Blob Object Store

AWS S3 Object Store

Google Storage via S3 Interop Object Store

Older Example (Detailed)

**This is the older example from the original PR which used a custom MinIO server. It is much more contrived but also is a complete walk-through of the concepts as applied to object store. Past @jmchilton put some time into these. **

My notes on setting up MinIO for this example - need a bucket to attach:

docker run -p 9000:9000 -p 9001:9001 -d --name minio  minio/minio:latest server /data --console-address ":9001"
brew install mc

mc alias set dev http://localhost:9000/ minioadmin minioadmin
mc admin info dev
mc admin user add dev gxuser1 gxuser1secret
mc admin user add dev gxuser2 gxuser2secret
mc admin policy attach dev readwrite  --user gxuser1
mc admin policy attach dev readwrite  --user gxuser2
mc mb --with-lock dev/gxuser1
mc mb --with-lock dev/gxuser2

Next setup a sophisticated distributed object store, going to build on the MSI example I used for #14073.

<?xml version="1.0"?>
<!--
    Huge chunks of text were stolen wholesale from MSI's data storage website
    (https://www.msi.umn.edu/content/data-storage). I've made large changes and adapted
    this for demonstration purposes - none of the text or policies or guarantees reflect
    actual current MSI or UMN policies.
-->
<!--
<object_store type="disk" store_by="uuid">
    <files_dir path="database/objects"/>
    <extra_dir type="temp" path="database/tmp"/>
    <extra_dir type="job_work" path="database/jobs_directory"/>
</object_store>
-->
<object_store type="distributed">
    <backends>
        <backend id="high_performance" allow_selection="true" type="disk" weight="1" name="High Performance Storage">
            <description>All MSI researchers have access to a high-performance, high capacity primary storage platform. This system currently provides 3.5 PB (petabytes) of storage. The integrity of the data is protected by daily snapshots and tape backups. It has sustained read and write speeds of up to 25 GB/sec.

There is default access to this storage by any MSI group with an active account. Very large needs can be also met, but need to be approved by the MSI HPC Allocation Committee. More details are available on the [Storage Allocations](https://www.msi.umn.edu/content/storage-allocations) page.

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).
</description>
            <files_dir path="/Users/jxc755/workspace/galaxy/database/objects/deafult"/>
            <badges>
                <faster />
                <more_stable />
                <backed_up>Backed up to MSI's long term tape drive nightly. More information about our tape drive can be found on our [Archive Tier Storage](https://www.msi.umn.edu/content/archive-tier-storage) page.</backed_up>
            </badges>
        </backend>
        <backend id="second" allow_selection="true" type="disk" weight="0" name="Second Tier Storage">
            <quota source="second_tier" />
            <description>MSI first added a Ceph object storage system in November 2014 as a second tier storage option. The system currently has around 10 PB of usable storage installed.

MSI's second tier storage is designed to address the growing need for resources that support data-intensive research. It is tightly integrated with other MSI storage and computing resources in order to support a wide variety of research data life cycles and data analysis workflows. In addition, this object storage platform offers new access modes, such as Amazon’s S3 (Simple Storage Service) interface, so that researchers can better manage their data and more seamlessly share data with other researchers whether or not the other researcher has an MSI account or is at the University of Minnesota.

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).
</description>
            <files_dir path="/Users/jxc755/workspace/galaxy/database/objects/temp"/>
            <badges>
                <faster />
                <less_stable />
                <not_backed_up />
                <less_secure>MSI's enterprise level data security policies and montioring have not yet been integrated with Ceph storage.</less_secure>
                <short_term>The data stored here is purged after a month.</short_term>
            </badges>
        </backend>
        <backend id="experimental" allow_selection="true" type="disk" weight="0" name="Experimental Scratch" private="true">
            <quota enabled="false" />
            <description>MSI Ceph storage that is purged more aggressively (weekly instead of monthly) and so it only appropriate for short term methods development and such. The rapid deletion of stored data enables us to provide this storage without a quota.

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).
            </description>
            <files_dir path="/Users/jxc755/workspace/galaxy/database/objects/temp"/>
            <badges>
                <faster />
                <less_stable />
                <not_backed_up />
                <less_secure>MSI's enterprise level data security policies and montioring have not yet been integrated with Ceph storage.</less_secure>
                <short_term>The data stored here is purged after a week.</short_term>
            </badges>
        </backend>
        <backend id="surfs" allow_selection="true" type="disk" weight="0" name="SURFS" private="true">
            <quota source="umn_surfs" />
            <description>Much of the data analysis conducted on MSI’s high-performance computing resources uses data gathered from UMN shared research facilities (SRFs). In recognition of the need for short to medium term storage for this data, MSI provides a service, Shared User Research Facilities Storage (SURFS), enabling SRFs to deliver data directly to MSI users. By providing a designated location for this data, MSI can focus data backup and other processes to these key datasets.  As part of this service, MSI will provide the storage of the data for one year from its delivery date.

It's expected that the consumers of these data sets will be responsible for discerning which data they may wish to keep past the 1-year term, and finding an appropriate place to keep it. There are several possible storage options both at MSI and the wider university. You can explore your options using OIT’s digital [storage options chooser tool](https://it.umn.edu/services-technologies/comparisons/select-digital-storage-options).

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).</description>
            <badges>
                <slower />
                <more_secure>University of Minnesota data security analysist's have authorized this storage for the storage of human data.</more_secure>
                <more_stable />
                <backed_up />
            </badges>
        </backend>
    </backends>
</object_store>

Next add an object_store_templates.yml file to config/:

- id: personal_disk
  name: Personal Disk
  description: Folder in shared shared disk area bound to your user.
  variables:
    folder_name:
      type: string
      help: Folder to create for your user.
  configuration:
    type: disk
    files_dir: '/Users/jxc755/workspace/galaxy/database/general/{{ user.username }}/{{ variables.folder_name }}'
    badges:
    - type: faster
    - type: less_secure
    - type: backed_up

- id: secure_disk
  name: Secure Disk
  description: Folder in shared shared disk area bound to your user.
  variables:
    folder_name:
      type: string
      help: Folder to create for your user.
  configuration:
    type: disk
    files_dir: '/Users/jxc755/workspace/galaxy/database/secure/{{ user.username }}/{{ variables.folder_name }}'

- id: minio
  name: Institutional S3 Storage
  description: Connect to our institutional MinIO storage service.
  variables:
    access_key:
      type: string
      help: A description of the user account used to connect to your storage.
    bucket:
      type: string
      help: The bucket to connect to.
  secrets:
    secret_key:
      help: The secret key used to connect to MinIO with for the given access key.
  configuration:
    type: generic_s3
    auth:
      access_key: '{{ variables.access_key }}'
      secret_key: '{{ secrets.secret_key }}'
    bucket:
      name: '{{ variables.bucket }}'
      use_reduced_redundancy: false
    connection:
      host: localhost
      port: 9000
      is_secure: false
      conn_path: ""
    badges:
    - type: slower
    - type: less_secure
    - type: less_stable

This sets up three templates users can create object stores from in the UI.

The first two just allow the user to setup folders under a shared project directories. This example makes sense when you really trust your users and you've got a variety of disk options mounted on Galaxy servers with different properties.

The third template allows the user to attach buckets from the MinIO server we setup - using access key, bucket names, and secret keys we've communicated to the user in some way.

The User Preferences menu now has a "Manage Your Object Stores" option:

Clicking "Create" will show the templates the user can create object stores from:

Let's build one of each of these:

As they are built we see them in the index:

Object store badges communicate information about the object store, its properties, and free Markdown populated by the admin for different object stores:

Some information about the type of object store is displayed also:

When you edit the object stores, regular settings (metadata and admin defined variables) are presented in a different way than secrets stored in Galaxy's vault:

Workflows, tools, histories will all now allow these object stores to be selected as the "preferred" object store. The user can also select this as their preference for all analyses:

These two new user-bound object stores are now available right alongside the admin defined ones in object_store_conf.xml.

Here I've set the history default and ran and job and we can see the result in a MinIO because the path is the path to object store cache:

Looking in the object store management window:

We can see the file that was created.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[bgruening]

This is absolutely amazing and magic 🪄

[itisAliRH]

This feature is amazing! Thank you, @jmchilton!

I attempted to set up and test it on my local. The overall functionality is great, but some aspects must be refined and improved. I will document these in the #18128 issue.

[davelopez]

This is utterly amazing! I'm pretty sure users will find this extremely useful.

I tested it a bit and the only things I could find were the following:

• Removing file source instances does not remove it from the UI You just fixed it while I was reviewing! great!
• If you don't provide a description for a file source instance it will fail validation when opening the list of file sources. This is not a bug of this PR, but rather in the schema definition of FilesSourcePlugin as the doc field should be optional.

  --- a/lib/galaxy/schema/remote_files.py
  +++ b/lib/galaxy/schema/remote_files.py
  @@ -55,8 +55,8 @@ class FilesSourcePlugin(Model):
           description="The display label for this plugin.",
           examples=["Library Import Directory"],
       )
  -    doc: str = Field(
  -        ...,
  +    doc: Optional[str] = Field(
  +        None,
           title="Documentation",

• I also missed the ability to set the file sources as writeable by the user. I guess the admin can set it up in the template and it will work, but this can be a follow-up enhancement.

In general, we can iron out some small things after we battle-test this, especially around the UX, but it seems to be working as advertised and has pretty good testing coverage 🚀

[jmchilton]

@davelopez Thanks for the kind and detailed review. I've double checked that doc being null is allowed in the file source plugins themselves and then I made that schema switch you mentioned. Thanks so much for catching that!

[mvdbeek]

There's a conflict here, could you resolve that ?

[jdavcs]

@jmchilton I just noticed this.. Can you, please, use alembic to autogenerate migration hashes next time? Otherwise we get output like this when running the history command:
c14a3c93d66a -> c14a3c93d66b (gxy) (head), add user defined file sources - it takes some time to see the a vs b suffix; until then it looks like a terrible bug.. Also, with distinctly different hashes, when specifying a revision identifier it's usually enough to indicate the first 3-4 characters; here it would have to be all 12.

[jmchilton]

The files appear right next to each other which makes editing them together easier... besides there is no hard proof I falsified that hash - just speculation 😇😆 . If this ever happens again I'll be good and respect the process even if I think it is arbitrary.