[WIP] Empower Users to Bring Their Own Storage

[jmchilton]

Pretty close to an MVP I think?

Background

This is very heavily based on #14073 and #12940.

#14073 is...

• used to give users a way to selecting an object store they define for histories, tool runs, workflow executions, etc...
• provides a visual framework for describing aspects of object stores data is defined in and exposes this
• used to make these data sources both private and untethered from Galaxy's default quota mechanism

#12940 is used to store secrets and in a structured way with multiple potential backends.

Implementation

Object Store Templates

Admins (and potentially in the future the core project) can define a set of object store "templates" - the configured set of these templates the configured "catalog" of object store templates. These are currently defined in object_store_templates.yml in the config directory. Ultimately I think admin-defined things are important and have some really interesting applications but I doubt the uptake will be nearly as high if we don't also ship a default set of templates at some point in the future after we're confident about the work. I don't think this set of templates should be included in the initial PR though.

Very strict Pydantic models are included for the templates and for the resulting object store configurations that they would yield when bound to user supplied "variables" and "secrets". As this PR documentation is outdated over time - the models will remain the source of truth about the documentation. We are going to store configurations of object stores in the database so the JSON blobs we define should be extremely well defined and well tested so we can have old blobs continue to work as the interface to object stores evolve over time. Proposed configurations for disk, s3, and azure storage have been included. These expose the relevant knobs available in our object store configurations currently and should be adapted as we migrate the object store code.

The templates in the catalog can be hidden and new versions can be appended and the old ones will be automatically hidden - but admins should be warned that older definitions should remain for existing defined storage.

The templates are parameterized with variables and secrets - and can include admin supplied fields (and presumably app value information will be trivial to implement).

I'm using jinja templating as opposed to Python string templating or mako templating. Various plugins to Galaxy have used all three approaches, I've gone back and forth on this but jinja seems the best fit because it preserves type information (in this implementation) - which seems to be where Galaxy is heading and dovetails well with the level structure and typing we're using throughout this implementation (from the database to typescript schema consumed by the frontend).

Database Models

The templates can be used by users to create UserObjectStore model instances. I've used a prototype to separate the implementation from the object store code so an object store library consumer could store these on disk or in some other persistence store but for the purposes of the Galaxy application - instantiations of templates created by users are stored in the database in UserObjectStore instances and are called "object store instances" in the API. (I think not calling them User Object Stores in the API makes sense because one can easily imagine group or role implementations of these things in the database and one would expect the API to work with all of those).

The UserObjectStore model is:

class UserObjectStore(Base, RepresentById):
    __tablename__ = "user_object_store"

    id = Column(Integer, primary_key=True)
    user_id = Column(Integer, ForeignKey("galaxy_user.id"), index=True)
    create_time = Column(DateTime, default=now)
    update_time = Column(DateTime, default=now, onupdate=now, index=True)
    # user specified name of the instance they've created
    name = Column(String(255), index=True)
    # user specified description of the instance they've created
    description = Column(Text)
    # the template store id
    object_store_template_id = Column(String(255), index=True)
    # the template store version (0, 1, ...)
    object_store_template_version = Column(Integer, index=True)
    # Full template from object_store_templates.yml catalog.
    # For tools we just store references, so here we could easily just use
    # the id/version and not record the definition... as the templates change
    # over time this choice has some big consequences despite being easy to swap
    # implementations.
    object_store_template_definition = Column(JSONType)
    # Big JSON blob of the variable name -> value mapping defined for the store's
    # variables by the user.
    object_store_template_variables = Column(JSONType)
    # Track a list of secrets that were defined for this object store at creation
    object_store_template_secrets = Column(JSONType)

Backend Plumbing

The catalog and related models all so far are decoupled from the rest of Galaxy outside the object store. The layer above that is in lib/galaxy/managers/object_store_instances.py that ties together the database objects, the vault, templates, and object store factory methods to implement most of the target functionality. Code for creating, updating, and upgrading user object stores from one template version to the next are all defined in this file as well as some relevant CRUD code.

I've already included an API endpoint for fetching the template catalog - the next step will an API endpoint for creating an object store from a template ID, the user specified variables, and user specified secretes. The initial form that targets that endpoint is already there and handles basic variables and secrets using the Galaxy Form framework. The backend will have to create database objects and vault secrets for each object store created by a user. So for instance if a user creates an object store named "my-cool-objects" which defines a variable "foo" and a secret "bar". A new object will be added to the database for this object store (maybe UserObjectStoreInstance with a name field of "my-cool-objects") and the variable foo will be attached to that object (maybe UserObjectStoreInstanceVariable or maybe just in a JSON attribute on UserObjectStoreInstance with all the template variables). user_vault.write_secret("/object_stores/my-cool-objects/bar", "<supplied value>") will also be done to securely store supplied variables.

The guts of taking the templated object store configuration and the variables and secrets are already there in the initial PR, the next step once we defined the objects above and attach them to a job's outputs - would be serialize the object store the job. We already serialize an object store configuration and we already serialize a file source config that realizes abstract variables into concrete ones for a job - I think those two patterns can be combined to take care of the guts of the interaction between Galaxy and jobs.

Another implementation detail will be generalize how Dataset objects are mapped to ObjectStores. I think the backward compatible thing is just to keep using object_store_id and if it is a simple string do what we do now, but if it is a URI - say "user://<user_id>/my-cool-objects`` then to resolve the object store as needed. This will require a bunch of fiddly tracking and exceptions when dealing with maintenance scripts and such but I've got some experience doing this as part of #14073.

The MVP I think will benefit from UI elements for managing existing object stores defined this way and providing information about them. I haven't worked through the details of this.

New APIs

    "/api/object_store_instances": {
        /** Get a list of persisted object store instances defined by the requesting user. */
        get: operations["object_stores__instances_index"];
        /** Create a user-bound object store. */
        post: operations["object_stores__create_instance"];
    };
    "/api/object_store_instances/{user_object_store_id}": {
        /** Get a list of persisted object store instances defined by the requesting user. */
        get: operations["object_stores__instances_get"];
        /** Update or upgrade user object store instance. */
        put: operations["update_instance_api_object_store_instances__user_object_store_id__put"];
    };
    "/api/object_store_templates": {
        /** Get a list of object store templates available to build user defined object stores from */
        get: operations["object_stores__templates_index"];
    };

Alternatives

The PR write up of #14073 describes in detail how it provided several abstractions that would be needed to address limitations of the work proposed in #14073. In additions to the description of the limitations described there - this work will be implemented with a keen eye toward implementation efficiency and will be usable with essentially any concrete object store implementation as opposed to tightly coupling to the cloud object store. I am confident the result of this will allow admins to address a greater number of potential scenarios.

Detailed Example

My notes on setting up MinIO for this example - need a bucket to attach:

docker run -p 9000:9000 -p 9001:9001 -d --name minio  minio/minio:latest server /data --console-address ":9001"
brew install mc

mc alias set dev http://localhost:9000/ minioadmin minioadmin
mc admin info dev
mc admin user add dev gxuser1 gxuser1secret
mc admin user add dev gxuser2 gxuser2secret

Create gxuser1security.txt as:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
            "s3:ListBucket",
            "s3:PutObject",
            "s3:GetObject",
            "s3:DeleteObject"
        ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::gxuser1/*", "arn:aws:s3:::gxuser1"
      ],
      "Sid": "BucketAccessForUser"
    }
  ]
}

mc admin policy add local gxuser1-bucket-policy ./gxuser1security.txt
# attach security policy... following command not working for me...
# mc admin policy attach dev gxuser1-bucket-policy --user gxuser1
# attach in the GUI - at http://localhost:9001/ - login with minioadmin / minioadmin
# Also in the UI create a new bucket called gxuser1

Next setup a sophisticated distributed object store, going to build on the MSI example I used for #14073.

<?xml version="1.0"?>
<!--
    Huge chunks of text were stolen wholesale from MSI's data storage website
    (https://www.msi.umn.edu/content/data-storage). I've made large changes and adapted
    this for demonstration purposes - none of the text or policies or guarantees reflect
    actual current MSI or UMN policies.
-->
<!--
<object_store type="disk" store_by="uuid">
    <files_dir path="database/objects"/>
    <extra_dir type="temp" path="database/tmp"/>
    <extra_dir type="job_work" path="database/jobs_directory"/>
</object_store>
-->
<object_store type="distributed">
    <backends>
        <backend id="high_performance" allow_selection="true" type="disk" weight="1" name="High Performance Storage">
            <description>All MSI researchers have access to a high-performance, high capacity primary storage platform. This system currently provides 3.5 PB (petabytes) of storage. The integrity of the data is protected by daily snapshots and tape backups. It has sustained read and write speeds of up to 25 GB/sec.

There is default access to this storage by any MSI group with an active account. Very large needs can be also met, but need to be approved by the MSI HPC Allocation Committee. More details are available on the [Storage Allocations](https://www.msi.umn.edu/content/storage-allocations) page.

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).
</description>
            <files_dir path="/Users/jxc755/workspace/galaxy/database/objects/deafult"/>
            <badges>
                <faster />
                <more_stable />
                <backed_up>Backed up to MSI's long term tape drive nightly. More information about our tape drive can be found on our [Archive Tier Storage](https://www.msi.umn.edu/content/archive-tier-storage) page.</backed_up>
            </badges>
        </backend>
        <backend id="second" allow_selection="true" type="disk" weight="0" name="Second Tier Storage">
            <quota source="second_tier" />
            <description>MSI first added a Ceph object storage system in November 2014 as a second tier storage option. The system currently has around 10 PB of usable storage installed.

MSI's second tier storage is designed to address the growing need for resources that support data-intensive research. It is tightly integrated with other MSI storage and computing resources in order to support a wide variety of research data life cycles and data analysis workflows. In addition, this object storage platform offers new access modes, such as Amazon’s S3 (Simple Storage Service) interface, so that researchers can better manage their data and more seamlessly share data with other researchers whether or not the other researcher has an MSI account or is at the University of Minnesota.

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).
</description>
            <files_dir path="/Users/jxc755/workspace/galaxy/database/objects/temp"/>
            <badges>
                <faster />
                <less_stable />
                <not_backed_up />
                <less_secure>MSI's enterprise level data security policies and montioring have not yet been integrated with Ceph storage.</less_secure>
                <short_term>The data stored here is purged after a month.</short_term>
            </badges>
        </backend>
        <backend id="experimental" allow_selection="true" type="disk" weight="0" name="Experimental Scratch" private="true">
            <quota enabled="false" />
            <description>MSI Ceph storage that is purged more aggressively (weekly instead of monthly) and so it only appropriate for short term methods development and such. The rapid deletion of stored data enables us to provide this storage without a quota.

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).
            </description>
            <files_dir path="/Users/jxc755/workspace/galaxy/database/objects/temp"/>
            <badges>
                <faster />
                <less_stable />
                <not_backed_up />
                <less_secure>MSI's enterprise level data security policies and montioring have not yet been integrated with Ceph storage.</less_secure>
                <short_term>The data stored here is purged after a week.</short_term>
            </badges>
        </backend>
        <backend id="surfs" allow_selection="true" type="disk" weight="0" name="SURFS" private="true">
            <quota source="umn_surfs" />
            <description>Much of the data analysis conducted on MSI’s high-performance computing resources uses data gathered from UMN shared research facilities (SRFs). In recognition of the need for short to medium term storage for this data, MSI provides a service, Shared User Research Facilities Storage (SURFS), enabling SRFs to deliver data directly to MSI users. By providing a designated location for this data, MSI can focus data backup and other processes to these key datasets.  As part of this service, MSI will provide the storage of the data for one year from its delivery date.

It's expected that the consumers of these data sets will be responsible for discerning which data they may wish to keep past the 1-year term, and finding an appropriate place to keep it. There are several possible storage options both at MSI and the wider university. You can explore your options using OIT’s digital [storage options chooser tool](https://it.umn.edu/services-technologies/comparisons/select-digital-storage-options).

More information about MSI Storage can be found [here](https://www.msi.umn.edu/content/data-storage).</description>
            <badges>
                <slower />
                <more_secure>University of Minnesota data security analysist's have authorized this storage for the storage of human data.</more_secure>
                <more_stable />
                <backed_up />
            </badges>
        </backend>
    </backends>
</object_store>

Next add an object_store_templates.yml file to config/:

- id: personal_disk
  name: Personal Disk
  description: Folder in shared shared disk area bound to your user.
  variables:
    folder_name:
      type: string
      help: Folder to create for your user.
  configuration:
    type: disk
    files_dir: '/Users/jxc755/workspace/galaxy/database/general/{{ user.username }}/{{ variables.folder_name }}'
    badges:
    - type: faster
    - type: less_secure
    - type: backed_up

- id: secure_disk
  name: Secure Disk
  description: Folder in shared shared disk area bound to your user.
  variables:
    folder_name:
      type: string
      help: Folder to create for your user.
  configuration:
    type: disk
    files_dir: '/Users/jxc755/workspace/galaxy/database/secure/{{ user.username }}/{{ variables.folder_name }}'

- id: minio
  name: Institutional S3 Storage
  description: Connect to our institutional MinIO storage service.
  variables:
    access_key:
      type: string
      help: A description of the user account used to connect to your storage.
    bucket:
      type: string
      help: The bucket to connect to.
  secrets:
    secret_key:
      help: The secret key used to connect to MinIO with for the given access key.
  configuration:
    type: generic_s3
    auth:
      access_key: '{{ variables.access_key }}'
      secret_key: '{{ secrets.secret_key }}'
    bucket:
      name: '{{ variables.bucket }}'
      use_reduced_redundancy: false
    connection:
      host: localhost
      port: 9000
      is_secure: false
      conn_path: ""
    badges:
    - type: slower
    - type: less_secure
    - type: less_stable

This sets up three templates users can create object stores from in the UI.

The first two just allow the user to setup folders under a shared project directories. This example makes sense when you really trust your users and you've got a variety of disk options mounted on Galaxy servers with different properties.

The third template allows the user to attach buckets from the MinIO server we setup - using access key, bucket names, and secret keys we've communicated to the user in some way.

The User Preferences menu now has a "Manage Your Object Stores" option:

Clicking "Create" will show the templates the user can create object stores from:

Let's build one of each of these:

As they are built we see them in the index:

Object store badges communicate information about the object store, its properties, and free Markdown populated by the admin for different object stores:

Some information about the type of object store is displayed also:

When you edit the object stores, regular settings (metadata and admin defined variables) are presented in a different way than secrets stored in Galaxy's vault:

Workflows, tools, histories will all now allow these object stores to be selected as the "preferred" object store. The user can also select this as their preference for all analyses:

These two new user-bound object stores are now available right alongside the admin defined ones in object_store_conf.xml.

Here I've set the history default and ran and job and we can see the result in a MinIO because the path is the path to object store cache:

Looking in the object store management window:

We can see the file that was created.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[jdavcs]

Would you mind moving the database migration into a separate commit? Same reasoning as here: #15663 (comment)

[sanjaysrikakulam]

Hey! Can you please share the timeline for this feature and the Galaxy version in which it will become available?