configure specific object stores for Galaxy groups and/or roles

[bgruening]

It would be nice if we can enable specific object stores to specific groups/roles. A use case is that a specific community contributes its own (local) storage and it would be nice to enable this storage for this particular user group.

Is that already possible? If not, what should this configuration look like?

        <backend id="files11" type="disk" weight="1" store_by="uuid" group="plants">
            <files_dir path="/data/plants/galaxy_db/files"/>
            <extra_dir type="temp" path="/data/jwd/tmp"/>
            <extra_dir type="job_work" path="/data/jwd/main"/>
        </backend>

maybe?

[mvdbeek]

You should be able to send users to specific object stores via the destinations they are mapped to: #6552. Within a dynamic rule you should then be able to map a user to a role or other fancy things.

[bgruening]

But I would like to use the normal destinations and just use different storage. Does this mean I need to duplicate most of the destinations?

[mvdbeek]

I don't know. Maybe you can use one of the fancy job rule things to recreate your normal destinations and modify the object store id parameter. It shouldn't be too hard either to add an additional layer that maps users to preferred object stores. The
ObjectStore Populator seems like it could be the right layer of abstraction. That's how it is used for jobs: https://github.com/mvdbeek/galaxy/blob/56982bef8e85d03b4d5584dc43a411c911a01a5a/lib/galaxy/jobs/__init__.py#L1539

[bgruening]

That looks good, any idea how a configuration should look like?

[bgruening]

@jmchilton let me know if you have a prefered syntaxed.

[mvdbeek]

Maybe using an additional writeable tag within the backend. Or a permissions tag that we could later extend ? #10977 is adding a quota tag there, it could look like that ?

[natefoo]

Via static config would be a little clunky, for matching groups in the DB to groups in the config file, etc. It seems like if we want a way to do this beyond writing it in to your own dynamic job rule, we probably need a permission in the RBAC system we can associate with a role? Although then you would need a role to associate with the object store ID out of a config file...

There's also the question of what to do with users who have multiple associations, which is easier for an admin to handle in a dynamic rule than for us to account for in a config syntax, but ultimately probably works best with a user preference or switch in the history? I think John worked on something related to this a year or so ago.

[bgruening]

Assuming all jobs are going to a dynamic rule to determine the destination, we could set the object_store_id in as part of the dynamic job rules:

job_destination = app.job_config.get_destination(destination_id)
job_destination.params['object_store_id'] = 'foo'

[natefoo]

@bgruening and I have both tested the above in a dynamic rule and it works, presumably thanks to lazy dataset creation. Hopefully there are no remaining cases where dataset creation does not happen at job time?

[bgruening]

This seems to work very nicely and is even easier wit TPV.

[hexylena]

even easier wit TPV.

do you have a nice TPV example there? I'd like to copy that to our existing FAQ in the admin training.