Add onedata objectstore #17540
Add onedata objectstore #17540
Conversation
|
This looks pretty cool, anything we can do to help ? |
|
Hello, sorry, I accidentally opened it too early. Although this is a working prototype, we are still working on improvements. In a few days, when the final version is ready, I will reopen the pr. |
|
@mvdbeek it's finally ready - I'm reopening the pr. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
lib/galaxy/objectstore/onedata.py
Outdated
| log.debug(f"Configuring Onedata connection to {self.onezone_domain} " | ||
| f"(disable_tls_certificate_validation={self.disable_tls_certificate_validation})") |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
…ile source plugin
…nedata file source plugin
bwalkowi
force-pushed
the
add-onedata-objectstore
branch
from
be5fcf1 to
85d1cd7
Compare
|
I specified the Onedata object store dependencies as optional so that they would not be installed by default and that is why the automated tests failed. Is there a specific requirements.txt file used during testing that I can add them to? |
|
@bwalkowi can you please fix the linting errors and update with the main branch. Thanks! |
…d-onedata-objectstore
|
@bgruening done! |
|
The integration test failures look legitimate.
The conditional dependency handling looks right though - I don't know immediately what to do. |
|
Conditional dependencies that we're actually testing need to be added to https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/dependencies/dev-requirements.txt and https://github.com/galaxyproject/galaxy/blob/dev/pyproject.toml#L129 |
|
nice. It seems everything but code scanning is green. Do you have any recommendations regarding |
|
You haven't touched these, so no need to change that here. |
| @@ -139,6 +139,7 @@ isort = "*" | |||
| lxml = "!=4.2.2" | |||
| markdown-it-reporter = "*" | |||
| myst-parser = "*" | |||
| onedatafilerestclient = "==21.2.5rc1" | |||
There was a problem hiding this comment.
| onedatafilerestclient = "==21.2.5rc1" | |
| onedatafilerestclient = "*" |
|
This PR was merged without a "kind/" label, please correct. |
|
I made a bunch of changes to this with the PR #18174 . By using a new superclass for caching object stores there was a lot less duplication with other object stores and you should have gotten the bug fixes we put into #18117. We merged that PR but I would re-run through it with your setup and let us know if anything broke and send patches. Thanks so much for the contribution! |
|
nice! sure thing, I will test it and try to send eventual patches within a week. |
Object store plugin for Onedata distributed data management system (https://onedata.org/). Requires Onedata version >= 21.02.5.
How to test the changes?
(Select all options that apply)
I've included appropriate automated tests.
This is a refactoring of components with existing test coverage.
Instructions for manual testing are as follows:
Add the
onedatafilerestclientpackage to the currently used requirements (can be found in conditional-requirements.txt).Create
object_store_conf.xmlfile in the config dir, copying the part concerning Onedata fromobject_store_conf.xml.sample.Fill in the config (
access_token,onezone_domain,space,disable_tls_certificate_validation- you may wantTrue- see below).You will need an account in a Onedata system instance, however, version 21.02.5 is not officially released yet. The best way to test is to use the "demo mode", which allows easily setting up a dockerized Onedata environment: https://onedata.org/#/home/documentation/topic/stable/demo-mode
Create galaxy.yml file in the config dir with the following content:
Start the galaxy server, log in.
Upload some data and run some jobs. Open the Onedata UI and navigate to your space to see the stored Galaxy datasets.
License