Cloud ObjectStore #4352

VJalili · 2017-07-27T17:11:34Z

The Cloud ObjectStore is a provider-agnostic cloud-based storage. In other words, the Cloud ObjectStore can read/write from various cloud-based storages without a provider-specific implementation; it leverages on CloudBridge which provides a unified access to various providers.

This PR sets a base for the User-based ObjectStore we have been working on. Initially, the Cloud ObjectStore was part of the User-based ObjectStore (PR #4314), and it is being submitted as a separate PR following the comments on the PR #4314.

To simplify the comparison with S3ObjectStore, the Cloud ObjectStore provides the very same functionality as S3ObjectStore, i.e., read/write to AWS S3 buckets. The only difference is: S3ObjectStore leverages on boto while Cloud uses CloudBridge.

pinging @bgruening @nsoranzo and @erasche

mvdbeek · 2017-07-28T08:58:34Z

lib/galaxy/objectstore/cloud.py

+    import boto
+
+    from boto.exception import S3ResponseError
+    from boto.s3.key import Key


The linter doesn't like unused imports (this line and the following), but you can ignore this by doing

from boto.s3.key import Key # noqa: F401 from boto.s3.connection import S3Connection # noqa: F401

Although I'm not sure in which scenario it would help to import unused dependencies ...
If external code needs it, it should probably import the dependencies where they are needed.

@mvdbeek Thanks, the imports are updated accordingly.
Actually, boto import is temporary solution for handling AWS S3 exceptions until cloudbridge implements such exceptions for all the providers.

afgane

Looks pretty good. Clearly, this is still AWS-only even if it implements the CloudBridge interface but I guess that's just the first step.
There are a few comment lines and debug print statements that were also in the previous implementation. I think those can be removed now. Finally, I made a few comments inline, some of which repeat themselves in multiple places but the comment should apply to whole the file.

afgane · 2017-08-01T21:10:13Z

lib/galaxy/dependencies/conditional-requirements.txt

@@ -13,6 +13,7 @@ graphitesend
 azure-storage==0.32.0
 # PyRods not in PyPI
 python-ldap==2.4.27
+cloudbridge==0.3.1


Any reason not to go with the latest current release (0.3.2)?

No specific reason, I updated to the current latest version (also see starforge PR #138).

afgane · 2017-08-01T21:12:59Z

lib/galaxy/objectstore/cloud.py

+from ..objectstore import convert_bytes, ObjectStore
+from cloudbridge.cloud.factory import CloudProviderFactory, ProviderList
+
+try:


Presumably, this is for backward compatibility? Either way, it'd be good to have a comment why is this necessary in addition to the CloudBridge import.

boto was only used to handle some exceptions. However, I replaced all boto-specific exceptions with a generic exception. This is totally a temporary patch till exceptions are properly wrapped and thrown by CloudBridge.

afgane · 2017-08-01T21:35:54Z

lib/galaxy/objectstore/cloud.py

+                    bucket = self.conn.object_store.create(bucket_name)
+                log.debug("Using cloud object store with bucket '%s'", bucket.name)
+                return bucket
+            except S3ResponseError:


This is not imported anywhere. Also, create method returns S3CreateError (http://boto.cloudhackers.com/en/latest/ref/boto.html#module-boto.exception).
More importantly, this will not work cross-provider so having an extra, general except may be desirable?
Same comment applies to other refs to S3ResponseError.
(xref CloudVE/cloudbridge#9 and custom/standardized CloudBridge exceptions).

I agree, and as explained in the previous comments, as a temporary patch, replaced S3/boto-specific exceptions with generic exceptions.

afgane · 2017-08-01T21:48:10Z

lib/galaxy/objectstore/cloud.py

+            self.secret_key = a_xml.get('secret_key')
+            b_xml = config_xml.findall('bucket')[0]
+            self.bucket = b_xml.get('name')
+            self.use_rr = string_as_bool(b_xml.get('use_reduced_redundancy', "False"))


It doesn't look like we're using this option anymore (which is understandable from the multi-cloud perspective), so probably best to remove this line and relevant config.

afgane · 2017-08-01T21:51:12Z

lib/galaxy/objectstore/cloud.py

+                        created_obj = self.bucket.create_object(rel_path)
+                        created_obj.upload(source_file)
+                    else:
+                        self.bucket.get(rel_path).upload(source_file)


Should probably keep the same code structure as in the if case (or the other way around, as long as it's consistent).

afgane · 2017-08-01T21:53:58Z

lib/galaxy/objectstore/cloud.py

+                              rel_path)
+                    return True
+                # FIXME: don't need to differenciate between uploading from a string or file,
+                # because CloudBridge handles this internally.


What's there to fix? if and else here look the same so shouldn't it just be consolidated?

Updated. But I'm postponing function signature change for future, just to keep the changes at this PR at a minimal possible level.

afgane · 2017-08-01T21:57:54Z

lib/galaxy/objectstore/cloud.py

+            if not os.path.exists(cache_dir):
+                os.makedirs(cache_dir)
+
+            # Although not really necessary to create S3 folders (because S3 has


Should probably remove this comment.

afgane · 2017-08-01T22:01:12Z

lib/galaxy/objectstore/cloud.py

+            if entire_dir and extra_dir:
+                shutil.rmtree(self._get_cache_path(rel_path))
+                results = self.bucket.list(prefix=rel_path)
+                for key in results:


May be better to replace references like this that use AWS terminology to something cloud agnostic? e.g., object. This applies throughout the file (e.g., _key_exists).

Agree. However, as I mentioned in other comments, I'm trying to keep Cloud very similar to S3ObjectStore (following the comments on the PR #4314). Using a general terminology might escalate the challenges :) But we'll surely use more general nomenclature in future updates.

VJalili · 2017-08-02T19:36:40Z

@afgane Thanks for the comments, I updated the code accordingly. Few points:

Clearly, this is still AWS-only even if it implements the CloudBridge interface but I guess that's just the first step.

Correct. This PR sets the ground for User-based ObjectStore (initially discussed at #4314); which is a user-based and provider-agnostic ObjectStore. However, following the comments at #4314, I made Cloud with least possible changes to S3ObjectStore to (a) simplify comparison, (b) incremental and smaller impacts on the Galaxy codebase. Accordingly, you may see lots of S3-specific terminology and comments. I think we could be more provider-agnostic in future updates.

I'll comment inline other comments.

afgane · 2017-08-02T22:02:26Z

It's generally not great to use the generic Exception if it can be helped but given this is an intermediary step, I think it's ok.

There are a couple of tests still failing though. How do the dependencies that are still getting built (i.e., the open PR for CloudBridge wheel) get pulled into the test env? Could that be it?

VJalili · 2017-08-02T22:09:34Z

Maybe the tests are failing because the current latest version of the CloudBridge is not built yet (galaxyproject/starforge#138)

nsoranzo · 2017-08-07T16:47:18Z

Unit tests are failing with:

File "/home/travis/build/galaxyproject/galaxy/lib/galaxy/objectstore/cloud.py", line 25, in <module>

    from cloudbridge.cloud.factory import CloudProviderFactory, ProviderList

ImportError: No module named cloudbridge.cloud.factory

Refactor bug reporting into plugins / Remove bug report mako

Small GRT bugfixes

Created alphabetic and numerical sort tool for collection operations

Vcf gz datatype

- They now uniformly handle server option processing. - Reports and Tool Shed now respect many new and important options such as --skip_venv. - Reports and Tool Shed now do extra checking around Python version that Galaxy does. In addition to improving the reports and tool shed startup scripts and reducing cognative load with respect to option handling, this is an important pre-condition to us being able us to switch on uwsgi by default for all three services simultaneously without a bunch of copy and pasting.

Initial outline of allowing launch via uwsgi.

This has been rebased several times and includes a variety of fixes (some thanks to martenson).

- Implement kwalify schema to validate/define the schema. - Replace ini configuration with YAML (completely backward compatible). - Rebuild a sample configuration from the schema (config/tool_shed.yml.sample).

Fix quota function name

replace exclamation with a clock icon

…ctStore # Conflicts: # lib/galaxy/objectstore/cloud.py

don't uppercase envoverride

- Added `Cloud` to objectstore import.

requests via a different PR.

a generic exception--a temporary solution till CloudBridge wraps the exceptions properly.

because CloudBridge internally maps to appropriate functions depending on the input type.

Should use the wheel created at [this PR](galaxyproject/starforge#139).

…into CloudObjectStore

VJalili · 2017-08-23T19:22:58Z

Some thing went awfully wrong with rebase. Please follow up #4487

VJalili · 2017-08-28T22:28:59Z

@nuwang @afgane due to an issue with this branch due to a white-space removal rebase, please follow up on a new PR #4487

VJalili mentioned this pull request Jul 27, 2017

User-based ObjectStore #4314

Closed

galaxybot added the triage label Jul 27, 2017

galaxybot added this to the 17.09 milestone Jul 27, 2017

mvdbeek reviewed Jul 28, 2017

View reviewed changes

afgane requested changes Aug 1, 2017

View reviewed changes

hexylena and others added 21 commits August 10, 2017 17:53

Also load yaml sample files. cc @jmchilton, is this ok?

538ef00

correct name to type

3e8919c

fix import order

5563ecc

fix css class

00ee109

not a useful test in this location

4c36b5a

Remove unneeded messages.mako.

a2b0206

rebuild client

4146123

Merge pull request galaxyproject#4305 from erasche/bug-report-actions

6d7d4e9

Refactor bug reporting into plugins / Remove bug report mako

Merge pull request galaxyproject#4408 from erasche/grt-pardir

fcd06a9

Small GRT bugfixes

Merge pull request galaxyproject#4329 from glormph/collection_list_sort

e3de5e2

Created alphabetic and numerical sort tool for collection operations

Merge pull request galaxyproject#4254 from ffinfo/vcf_gz_datatype

bcdf2c8

Vcf gz datatype

Merge branch 'dev' into revise_grid_forms

583765c

Merge branch 'revise_grid_forms' into admin_grid_roles

be19b36

Merge branch 'admin_grid_roles' into admin_grid_groups

b38ccbc

Fix lint

ad2f39c

Centralize launching server for reports and tool shed.

72e5a00

Initial outline of allowing launch via uwsgi.

Update galaxy.util.properties to allow YAML.

1bdb99b

Scripts to manage configuration infrastructure.

8189a1f

This has been rebased several times and includes a variety of fixes (some thanks to martenson).

Add uwsgi_schema.yml.

7973a91

Update tool shed to use newer configuration schema.

90e899d

- Implement kwalify schema to validate/define the schema. - Replace ini configuration with YAML (completely backward compatible). - Rebuild a sample configuration from the schema (config/tool_shed.yml.sample).

guerler and others added 25 commits August 21, 2017 08:28

Fix quota function name

092ad6d

Merge pull request galaxyproject#4469 from guerler/fix_quota_api

63d1d42

Fix quota function name

replace exclamation with a clock icon

a6632f3

Merge pull request galaxyproject#4485 from martenson/remove-exclamation

27bd713

replace exclamation with a clock icon

Rebase.

351ac64

Removed white spaces.

f51add2

Merge remote-tracking branch 'origin/CloudObjectStore' into CloudObje…

4546ff7

…ctStore # Conflicts: # lib/galaxy/objectstore/cloud.py

Merge pull request galaxyproject#4458 from xgaia/interactive_environment

a834381

don't uppercase envoverride

Added CloudBridge to the requirements list.

4b9d720

Added CloudBridge to the dependencies check.

6cae07e

(1) Added Cloud to ObjectStore; (2) updated CloudBridge dependencies.

d69807f

- Updated Cloud to account for changes in CloudBridge interface.

7e74613

- Added `Cloud` to objectstore import.

Reverted the changes on package versions, as these changes are

1565fca

requests via a different PR.

Changed boto import style.

2a7a9bd

Updated CloudBridge to its current latest version.

8da60dc

Commented on boto import, and removed a use_reduced_redundancy config.

b256e8b

Removed boto import, and replaced all S3-specific exception catches with

916d8b1

a generic exception--a temporary solution till CloudBridge wraps the exceptions properly.

Consolidated two upload methods, i.e., upload from file and string,

8742eb2

because CloudBridge internally maps to appropriate functions depending on the input type.

Removed a S3-related comment, and swapped an if/else condition.

36d49aa

Removed a comment section.

4e8e20f

Changed indentation of two lines.

7bb5d21

Update CloudBridge to its current latest

e1ac03d

Should use the wheel created at [this PR](galaxyproject/starforge#139).

Add Cloud configuration to object_store_conf.xml.sample

7df7ed2

Removed white spaces.

d1e5c41

Merge branch 'CloudObjectStore' of https://github.com/VJalili/galaxy …

d3d0abe

…into CloudObjectStore

VJalili closed this Aug 23, 2017

VJalili deleted the CloudObjectStore branch August 23, 2017 19:26

VJalili mentioned this pull request Aug 28, 2017

Cloud ObjectStore #4487

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud ObjectStore #4352

Cloud ObjectStore #4352

VJalili commented Jul 27, 2017 •

edited

Loading

mvdbeek Jul 28, 2017

VJalili Jul 28, 2017

afgane left a comment

afgane Aug 1, 2017

VJalili Aug 2, 2017

afgane Aug 1, 2017

VJalili Aug 2, 2017

afgane Aug 1, 2017

VJalili Aug 2, 2017

afgane Aug 1, 2017

afgane Aug 1, 2017

afgane Aug 1, 2017

VJalili Aug 2, 2017 •

edited

Loading

afgane Aug 1, 2017

afgane Aug 1, 2017

VJalili Aug 2, 2017

VJalili commented Aug 2, 2017

afgane commented Aug 2, 2017

VJalili commented Aug 2, 2017

nsoranzo commented Aug 7, 2017

VJalili commented Aug 23, 2017 •

edited

Loading

VJalili commented Aug 28, 2017

Cloud ObjectStore #4352

Cloud ObjectStore #4352

Conversation

VJalili commented Jul 27, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afgane left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VJalili Aug 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VJalili commented Aug 2, 2017

afgane commented Aug 2, 2017

VJalili commented Aug 2, 2017

nsoranzo commented Aug 7, 2017

VJalili commented Aug 23, 2017 • edited Loading

VJalili commented Aug 28, 2017

VJalili commented Jul 27, 2017 •

edited

Loading

VJalili Aug 2, 2017 •

edited

Loading

VJalili commented Aug 23, 2017 •

edited

Loading