-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] pyDKB: storages #277
base: pyDKB-loggable-object
Are you sure you want to change the base?
Commits on Jan 9, 2020
-
pyDKB: introduce pyDKB.storages submodule.
Sometimes we have to interact with the same storage in different scripts. When it happens we got to implement same functionality multiple times: read configuration, check that Python module is available, create client, ... This submodule is a place to implement this once and then reuse whenever it is needed. Initial module structire: ``` pyDKB |---> storages |---> exceptions | |---> StorageException | |---> NotFound | |---> client |---> Client ```
Configuration menu - View commit details
-
Copy full SHA for b014051 - Browse repository at this point
Copy the full SHA b014051View commit details -
pyDKB/storages: add mechanics to get clients for known storages.
Suggested way of the `pyDKB.storages` usage is: ``` from pyDKB import storages def processA(...): rucio_client = storages.getClient('rucio', 'atlas') rucio_client.get_metadata(...) ... def processB(...): rucio_client = storages.getClient('rucio', 'atlas') rucio_client.get_metadata(...) ... ``` The client used in `processA` and `processB` (and at every call of a function) will be the same instance, initialized at first call. In other words, client classes for individual storages are "singleton" classes (as long as used via `getClient()` methods, not directly). "Scope" abstraction is introduced to keep project-specific fucntionality localised in a single submodule, preventing it from infiltration into other, more general modules. Each "scope" is nothing but a submodule of `pyDKB.storages`. It must contain pre-configured clients for the storages, used in a given project and support general access method to these clients (`getClient()` function).
Configuration menu - View commit details
-
Copy full SHA for fcce139 - Browse repository at this point
Copy the full SHA fcce139View commit details -
pyDKB/storages: add possibility to set default scope.
It allows to get clients simply by name, not specifying the scope every now and again: ``` from pyDKB import storages storages.setScope('atlas') def processA(...): s = storages.getClient('rucio') s.get_metadata(...) ```
Configuration menu - View commit details
-
Copy full SHA for d36a706 - Browse repository at this point
Copy the full SHA d36a706View commit details -
pyDKB/storages: remove reference to client object from the interface …
…class. The idea of the interface to provide common set of methods to interact with different storages, not to (re-)implement all possible methods of interaction with storages. But different storages may have specific methods, and calling them as `my_client.c.client_method_A()` will look a bit wierd: `my_client.client_method_A()` looks more natural. To avoid re-implementation of all useful methods in this way: ``` def client_methodA(self, *args, **kwargs): self.c.client_method_A(*args, **kwargs) ``` multiple inheritance will be used: ``` class MyClient(pyDKB.storages.client.Client, ParentClientClass): ... ``` In case of `ParentClientClass` having same methods as Client, by default `Client` (interface) method will be used, raising `NotImplementedError`. If `ParentClientClass` method should be used, it is to be specified explicitly.
Configuration menu - View commit details
-
Copy full SHA for c48b02b - Browse repository at this point
Copy the full SHA c48b02bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1e073dd - Browse repository at this point
Copy the full SHA 1e073ddView commit details -
pyDKB/storages: codestyle fix (reserved word).
"id" is a reserved word in Python, so it should not be used as variable or parameter name.
Configuration menu - View commit details
-
Copy full SHA for 9dd2f5e - Browse repository at this point
Copy the full SHA 9dd2f5eView commit details -
pyDKB/storages: implement
RucioClient.get()
method.Original Rucio `Client` does not have `get()` method.
Configuration menu - View commit details
-
Copy full SHA for 5602857 - Browse repository at this point
Copy the full SHA 5602857View commit details -
Configuration menu - View commit details
-
Copy full SHA for b8ddd92 - Browse repository at this point
Copy the full SHA b8ddd92View commit details -
pyDKB/storages: add "bare" interface for ES client.
Currently it is nothing but a wrapper around standard `elasticsearch.Elasticsearch` class, except that its `get()` method is overridden with `Client.get()` that raises `NotImplementedError`.
Configuration menu - View commit details
-
Copy full SHA for ea6f358 - Browse repository at this point
Copy the full SHA ea6f358View commit details -
pyDKB/storages: add implementation of
ESClient.configure()
method.It allows to pass configuration to the client as hash. `__init__()` accepts parameters in the form consistent with the `elasticsearch.Elasticsearch()` parameters, so it is not possible to pass default index name when the object is created.
Configuration menu - View commit details
-
Copy full SHA for 04a2721 - Browse repository at this point
Copy the full SHA 04a2721View commit details -
Configuration menu - View commit details
-
Copy full SHA for b7df5cd - Browse repository at this point
Copy the full SHA b7df5cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f6bb07 - Browse repository at this point
Copy the full SHA 2f6bb07View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5c32433 - Browse repository at this point
Copy the full SHA 5c32433View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d3c5cd - Browse repository at this point
Copy the full SHA 1d3c5cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9111b75 - Browse repository at this point
Copy the full SHA 9111b75View commit details -
Co-Authored-By: Evildoor <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 37de231 - Browse repository at this point
Copy the full SHA 37de231View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6360f5d - Browse repository at this point
Copy the full SHA 6360f5dView commit details
Commits on Jan 10, 2020
-
pyDKB/misc: improve error handling in
try_to_import()
.There's only two possibilities: an exception was or was not thrown. Whatever exception it is, it indicates that we failed to import something we wanted to import => so we must return a value that indicates "import has failed" -- `False`.
Configuration menu - View commit details
-
Copy full SHA for 5891f41 - Browse repository at this point
Copy the full SHA 5891f41View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f06b83 - Browse repository at this point
Copy the full SHA 2f06b83View commit details -
pyDKB/misc: introduce special return vale for
try_to_import()
.Who knows what value has this or that attribute one wants to import, right? And `False` looks like a pretty possible one... while string "NOT IMPORTED VALUE" should be less expected and I believe it won't conflict with real attributes values (except itself, but hopefully no one will use this function to import something added as service variable for this function itself).
Configuration menu - View commit details
-
Copy full SHA for 10ac41a - Browse repository at this point
Copy the full SHA 10ac41aView commit details