Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] pyDKB: storages #277

Draft
wants to merge 20 commits into
base: pyDKB-loggable-object
Choose a base branch
from
Draft

Commits on Jan 9, 2020

  1. pyDKB: introduce pyDKB.storages submodule.

    Sometimes we have to interact with the same storage in different
    scripts. When it happens we got to implement same functionality multiple
    times: read configuration, check that Python module is available, create
    client, ...
    
    This submodule is a place to implement this once and then reuse whenever
    it is needed.
    
    Initial module structire:
    
    ```
    pyDKB
      |---> storages
              |---> exceptions
              |       |---> StorageException
              |       |---> NotFound
              |
              |---> client
                      |---> Client
    ```
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    b014051 View commit details
    Browse the repository at this point in the history
  2. pyDKB/storages: add mechanics to get clients for known storages.

    Suggested way of the `pyDKB.storages` usage is:
    ```
    from pyDKB import storages
    
    def processA(...):
        rucio_client = storages.getClient('rucio', 'atlas')
        rucio_client.get_metadata(...)
        ...
    
    def processB(...):
        rucio_client = storages.getClient('rucio', 'atlas')
        rucio_client.get_metadata(...)
        ...
    ```
    
    The client used in `processA` and `processB` (and at every call of a
    function) will be the same instance, initialized at first call. In other
    words, client classes for individual storages are "singleton" classes
    (as long as used via `getClient()` methods, not directly).
    
    "Scope" abstraction is introduced to keep project-specific fucntionality
    localised in a single submodule, preventing it from infiltration into
    other, more general modules. Each "scope" is nothing but a submodule of
    `pyDKB.storages`. It must contain pre-configured clients for the storages,
    used in a given project and support general access method to these
    clients (`getClient()` function).
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    fcce139 View commit details
    Browse the repository at this point in the history
  3. pyDKB/storages: add possibility to set default scope.

    It allows to get clients simply by name, not specifying the scope every
    now and again:
    ```
    from pyDKB import storages
    storages.setScope('atlas')
    
    def processA(...):
        s = storages.getClient('rucio')
        s.get_metadata(...)
    ```
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    d36a706 View commit details
    Browse the repository at this point in the history
  4. pyDKB/storages: remove reference to client object from the interface …

    …class.
    
    The idea of the interface to provide common set of methods to interact
    with different storages, not to (re-)implement all possible methods of
    interaction with storages. But different storages may have specific
    methods, and calling them as `my_client.c.client_method_A()` will look a
    bit wierd: `my_client.client_method_A()` looks more natural.
    
    To avoid re-implementation of all useful methods in this way:
    ```
      def client_methodA(self, *args, **kwargs):
        self.c.client_method_A(*args, **kwargs)
    ```
    
    multiple inheritance will be used:
    ```
      class MyClient(pyDKB.storages.client.Client, ParentClientClass):
        ...
    ```
    
    In case of `ParentClientClass` having same methods as Client, by default
    `Client` (interface) method will be used, raising `NotImplementedError`.
    If `ParentClientClass` method should be used, it is to be specified
    explicitly.
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    c48b02b View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    1e073dd View commit details
    Browse the repository at this point in the history
  6. pyDKB/storages: codestyle fix (reserved word).

    "id" is a reserved word in Python, so it should not be used as variable
    or parameter name.
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    9dd2f5e View commit details
    Browse the repository at this point in the history
  7. pyDKB/storages: implement RucioClient.get() method.

    Original Rucio `Client` does not have `get()` method.
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    5602857 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    b8ddd92 View commit details
    Browse the repository at this point in the history
  9. pyDKB/storages: add "bare" interface for ES client.

    Currently it is nothing but a wrapper around standard
    `elasticsearch.Elasticsearch` class, except that its `get()` method is
    overridden with `Client.get()` that raises `NotImplementedError`.
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    ea6f358 View commit details
    Browse the repository at this point in the history
  10. pyDKB/storages: add implementation of ESClient.configure() method.

    It allows to pass configuration to the client as hash.
    `__init__()` accepts parameters in the form consistent with the
    `elasticsearch.Elasticsearch()` parameters, so it is not possible to
    pass default index name when the object is created.
    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    04a2721 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    b7df5cd View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    2f6bb07 View commit details
    Browse the repository at this point in the history
  13. pyDKB/storages: typo fix.

    mgolosova committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    5c32433 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    1d3c5cd View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    9111b75 View commit details
    Browse the repository at this point in the history
  16. pyDKB/storages: typo fix.

    Co-Authored-By: Evildoor <[email protected]>
    mgolosova and Evildoor committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    37de231 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    6360f5d View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2020

  1. pyDKB/misc: improve error handling in try_to_import().

    There's only two possibilities: an exception was or was not thrown.
    Whatever exception it is, it indicates that we failed to import
    something we wanted to import => so we must return a value that
    indicates "import has failed" -- `False`.
    mgolosova committed Jan 10, 2020
    Configuration menu
    Copy the full SHA
    5891f41 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2f06b83 View commit details
    Browse the repository at this point in the history
  3. pyDKB/misc: introduce special return vale for try_to_import().

    Who knows what value has this or that attribute one wants to import,
    right? And `False` looks like a pretty possible one... while string "NOT
    IMPORTED VALUE" should be less expected and I believe it won't conflict
    with real attributes values (except itself, but hopefully no one will
    use this function to import something added as service variable for this
    function itself).
    mgolosova committed Jan 10, 2020
    Configuration menu
    Copy the full SHA
    10ac41a View commit details
    Browse the repository at this point in the history