Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[obsolete] DF/091: safe dataset metadata update #245

Open
wants to merge 9 commits into
base: pyDKB-storages
Choose a base branch
from

Commits on Apr 10, 2019

  1. DF/91: allow stage to use DKB ES storage as second (backup) source.

    If `--es-config FILE` parameter is specified, use DKB ES storage
    as a backup metadata source in case that in primary source (Rucio)
    information was removed.
    
    Information from Rucio is removed together with the dataset, so if Rucio
    says that dataset is deleted (or not found), the stage can query
    internat DKB storage (ES) to get missed information. This functionality
    is mostly required for "archived" data reprocessing (when we change the
    ETL process or recover missed records).
    mgolosova committed Apr 10, 2019
    Configuration menu
    Copy the full SHA
    75a0ec5 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2019

  1. Merge remote-tracking branch 'origin/pyDKB-storages' into 91-ds-safe

    This branch requires the new functionality, added to the pyDKB library.
    mgolosova committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    f63c1ab View commit details
    Browse the repository at this point in the history
  2. pyDKB/storages: add possibility to log storage configuration.

    Configuration parameters, started with '__', won't be logged.
    mgolosova committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    6ce0fac View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    3537f08 View commit details
    Browse the repository at this point in the history
  4. pyDKB/storages: improve configuration logging (add default parameters).

    Without it we won't see in the log those parameters that have default values,
    if they are missed in the user passed config.
    mgolosova committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    0a3a321 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8f4370f View commit details
    Browse the repository at this point in the history
  6. DF/091: refactoring (make code more DRY).

    Move metadata "update" with info from ES to a separate function.
    mgolosova committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    2734099 View commit details
    Browse the repository at this point in the history
  7. Fix codestyle.

    mgolosova committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    d39022d View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2019

  1. Configuration menu
    Copy the full SHA
    18781e4 View commit details
    Browse the repository at this point in the history