-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[obsolete] DF/091: safe dataset metadata update #245
base: pyDKB-storages
Are you sure you want to change the base?
Commits on Apr 10, 2019
-
DF/91: allow stage to use DKB ES storage as second (backup) source.
If `--es-config FILE` parameter is specified, use DKB ES storage as a backup metadata source in case that in primary source (Rucio) information was removed. Information from Rucio is removed together with the dataset, so if Rucio says that dataset is deleted (or not found), the stage can query internat DKB storage (ES) to get missed information. This functionality is mostly required for "archived" data reprocessing (when we change the ETL process or recover missed records).
Configuration menu - View commit details
-
Copy full SHA for 75a0ec5 - Browse repository at this point
Copy the full SHA 75a0ec5View commit details
Commits on Apr 26, 2019
-
Merge remote-tracking branch 'origin/pyDKB-storages' into 91-ds-safe
This branch requires the new functionality, added to the pyDKB library.
Configuration menu - View commit details
-
Copy full SHA for f63c1ab - Browse repository at this point
Copy the full SHA f63c1abView commit details -
pyDKB/storages: add possibility to log storage configuration.
Configuration parameters, started with '__', won't be logged.
Configuration menu - View commit details
-
Copy full SHA for 6ce0fac - Browse repository at this point
Copy the full SHA 6ce0facView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3537f08 - Browse repository at this point
Copy the full SHA 3537f08View commit details -
pyDKB/storages: improve configuration logging (add default parameters).
Without it we won't see in the log those parameters that have default values, if they are missed in the user passed config.
Configuration menu - View commit details
-
Copy full SHA for 0a3a321 - Browse repository at this point
Copy the full SHA 0a3a321View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f4370f - Browse repository at this point
Copy the full SHA 8f4370fView commit details -
DF/091: refactoring (make code more DRY).
Move metadata "update" with info from ES to a separate function.
Configuration menu - View commit details
-
Copy full SHA for 2734099 - Browse repository at this point
Copy the full SHA 2734099View commit details -
Configuration menu - View commit details
-
Copy full SHA for d39022d - Browse repository at this point
Copy the full SHA d39022dView commit details
Commits on Apr 27, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 18781e4 - Browse repository at this point
Copy the full SHA 18781e4View commit details