IDS-328: Submit ABI data to MyTardis #356
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JLoveUOA
approved these changes
Dec 19, 2023
JLoveUOA
pushed a commit
that referenced
this pull request
Jan 24, 2024
* Initial implementation of submitting metadata to MyTardis for ABI data * Use diskcache for temporary md5 caching * Isolate the temporary caching logic so it can be removed more easily * Fix inconsistent prefixing of dataset names, and add some logging * Initial implementation of command-line interface for ABI ingestion * Add optional sorting of file/directory entries when iterating over them * Fix an outdated docstrnig * Delete some temporary code (caching and comments) * Remove a dependency which was used temporarily during development * Move the Timer class to a separate module * Fix tests that were broken by the addition of the sorting option to DirectoryNode * Move the command-line ingestion runner code to a new script * Replace print statements with logger calls
JLoveUOA
pushed a commit
that referenced
this pull request
Jan 26, 2024
* Initial implementation of submitting metadata to MyTardis for ABI data * Use diskcache for temporary md5 caching * Isolate the temporary caching logic so it can be removed more easily * Fix inconsistent prefixing of dataset names, and add some logging * Initial implementation of command-line interface for ABI ingestion * Add optional sorting of file/directory entries when iterating over them * Fix an outdated docstrnig * Delete some temporary code (caching and comments) * Remove a dependency which was used temporarily during development * Move the Timer class to a separate module * Fix tests that were broken by the addition of the sorting option to DirectoryNode * Move the command-line ingestion runner code to a new script * Replace print statements with logger calls
JLoveUOA
pushed a commit
that referenced
this pull request
Jan 26, 2024
* Initial implementation of submitting metadata to MyTardis for ABI data * Use diskcache for temporary md5 caching * Isolate the temporary caching logic so it can be removed more easily * Fix inconsistent prefixing of dataset names, and add some logging * Initial implementation of command-line interface for ABI ingestion * Add optional sorting of file/directory entries when iterating over them * Fix an outdated docstrnig * Delete some temporary code (caching and comments) * Remove a dependency which was used temporarily during development * Move the Timer class to a separate module * Fix tests that were broken by the addition of the sorting option to DirectoryNode * Move the command-line ingestion runner code to a new script * Replace print statements with logger calls
JLoveUOA
pushed a commit
that referenced
this pull request
Jan 26, 2024
* Initial implementation of submitting metadata to MyTardis for ABI data * Use diskcache for temporary md5 caching * Isolate the temporary caching logic so it can be removed more easily * Fix inconsistent prefixing of dataset names, and add some logging * Initial implementation of command-line interface for ABI ingestion * Add optional sorting of file/directory entries when iterating over them * Fix an outdated docstrnig * Delete some temporary code (caching and comments) * Remove a dependency which was used temporarily during development * Move the Timer class to a separate module * Fix tests that were broken by the addition of the sorting option to DirectoryNode * Move the command-line ingestion runner code to a new script * Replace print statements with logger calls
JLoveUOA
pushed a commit
that referenced
this pull request
Jan 29, 2024
* Initial implementation of submitting metadata to MyTardis for ABI data * Use diskcache for temporary md5 caching * Isolate the temporary caching logic so it can be removed more easily * Fix inconsistent prefixing of dataset names, and add some logging * Initial implementation of command-line interface for ABI ingestion * Add optional sorting of file/directory entries when iterating over them * Fix an outdated docstrnig * Delete some temporary code (caching and comments) * Remove a dependency which was used temporarily during development * Move the Timer class to a separate module * Fix tests that were broken by the addition of the sorting option to DirectoryNode * Move the command-line ingestion runner code to a new script * Replace print statements with logger calls
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Submit the ABI metadata to MyTardis. The ABI ingestion runner script was already parsing the data directory to assemble the metadata, but not actually sending the data to MyTardis; it was just logging it. Now we use the
IngestionFactoryto send the data to MyTardis.This PR also creates a basic initial command-line interface for the ingestion runner script, because it's very simple.