Non redundant provenance #101
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
this allows for version conflict handling
a default value could lead to problems in future
record skipping is based on provenance software version this updated version writes null values for latest products, rather than not writing the metadata attribute
alexdunnjpl
requested review from
tloubrieu-jpl,
nutjob4life and
collinss-jpl
as code owners
|
Benchmarking against sbnpsi results in speed-up from 5m30s to 4m20s due to inherent speed improvements, but sbnpsi only has ~250 non-singleton products out of 1.5M total. Results are likely to be significantly more impressive when it's actually avoiding a significant quantity of avoidable db writes . |
nutjob4life
approved these changes
Jan 26, 2024
This was referenced Jan 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rebased on #100 - consider only commit 3f94d33
🗒️ Summary
Implements #92
Modifies behaviour in that now, the latest version of a product will be assigned
"ops:Provenance/ops:superseded_by": nullrather than not having the attribute assigned at all.Implements software-version-based reprocessing avoidance, as already exists for repairkit and ancestry.
Reads all documents, builds version chains for distinct LIDs, drops all singleton products (as no links exist), builds links, tainting any products with changed successor data, then produces updates, skipping up-to-date records unless they have been tainted.
⚙️ Test Data and/or Report
Functional tests pass, but none are relevant to provenance, per #13
Manually tested, comparing updates produced before/after change.
♻️ Related Issues
fixes #92