[stardog] Disable backup restore testing #348
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ramón Cahenzli <[email protected]>
Signed-off-by: Ramón Cahenzli <[email protected]>
psy-q
force-pushed
the
disable_backup_restore_testing
branch
from
834f920 to
198207e
Compare
|
Instead of commenting out stuff, why not simply remove it? This is Git after all.... |
Signed-off-by: Ramón Cahenzli <[email protected]>
psy-q
force-pushed
the
disable_backup_restore_testing
branch
from
7b14667 to
d35e9e6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
The current tactic of comparing restored triple counts to backed up triple counts has caused problems with phantom databases remaining in Stardog's memory for months, leading to cluster sync problems. We're now also seeing write lock acquisition failures during the backup jobs, especially when long transactions are happening, which makes it less likely to actually get a working backup at all, let alone a restore. Backups on one large database rarely succeed even after running for 22h.
The restore also puts a lot of additional IOPS/CPU load on all nodes that could be better spent serving production workloads. The nodes not only have to restore the backup but also recompute indexes afterwards.
Thirdly, according to Stardog this measure is no longer necessary since the bug causing partial backups is said to be fixed.
We propose to create a separate, non-clustered Stardog node to test restores against instead of using the cluster. Until that's ready, this PR wants to comment out the restore functionality to see if it fixes some of the cluster sync problems.
Checklist
[chart]Signed-off by: Ramón Cahenzli [email protected]