Create script to verify row count for non-course entities #12824
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NicolasCwy
force-pushed
the
v9-non-course-data-migration
branch
from
f201159 to
2bd4b39
Compare
NicolasCwy
added
c.Task
FergusMok
reviewed
Feb 25, 2024
src/client/java/teammates/client/scripts/sql/VerifyNonCourseEntityCounts.java
Show resolved
Hide resolved
FergusMok
approved these changes
Feb 25, 2024
ziqing26
approved these changes
Feb 25, 2024
src/client/java/teammates/client/scripts/sql/VerifyNonCourseEntityCounts.java
Show resolved
Hide resolved
ziqing26
merged commit 59fbae7
into
TEAMMATES:v9-non-course-data-migration
1 check passed
ziqing26
added a commit
that referenced
this pull request
Mar 10, 2024
* Add seed script * [#12048] Add DataMigrationEntitiesBaseScriptSql and DataMigrationForAccountSql (#12766) * Add hibernate connection * Add DataMigrationEntitiesBaseScriptSql and example Account Migration * [#12048] Add verify to seed db (#12767) * Add verify * Revert secrets * Clean up seed script (#12768) * Fix account migration script * Add client property * Add SQL notification migration script * Modify seed script to persist to local database store * Data migration for account * Add seed data for data migration * Add Account and Read Notification * Remove comments * [#12048] Add migration script for Usage Statistics (#12798) * Add migration script for UsageStatistics * Add seed script for usage statistics and add ofy support * Set preview to true by default * [#12048] Add migration script for Account Request (#12799) * Add migration script for acc req * Update migration script * Add checks to notification script (#12836) * Add checks to notification script * Fix comments * Create script to verify row count for non-course entities (#12824) * Create script to verify row count * Add read notification verification * Add comment * Add base script for verifying migrated attributes (#12841) * Add datastore entity comparison function (except readNotification) * Add verify attribute entity base script Co-authored-by: Kevin Foong <[email protected]> * Add verify usage statistics * Add verification script for account request * Amend base script to fetch lazily loaded keys * Fix migration verification for notification * Add migration verification script for account * Save progress * Add changes * Fix bug * Add changes * Add support for pagination * Add changes (#12846) * Add a script to remove datastore non course entities and fix progress bar in seeddb * Add warning for the script * Add user check for removal script * Revert remove script * fix seed script to populate notification UUID correctly * Remove unnecessary whitespace * Add connection verification script * Remove only one entity in Verify Connection Script * Fix seeding of data for data migration (#12873) * Fix notification and readNotification from having different endtime * Fix seed data to create notification to nearest millisecond * Remove commented out println * Uncomment databundle persistance * Remove unnecessary comments and fix format * Revert logs in EntitesDb * V9 Migration: Fix verification pagination, improve logging (#12874) * Fix numPages division bug, add logging, testing paging * Add logging * Remove unnecessary function * Remove unnecessary files, shift some functions * Clean up branch * Fix lint errors * Fix lint errors v2 * Fix lint errors v3 * Fix spotBugsTest --------- Co-authored-by: Nicolas Chang <[email protected]> Co-authored-by: FergusMok <[email protected]> Co-authored-by: Kevin Foong <[email protected]> Co-authored-by: Kevin Foong <[email protected]> Co-authored-by: Kevin Foong <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Outline of Solution
Script compares datastore entity count with the postgres entities count to ensure that all entities have been migrated.
Other alternatives
Datastore Statistics: Apparently these aren't updated in realtime.
Todo
Add limited batch size and cursor for datastore query? Similar to other SQL data migration scripts(since count aggregate is used for the most part)