Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TimescaleDB: Add psql to timescaledb migration scripts #1364

Merged
merged 22 commits into from
Jan 8, 2021

Conversation

Nana-EC
Copy link
Collaborator

@Nana-EC Nana-EC commented Dec 16, 2020

Detailed description:
Mirror node using existing v1 schema will need to migrate from a postgres db to a new timescaledb.
We should offer some scripts to help operators migrate with ease

  • Add a csvBackupTables.sql file that backs up v1 tables to separate csv files for future restore
  • Add a createHyperTables.sql file that updated the table schema differences and create Hyper table prior to restore
  • Add a csvRestoreTables.sql file that uses COPY to restore back files for each of the tables
  • Add a updateSchema.sql file that handles schema difference and runs after data migration to ensure data is in sync
  • Add a migration.config file that hold the config values for a script to run all the migration steps
  • Add a migration.sh main script that orchestrates the migration utilizing all the stages
  • Add a init_v2.sql file that may be used to initiate a timescale db node by creating the database and users with appropriate permissions
  • Rename init.sql to init_v1.sql
  • Update the installation.md file with instruction on timescaledb setup
  • Update the operations.md file with instructions on v1 to v2 migration procedure
  • Update importer config with concept of importer user that differs from owner user
  • Update helm chart to support deployment of timescaledb with new custom schema

Which issue(s) this PR fixes:
Fixes #1309

Special notes for your reviewer:
Verified flow in

  • local postgres docker instance to local timescaledb instance

To-do:

  • GCP cloud sql to GKE - May need additional work to setup secrets, IAM access etc

Checklist

  • Documentation added
  • Tests updated

@Nana-EC Nana-EC added enhancement Type: New feature P2 database Area: Database labels Dec 16, 2020
@Nana-EC Nana-EC added this to the Mirror 0.25.0 milestone Dec 16, 2020
@Nana-EC Nana-EC requested a review from a team December 16, 2020 07:29
@Nana-EC Nana-EC self-assigned this Dec 16, 2020
@steven-sheehy steven-sheehy removed this from the Mirror 0.25.0 milestone Dec 18, 2020
@codecov
Copy link

codecov bot commented Dec 22, 2020

Codecov Report

Merging #1364 (2af6559) into master (0e8aa40) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #1364   +/-   ##
=========================================
  Coverage     85.36%   85.36%           
- Complexity      364      365    +1     
=========================================
  Files           257      257           
  Lines          6367     6367           
  Branches        689      689           
=========================================
  Hits           5435     5435           
  Misses          707      707           
  Partials        225      225           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e8aa40...2af6559. Read the comment docs.

@Nana-EC Nana-EC marked this pull request as ready for review December 23, 2020 04:36
@Nana-EC Nana-EC force-pushed the psql-tsdb-migrate branch from 7fe49cc to 9345041 Compare January 4, 2021 22:50
@steven-sheehy steven-sheehy added this to the Mirror 0.26.0 milestone Jan 5, 2021
docker-compose.yml Outdated Show resolved Hide resolved
docs/installation.md Outdated Show resolved Hide resolved
docs/operations.md Outdated Show resolved Hide resolved
docs/operations.md Outdated Show resolved Hide resolved
docs/operations.md Outdated Show resolved Hide resolved
Nana-EC added 18 commits January 8, 2021 10:09
Signed-off-by: Nana-EC <[email protected]>
Signed-off-by: Nana-EC <[email protected]>
Signed-off-by: Nana-EC <[email protected]>
Signed-off-by: Nana-EC <[email protected]>
@Nana-EC Nana-EC force-pushed the psql-tsdb-migrate branch from 579f86e to 2af6559 Compare January 8, 2021 16:13
Copy link
Member

@steven-sheehy steven-sheehy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sonarqubecloud
Copy link

sonarqubecloud bot commented Jan 8, 2021

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

Copy link
Collaborator

@xin-hedera xin-hedera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Nana-EC Nana-EC merged commit 7adb6e2 into master Jan 8, 2021
@Nana-EC Nana-EC deleted the psql-tsdb-migrate branch January 8, 2021 17:48
@steven-sheehy steven-sheehy linked an issue Jan 8, 2021 that may be closed by this pull request
Nana-EC added a commit that referenced this pull request Jan 8, 2021
* Fix token_account primary index

#1364 updated the token_account table to have a primary index on the timestamp.
However, an account can be associated with multiple tokens in a single TokenAssociate transaction.

- Update `V1.33.0__drop_token_account_id.sql` migration file to set `token_account` primary index to use `created_timestamp, token_id`
- Update `V2.0.2__time_scale_index_init.sql` file to set `token_account` primary index to use `created_timestamp, token_id`
- Update `TokenAccountRepositoryTest` with test to capture scenario

Signed-off-by: Nana-EC <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
database Area: Database enhancement Type: New feature P2
Projects
None yet
4 participants