Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge the Databases #1372

Merged
merged 36 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
d68cfd6
Remove Distribute SQL steps
Mythicaeda Feb 5, 2024
e769cfc
Move Merlin DB Files
Mythicaeda Feb 5, 2024
6fe6d15
Move Scheduler DB Files
Mythicaeda Feb 5, 2024
9a61141
Move Sequencing DB Files
Mythicaeda Feb 5, 2024
6ae9b8a
Move UI DB Files
Mythicaeda Feb 5, 2024
2134c69
Add Shared Utility Functions
Mythicaeda Feb 6, 2024
5a481f6
Add Gitkeep for new DB
Mythicaeda Mar 1, 2024
f9c2a79
Add Schema Qualifiers (Permissions)
Mythicaeda Feb 6, 2024
4e14365
Add Schema Qualifiers (Tags)
Mythicaeda Feb 6, 2024
70b17a2
Add Schema Qualifiers (Hasura)
Mythicaeda Mar 1, 2024
ea142c2
Add Schema Qualifiers (Merlin)
Mythicaeda Mar 1, 2024
81b16f2
Add Schema Qualifiers (Scheduler)
Mythicaeda Mar 1, 2024
9e72667
Add Schema Qualifiers (Sequencing)
Mythicaeda Mar 1, 2024
51f66e2
Add Schema Qualifiers (UI)
Mythicaeda Mar 1, 2024
bdbef83
Create Init Files
Mythicaeda Mar 4, 2024
d61bf09
Update DB Init script
Mythicaeda Mar 4, 2024
39ce38c
Reorganize Hasura Metadata
Mythicaeda Mar 4, 2024
18262f7
Update Hasura Metadata
Mythicaeda Mar 4, 2024
7fd73a4
Update Docker-Compose
Mythicaeda Mar 4, 2024
e1f4cf8
Add foreign keys (Scheduler)
Mythicaeda Mar 7, 2024
1a79f40
Add foreign keys (UI)
Mythicaeda Mar 7, 2024
665efd1
Add foreign keys (Sequencing)
Mythicaeda Mar 7, 2024
52ef1bf
Replace Manual Configurations in Hasura Metadata
Mythicaeda Mar 11, 2024
67d96ef
Update DB Actions in Backend (Merlin)
Mythicaeda Mar 11, 2024
4342368
Update DB Actions in Backend (Scheduler)
Mythicaeda Mar 12, 2024
2fd7462
Update DB Actions in Backend (Sequencing)
Mythicaeda Mar 12, 2024
c5fec88
Update DB Actions in Backend (Timeline Libray)
Mythicaeda Mar 21, 2024
57dfc32
Merge merlin.status_t and scheduer.status_t into util_functions.reque…
Mythicaeda Mar 13, 2024
bdbc423
Update DB Tests
Mythicaeda Mar 13, 2024
b79c1a9
Update E2ETests
Mythicaeda Mar 13, 2024
cdaa892
Support one DB user per Aerie service
Mythicaeda Mar 13, 2024
a756a88
Update DB Comparison Workflow
Mythicaeda Mar 21, 2024
162141c
Update Aerie Migration Script
Mythicaeda Mar 21, 2024
3ca416a
Add new envvars to Workflows
Mythicaeda Mar 26, 2024
3c18852
Merge Aerie DB Script
Mythicaeda Mar 26, 2024
3a285c9
Handle Orphaned Rows Before Adding FKeys
Mythicaeda Mar 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
18 changes: 16 additions & 2 deletions .env.template
Original file line number Diff line number Diff line change
@@ -1,6 +1,20 @@
AERIE_USERNAME=
AERIE_PASSWORD=

GATEWAY_USERNAME=
GATEWAY_PASSWORD=

MERLIN_USERNAME=
MERLIN_PASSWORD=

SCHEDULER_USERNAME=
SCHEDULER_PASSWORD=

SEQUENCING_USERNAME=
SEQUENCING_PASSWORD=

POSTGRES_PASSWORD=
POSTGRES_USER=

HASURA_GRAPHQL_ADMIN_SECRET=
HASURA_GRAPHQL_JWT_SECRET=
POSTGRES_USER=
POSTGRES_PASSWORD=
105 changes: 18 additions & 87 deletions .github/scripts/compareDatabasesDown.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,96 +3,27 @@
mkdir results
mkdir comparison

PGCMPINPUT1=./pgdumpv1_0_1/AerieMerlinV1_0_1 PGCMPINPUT2=./pgdumpmigrateddown/AerieMerlinMigratedDown PGCLABEL1=AerieMerlinV1_0_1 PGCLABEL2=AerieMerlinMigratedDown PGCFULLOUTPUT=./comparison/fulloutputMerlin.txt PGCUNEXPLAINED=./comparison/unexplainedMerlin.txt PGCBADEXPLAIN=./comparison/badexplanationsMerlin.txt PGDB=postgres PGBINDIR=/usr/bin PGCEXPLANATIONS=./explanations_merlin ./pgcmp
PGCMPINPUT1=./pgdumpV2_8_0/AerieV2_8_0 \
PGCMPINPUT2=./pgdumpmigrateddown/AerieMigratedDown \
PGCLABEL1=AerieV2_8_0 \
PGCLABEL2=AerieMigratedDown \
PGCFULLOUTPUT=./comparison/fulloutput.txt \
PGCUNEXPLAINED=./comparison/unexplained.txt \
PGCBADEXPLAIN=./comparison/badexplanations.txt \
PGDB=postgres \
PGBINDIR=/usr/bin \
PGCOMITSCHEMAS="('hdb_catalog'),('pg_catalog'),('information_schema')" \
./pgcmp
return_code=$?
if [ $return_code -ne 0 ]; then
echo "AerieMerlin comparison failed - return code=$return_code"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "==================\nAerieMerlin Results\n==================\n" >> results/fulloutput
printf "==================\nAerieMerlin Results\n==================\n" >> results/unexplained
printf "==================\nAerieMerlin Results\n==================\n" >> results/badexplanations
printf "==================\nAerieMerlin Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputMerlin.txt >> results/fulloutput
cat ./comparison/unexplainedMerlin.txt >> results/unexplained
cat ./comparison/badexplanationsMerlin.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
else
echo "AerieMerlin comparison succeeded"
fi

PGCMPINPUT1=./pgdumpv1_0_1/AerieSchedulerV1_0_1 PGCMPINPUT2=./pgdumpmigrateddown/AerieSchedulerMigratedDown PGCLABEL1=AerieSchedulerV1_0_1 PGCLABEL2=AerieSchedulerMigratedDown PGCFULLOUTPUT=./comparison/fulloutputScheduler.txt PGCUNEXPLAINED=./comparison/unexplainedScheduler.txt PGCBADEXPLAIN=./comparison/badexplanationsScheduler.txt PGDB=postgres PGBINDIR=/usr/bin PGCEXPLANATIONS=./explanations ./pgcmp
retcode=$?
if [ $retcode -gt $return_code ]; then
return_code=$retcode
fi

if [ $retcode -ne 0 ]; then
echo "AerieScheduler comparison failed - return code=$retcode"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/fulloutput
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/unexplained
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/badexplanations
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputScheduler.txt >> results/fulloutput
cat ./comparison/unexplainedScheduler.txt >> results/unexplained
cat ./comparison/badexplanationsScheduler.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
else
echo "AerieScheduler comparison succeeded"
fi

PGCMPINPUT1=./pgdumpv1_0_1/AerieSequencingV1_0_1 PGCMPINPUT2=./pgdumpmigrateddown/AerieSequencingMigratedDown PGCLABEL1=AerieSequencingV1_0_1 PGCLABEL2=AerieSequencingMigratedDown PGCFULLOUTPUT=./comparison/fulloutputSequencing.txt PGCUNEXPLAINED=./comparison/unexplainedSequencing.txt PGCBADEXPLAIN=./comparison/badexplanationsSequencing.txt PGDB=postgres PGBINDIR=/usr/bin PGCEXPLANATIONS=./explanations ./pgcmp
retcode=$?
if [ $retcode -gt $return_code ]; then
return_code=$retcode
fi

if [ $retcode -ne 0 ]; then
echo "AerieSequencing comparison failed - return code=$retcode"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/fulloutput
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/unexplained
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/badexplanations
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputSequencing.txt >> results/fulloutput
cat ./comparison/unexplainedSequencing.txt >> results/unexplained
cat ./comparison/badexplanationsSequencing.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
else
echo "AerieSequencing comparison succeeded"
fi

PGCMPINPUT1=./pgdumpv1_0_1/AerieUIV1_0_1 PGCMPINPUT2=./pgdumpmigrateddown/AerieUIMigratedDown PGCLABEL1=AerieUIV1_0_1 PGCLABEL2=AerieUIMigratedDown PGCFULLOUTPUT=./comparison/fulloutputUI.txt PGCUNEXPLAINED=./comparison/unexplainedUI.txt PGCBADEXPLAIN=./comparison/badexplanationsUI.txt PGDB=postgres PGBINDIR=/usr/bin PGCEXPLANATIONS=./explanations ./pgcmp
retcode=$?
if [ $retcode -gt $return_code ]; then
return_code=$retcode
fi

if [ $retcode -ne 0 ]; then
echo "AerieUI comparison failed - return code=$retcode"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "\n==================\nAerieUI Results\n==================\n" >> results/fulloutput
printf "\n==================\nAerieUI Results\n==================\n" >> results/unexplained
printf "\n==================\nAerieUI Results\n==================\n" >> results/badexplanations
printf "\n==================\nAerieUI Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputUI.txt >> results/fulloutput
cat ./comparison/unexplainedUI.txt >> results/unexplained
cat ./comparison/badexplanationsUI.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
if [ $return_code -ne 0 ]; then
echo "Database schema comparison failed - return code=$return_code"
mv ./comparison/fulloutput.txt results/fulloutput
mv ./comparison/unexplained.txt results/unexplained
mv ./comparison/badexplanations.txt results/badexplanations
mv /tmp/perform-comparison.log results/perform-comparison.log
else
echo "AerieUI comparison succeeded"
echo "Database schema comparison succeeded"
fi

exit $return_code
105 changes: 18 additions & 87 deletions .github/scripts/compareDatabasesUp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,96 +3,27 @@
mkdir results
mkdir comparison

PGCMPINPUT1=./pgdumpmigrated/AerieMerlinMigrated PGCMPINPUT2=./pgdumpraw/AerieMerlinRaw PGCLABEL1=AerieMerlinMigrated PGCLABEL2=AerieMerlinRaw PGCFULLOUTPUT=./comparison/fulloutputMerlin.txt PGCUNEXPLAINED=./comparison/unexplainedMerlin.txt PGCBADEXPLAIN=./comparison/badexplanationsMerlin.txt PGDB=postgres PGBINDIR=/usr/bin ./pgcmp
PGCMPINPUT1=./pgdumpmigrated/AerieMigrated \
PGCMPINPUT2=./pgdumpraw/AerieRaw \
PGCLABEL1=AerieMigrated \
PGCLABEL2=AerieRaw \
PGCFULLOUTPUT=./comparison/fulloutput.txt \
PGCUNEXPLAINED=./comparison/unexplained.txt \
PGCBADEXPLAIN=./comparison/badexplanations.txt \
PGDB=postgres \
PGBINDIR=/usr/bin \
PGCOMITSCHEMAS="('hdb_catalog'),('pg_catalog'),('information_schema')" \
./pgcmp
return_code=$?
if [ $return_code -ne 0 ]; then
echo "AerieMerlin comparison failed - return code=$return_code"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "==================\nAerieMerlin Results\n==================\n" >> results/fulloutput
printf "==================\nAerieMerlin Results\n==================\n" >> results/unexplained
printf "==================\nAerieMerlin Results\n==================\n" >> results/badexplanations
printf "==================\nAerieMerlin Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputMerlin.txt >> results/fulloutput
cat ./comparison/unexplainedMerlin.txt >> results/unexplained
cat ./comparison/badexplanationsMerlin.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
else
echo "AerieMerlin comparison succeeded"
fi

PGCMPINPUT1=./pgdumpmigrated/AerieSchedulerMigrated PGCMPINPUT2=./pgdumpraw/AerieSchedulerRaw PGCLABEL1=AerieSchedulerMigrated PGCLABEL2=AerieSchedulerRaw PGCFULLOUTPUT=./comparison/fulloutputScheduler.txt PGCUNEXPLAINED=./comparison/unexplainedScheduler.txt PGCBADEXPLAIN=./comparison/badexplanationsScheduler.txt PGDB=postgres PGBINDIR=/usr/bin ./pgcmp
retcode=$?
if [ $retcode -gt $return_code ]; then
return_code=$retcode
fi

if [ $retcode -ne 0 ]; then
echo "AerieScheduler comparison failed - return code=$retcode"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/fulloutput
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/unexplained
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/badexplanations
printf "\n==================\nAerieScheduler Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputScheduler.txt >> results/fulloutput
cat ./comparison/unexplainedScheduler.txt >> results/unexplained
cat ./comparison/badexplanationsScheduler.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
else
echo "AerieScheduler comparison succeeded"
fi

PGCMPINPUT1=./pgdumpmigrated/AerieSequencingMigrated PGCMPINPUT2=./pgdumpraw/AerieSequencingRaw PGCLABEL1=AerieSequencingMigrated PGCLABEL2=AerieSequencingRaw PGCFULLOUTPUT=./comparison/fulloutputSequencing.txt PGCUNEXPLAINED=./comparison/unexplainedSequencing.txt PGCBADEXPLAIN=./comparison/badexplanationsSequencing.txt PGDB=postgres PGBINDIR=/usr/bin ./pgcmp
retcode=$?
if [ $retcode -gt $return_code ]; then
return_code=$retcode
fi

if [ $retcode -ne 0 ]; then
echo "AerieSequencing comparison failed - return code=$retcode"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/fulloutput
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/unexplained
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/badexplanations
printf "\n==================\nAerieSequencing Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputSequencing.txt >> results/fulloutput
cat ./comparison/unexplainedSequencing.txt >> results/unexplained
cat ./comparison/badexplanationsSequencing.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
else
echo "AerieSequencing comparison succeeded"
fi

PGCMPINPUT1=./pgdumpmigrated/AerieUIMigrated PGCMPINPUT2=./pgdumpraw/AerieUIRaw PGCLABEL1=AerieUIMigrated PGCLABEL2=AerieUIRaw PGCFULLOUTPUT=./comparison/fulloutputUI.txt PGCUNEXPLAINED=./comparison/unexplainedUI.txt PGCBADEXPLAIN=./comparison/badexplanationsUI.txt PGDB=postgres PGBINDIR=/usr/bin ./pgcmp
retcode=$?
if [ $retcode -gt $return_code ]; then
return_code=$retcode
fi

if [ $retcode -ne 0 ]; then
echo "AerieUI comparison failed - return code=$retcode"
touch results/fulloutput
touch results/unexplained
touch results/badexplanations
touch results/perform-comparison.log
printf "\n==================\nAerieUI Results\n==================\n" >> results/fulloutput
printf "\n==================\nAerieUI Results\n==================\n" >> results/unexplained
printf "\n==================\nAerieUI Results\n==================\n" >> results/badexplanations
printf "\n==================\nAerieUI Results\n==================\n" >> results/perform-comparison.log
cat ./comparison/fulloutputUI.txt >> results/fulloutput
cat ./comparison/unexplainedUI.txt >> results/unexplained
cat ./comparison/badexplanationsUI.txt >> results/badexplanations
cat /tmp/perform-comparison.log >> results/perform-comparison.log
if [ $return_code -ne 0 ]; then
echo "Database schema comparison failed - return code=$return_code"
mv ./comparison/fulloutput.txt results/fulloutput
mv ./comparison/unexplained.txt results/unexplained
mv ./comparison/badexplanations.txt results/badexplanations
mv /tmp/perform-comparison.log results/perform-comparison.log
else
echo "AerieUI comparison succeeded"
echo "Database schema comparison succeeded"
fi

exit $return_code
24 changes: 0 additions & 24 deletions .github/scripts/explanations

This file was deleted.

16 changes: 0 additions & 16 deletions .github/scripts/explanations_merlin_down

This file was deleted.

8 changes: 8 additions & 0 deletions .github/workflows/load-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,14 @@ env:
HASURA_GRAPHQL_JWT_SECRET: '{ "type": "HS256", "key": "oursupersecretsupersecurekey1234567890" }'
POSTGRES_USER: "postgres"
POSTGRES_PASSWORD: "postgres"
GATEWAY_USERNAME: "${{secrets.GATEWAY_USERNAME}}"
GATEWAY_PASSWORD: "${{secrets.GATEWAY_PASSWORD}}"
MERLIN_USERNAME: "${{secrets.MERLIN_USERNAME}}"
MERLIN_PASSWORD: "${{secrets.MERLIN_PASSWORD}}"
SCHEDULER_USERNAME: "${{secrets.SCHEDULER_USERNAME}}"
SCHEDULER_PASSWORD: "${{secrets.SCHEDULER_PASSWORD}}"
SEQUENCING_USERNAME: "${{secrets.SEQUENCING_USERNAME}}"
SEQUENCING_PASSWORD: "${{secrets.SEQUENCING_PASSWORD}}"

jobs:
load-test:
Expand Down
Loading
Loading