Agora Data Manager is a tool that loads the JSON files into Agora's document database instances in our AWS environments.
This project allows Agora maintainers to update the Agora database with new versions of gene data from Synapse. This is a manually triggered, self-service update.
To deploy an updated data version to the Agora development database
- Increment
data-version
indata-manifest.json
on thedevelop
branch. - Commit the change
- The Github action CI system automatically updates the dev DB
To deploy an updated data version to the Agora staging database:
- Merge the data-version update from the dev branch to the staging branch.
- The Github action CI system automatically updates the dev DB
To deploy an updated data version to the Agora production database:
- Merge the data-version update from the staging branch to the production branch.
- The Github action CI system automatically updates the dev DB
The following secrets need to be setup in Github for the scripts to deploy database updates:
Global secrets:
Variable | Description | Example |
---|---|---|
SYNAPSE_USERNAME | The Synapse service user | syn-service-user |
SYNAPSE_PASSWORD | The Synapse service user password | supersecret |
Context specific secrets for each environment that corresponds to a git branch (develop/staging/prod):
Variable | Description | Example |
---|---|---|
DB_HOST | The database host | dbcluster-mr0a782pfjnk.cluster-ctcayu3de2lt.us-east-1.docdb.amazonaws.com |
DB_USER | The database user | dbuser |
DB_PASS | The database password | supersecret |
agora2-infra repository deploys a bastian host in AWS for each environment which have access to
the databases. We manually configure a Github self-hosted runner
for each bastian host, a label is applied to each runner to match the corresponding git branch name (develop/staging/prod).
Each runner corresponds to an environment which corresponds to a git branch. The update is
executed from these runners. When a push happens on a branch (i.e. develop), the update
is executed on the agora-bastian-develop
runner which in turn updates the development database.
Github self hosted runners are deployed with a [Sceptre template config file])(https://github.com/Sage-Bionetworks/agora2-infra/blob/main/config/agoradev/develop/agora-bastian.yaml).
Self Hosted Runner setup:
- Deploy the template to the Agora AWS account.
- Login to AWS console and goto
EC2 -> select the deployed instance -> Connect -> Session Manager -> Connect
to gain ssh access to the instance. - Follow the instructions to install the Github self hosted runner. We installed it to the
/home/ssm-user/actions-runner
folder. - Run the
config.sh
script to configure the runner. !! Important !! Make sure to set the runnername
andlabel
corresponding to the desired deployment environment (develop/staging/prod)..
sh-4.2$ pwd
/home/ssm-user/actions-runner
sh-4.2$ ./config.sh --url https://github.com/Sage-Bionetworks/agora-data-manager --token XXXXXXXXXXXXXXXXX6VLI
--------------------------------------------------------------------------------
| ____ _ _ _ _ _ _ _ _ |
| / ___(_) |_| | | |_ _| |__ / \ ___| |_(_) ___ _ __ ___ |
| | | _| | __| |_| | | | | '_ \ / _ \ / __| __| |/ _ \| '_ \/ __| |
| | |_| | | |_| _ | |_| | |_) | / ___ \ (__| |_| | (_) | | | \__ \ |
| \____|_|\__|_| |_|\__,_|_.__/ /_/ \_\___|\__|_|\___/|_| |_|___/ |
| |
| Self-hosted runner registration |
| |
--------------------------------------------------------------------------------
# Authentication
√ Connected to GitHub
# Runner Registration
Enter the name of the runner group to add this runner to: [press Enter for Default]
Enter the name of runner: [press Enter for ip-10-XXX-XXX-XXX] agora-bastian-prod
This runner will have the following labels: 'self-hosted', 'Linux', 'X64'
Enter any additional labels (ex. label-1,label-2): [press Enter to skip] prod
√ Runner successfully added
√ Runner connection is good
# Runner settings
Enter name of work folder: [press Enter for _work]
√ Settings Saved.
- Setup the GH runner agent to run as a service
- Run the agent and then check the GH Runners page to make sure that the runner is in
Idle
status.