Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make dbt thread count configurable #190

Merged
merged 8 commits into from
Dec 6, 2024
Merged

feat: make dbt thread count configurable #190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b4a355b
feat: make dbt thread count configurable
njogz Nov 27, 2024
b5e093a
chore: add new env var to docker compose and helm template
njogz Nov 27, 2024
2db946a
chore: wait 1 minute for records
njogz Dec 2, 2024
27379e8
chore: match timeout to be 1 minute
njogz Dec 2, 2024
9dd91be
chore: wait 50s for records
njogz Dec 2, 2024
0eabf1f
chore: merge main and resolve conflicts
njogz Dec 5, 2024
f98f6e3
chore: try increasing timeout
njogz Dec 5, 2024
a0e2733
chore: update dbt configuration and environment variables
Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dbt/.dbt/profiles.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ default:
target: default
outputs:
default:
threads: "{{ env_var('DBT_THREAD_COUNT') | as_number }}"
type: postgres
host: "{{ env_var('POSTGRES_HOST') }}"
user: "{{ env_var('POSTGRES_USER') }}"
Expand Down
1 change: 1 addition & 0 deletions deploy/cht_sync/values.yaml.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ postgres:
port: 5432 # if postgres is outside the cluster

cht_pipeline_branch_url: "https://github.com/medic/cht-pipeline.git#main"
DBT_THREAD_COUNT: 1

# values shared by all couchdb instances
# can be omitted if couchdb instances do not share any values
Expand Down
1 change: 1 addition & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,4 @@ services:
- CHT_PIPELINE_BRANCH_URL=${CHT_PIPELINE_BRANCH_URL}
- DATAEMON_INTERVAL=${DATAEMON_INTERVAL}
- DBT_PACKAGE_TARBALL_URL=${DBT_PACKAGE_TARBALL_URL}
- DBT_THREAD_COUNT=${DBT_THREAD_COUNT}
1 change: 1 addition & 0 deletions env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ POSTGRES_PORT=5432
# dbt
CHT_PIPELINE_BRANCH_URL="https://github.com/medic/cht-pipeline.git#main"
DATAEMON_INTERVAL=5
DBT_THREAD_COUNT=1

# couchdb
COUCHDB_USER=medic
Expand Down
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"scripts": {
"postinstall": "cd couch2pg && npm ci",
"test:e2e": "npm run test:e2e-data && npm run test:e2e-stop-containers && npm run test:e2e-containers && npm run test:e2e-mocha && npm run test:e2e-stop-containers ",
"test:e2e-mocha": "mocha tests/**/*.spec.js --timeout 50000",
"test:e2e-mocha": "mocha tests/**/*.spec.js --timeout 70000",
"lint": "eslint --color --cache .",
"test:e2e-stop-containers": "docker compose --env-file ./tests/.e2e-env -f docker-compose.yml -f docker-compose.couchdb.yml -f docker-compose.postgres.yml -f tests/dbt/docker-compose.yml -f docker-compose.bastion.yml kill && docker compose --env-file ./tests/.e2e-env -f docker-compose.yml -f docker-compose.couchdb.yml -f docker-compose.postgres.yml -f tests/dbt/docker-compose.yml -f docker-compose.bastion.yml rm -f",
"test:e2e-containers": "docker compose --env-file ./tests/.e2e-env -f docker-compose.yml -f docker-compose.couchdb.yml -f docker-compose.postgres.yml -f tests/dbt/docker-compose.yml -f docker-compose.bastion.yml up -d --build --force-recreate && npm run wait-for-couchdb",
Expand Down
3 changes: 2 additions & 1 deletion tests/.e2e-env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,5 @@ COUCHDB_HOST="host.docker.internal"
COUCHDB_PORT=5984
COUCHDB_SECURE=false
POSTGRES_HOST=postgres
BASTION_AUTHORIZED_KEYS_FILE=./tests/utils/bastion-ssh-key.pub
BASTION_AUTHORIZED_KEYS_FILE=./tests/utils/bastion-ssh-key.pub
DBT_THREAD_COUNT=1
Loading