Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CI tests for different target connection configs #75

Merged
merged 17 commits into from
Jan 5, 2021
45 changes: 44 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ orbs:
python: circleci/[email protected]

jobs:
integration-sqlserver:
integration-sqlserver: &sqlserver
docker:
- image: dataders/pyodbc:1.4
- image: mcr.microsoft.com/mssql/server:2019-latest
Expand All @@ -26,6 +26,46 @@ jobs:
- run:
name: Test adapter on SQL Server against dbt-adapter-tests
command: tox -e integration-sqlserver
connection-sqlserver:
<<: *sqlserver
steps:
- checkout
- python/install-packages:
pkg-manager: pip
- run:
name: wait for SQL Server container to set up
command: sleep 30
- run:
name: prep for connecting
command: |
mkdir -p ~/.dbt
cd test/integration
cp sample.profiles.yml ~/.dbt/profiles.yml

- run:
name: cnxn -- SQL Server - local sql cred
command: |
cd test/integration
dbt compile --target sqlserver_local_userpass
- run:
name: cnxn -- SQL Server - local sql cred encrypt
command: |
cd test/integration
dbt compile --target sqlserver_local_encrypt
- run:
name: cnxn -- Azure SQL - SQL cred
command: |
cd test/integration
dbt compile --target azuresql_sqlcred

# TODO drop support for this method in favor of
# https://github.com/dbt-msft/dbt-sqlserver/pull/71
# - run:
# name: cnxn -- Azure SQL - AAD Password
# command: |
# cd test/integration
# dbt compile --target azuresql_aad

integration-azuresql:
docker:
- image: dataders/pyodbc:1.4
Expand All @@ -41,6 +81,9 @@ jobs:
workflows:
main:
jobs:
- connection-sqlserver:
context:
- DBT_SYNAPSE_PROFILE
- integration-sqlserver
- integration-azuresql:
context:
Expand Down
3 changes: 2 additions & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ black~=20.8b1
pytest-dbt-adapter~=0.3.0
tox==3.2.0
flake8>=3.5.0
certifi==2020.6.20
certifi==2020.6.20
.
6 changes: 6 additions & 0 deletions test/integration/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@

name: 'sqlserver_integration_tests'
version: '1.0'
config-version: 2

profile: 'integration_tests'
3 changes: 3 additions & 0 deletions test/integration/models/test.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{# inane comment #}
{% set col_name = 'foo' %}
SELECT 1 as {{ col_name }}
42 changes: 42 additions & 0 deletions test/integration/sample.profiles.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# HEY! This file is used in the tsql_utils integrations tests with CircleCI.
# You should __NEVER__ check credentials into version control. Thanks for reading :)

config:
send_anonymous_usage_stats: False
use_colors: True

defaults:
sqlserver: &sqlserver
type: sqlserver
driver: "ODBC Driver 17 for SQL Server"
schema: "dbt_cnxn_test"
port: 1433
threads: 8

integration_tests:
target: sqlserver_local_userpass
outputs:
sqlserver_local_userpass: &sqlserver-local
<<: *sqlserver
host: localhost
database: msdb
username: SA
password: 5atyaNadella
sqlserver_local_encrypt:
<<: *sqlserver-local
encrypt: yes
trust_cert: yes
azuresql_sqlcred: &azuresql
<<: *sqlserver
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding, is this a way to extend de above mentioned profiles, the ones with &<some name>?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's right!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they're called YAML anchors, and I'm on the fence about them legibility-wise. Seems pretty easy to go overboard on. how would you rate the readability of this YAML file, @JCZuurmond?
https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the anchors, indeed you should not use it too much, but it narrows what your focusing on within each segment. For example with sqlserver_local_encrypt it is clear that encrypt: yes and trust_cert: yes is what should be tested - it's like a fixture in testing

host: "{{ env_var('DBT_AZURESQL_SERVER') }}"
database: "{{ env_var('DBT_AZURESQL_DB') }}"
username: "{{ env_var('DBT_AZURESQL_UID') }}"
password: "{{ env_var('DBT_AZURESQL_PWD') }}"
encrypt: yes
trust_cert: yes
azuresql_aad:
<<: *azuresql
authentication: ActiveDirectoryPassword
username: "{{ env_var('DBT_AZURESQL_AAD_USER') }}"
password: "{{ env_var('DBT_AZURESQL_AAD_PASS') }}"