-
Notifications
You must be signed in to change notification settings - Fork 102
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CI tests for different target connection configs #75
Merged
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
5a99979
test cnxn to 2 different targets
dataders 35ac9ae
test cnxn to 2 different targets
dataders 725176f
correct dir
dataders 560ed67
simplify command set
dataders 1877471
simplify cmd set
dataders 0638228
ensure correct dir
dataders c3c1bf4
standardize on single schema
dataders ab47460
anchors for simplicity
dataders 350ddfd
make dbt-sqlserver a dev req
dataders ee46db3
minimize repitition
dataders 16d1f8f
add local encryption method
dataders 0ac9d68
add AAD pass auth
dataders c180728
typo
dataders b6ab266
typo
dataders 10fc373
keep dbt compile occupied
dataders 6526f2b
test connections first
dataders ec33415
temp drop support for AAD Password
dataders File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,7 +4,7 @@ orbs: | |
python: circleci/[email protected] | ||
|
||
jobs: | ||
integration-sqlserver: | ||
integration-sqlserver: &sqlserver | ||
docker: | ||
- image: dataders/pyodbc:1.4 | ||
- image: mcr.microsoft.com/mssql/server:2019-latest | ||
|
@@ -26,6 +26,46 @@ jobs: | |
- run: | ||
name: Test adapter on SQL Server against dbt-adapter-tests | ||
command: tox -e integration-sqlserver | ||
connection-sqlserver: | ||
<<: *sqlserver | ||
steps: | ||
- checkout | ||
- python/install-packages: | ||
pkg-manager: pip | ||
- run: | ||
name: wait for SQL Server container to set up | ||
command: sleep 30 | ||
- run: | ||
name: prep for connecting | ||
command: | | ||
mkdir -p ~/.dbt | ||
cd test/integration | ||
cp sample.profiles.yml ~/.dbt/profiles.yml | ||
|
||
- run: | ||
name: cnxn -- SQL Server - local sql cred | ||
command: | | ||
cd test/integration | ||
dbt compile --target sqlserver_local_userpass | ||
- run: | ||
name: cnxn -- SQL Server - local sql cred encrypt | ||
command: | | ||
cd test/integration | ||
dbt compile --target sqlserver_local_encrypt | ||
- run: | ||
name: cnxn -- Azure SQL - SQL cred | ||
command: | | ||
cd test/integration | ||
dbt compile --target azuresql_sqlcred | ||
|
||
# TODO drop support for this method in favor of | ||
# https://github.com/dbt-msft/dbt-sqlserver/pull/71 | ||
# - run: | ||
# name: cnxn -- Azure SQL - AAD Password | ||
# command: | | ||
# cd test/integration | ||
# dbt compile --target azuresql_aad | ||
|
||
integration-azuresql: | ||
docker: | ||
- image: dataders/pyodbc:1.4 | ||
|
@@ -41,6 +81,9 @@ jobs: | |
workflows: | ||
main: | ||
jobs: | ||
- connection-sqlserver: | ||
context: | ||
- DBT_SYNAPSE_PROFILE | ||
- integration-sqlserver | ||
- integration-azuresql: | ||
context: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,4 +5,5 @@ black~=20.8b1 | |
pytest-dbt-adapter~=0.3.0 | ||
tox==3.2.0 | ||
flake8>=3.5.0 | ||
certifi==2020.6.20 | ||
certifi==2020.6.20 | ||
. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
|
||
name: 'sqlserver_integration_tests' | ||
version: '1.0' | ||
config-version: 2 | ||
|
||
profile: 'integration_tests' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{# inane comment #} | ||
{% set col_name = 'foo' %} | ||
SELECT 1 as {{ col_name }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
# HEY! This file is used in the tsql_utils integrations tests with CircleCI. | ||
# You should __NEVER__ check credentials into version control. Thanks for reading :) | ||
|
||
config: | ||
send_anonymous_usage_stats: False | ||
use_colors: True | ||
|
||
defaults: | ||
sqlserver: &sqlserver | ||
type: sqlserver | ||
driver: "ODBC Driver 17 for SQL Server" | ||
schema: "dbt_cnxn_test" | ||
port: 1433 | ||
threads: 8 | ||
|
||
integration_tests: | ||
target: sqlserver_local_userpass | ||
outputs: | ||
sqlserver_local_userpass: &sqlserver-local | ||
<<: *sqlserver | ||
host: localhost | ||
database: msdb | ||
username: SA | ||
password: 5atyaNadella | ||
sqlserver_local_encrypt: | ||
<<: *sqlserver-local | ||
encrypt: yes | ||
trust_cert: yes | ||
azuresql_sqlcred: &azuresql | ||
<<: *sqlserver | ||
host: "{{ env_var('DBT_AZURESQL_SERVER') }}" | ||
database: "{{ env_var('DBT_AZURESQL_DB') }}" | ||
username: "{{ env_var('DBT_AZURESQL_UID') }}" | ||
password: "{{ env_var('DBT_AZURESQL_PWD') }}" | ||
encrypt: yes | ||
trust_cert: yes | ||
azuresql_aad: | ||
<<: *azuresql | ||
authentication: ActiveDirectoryPassword | ||
username: "{{ env_var('DBT_AZURESQL_AAD_USER') }}" | ||
password: "{{ env_var('DBT_AZURESQL_AAD_PASS') }}" | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For my understanding, is this a way to extend de above mentioned profiles, the ones with
&<some name>
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that's right!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
they're called YAML anchors, and I'm on the fence about them legibility-wise. Seems pretty easy to go overboard on. how would you rate the readability of this YAML file, @JCZuurmond?
https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the anchors, indeed you should not use it too much, but it narrows what your focusing on within each segment. For example with
sqlserver_local_encrypt
it is clear thatencrypt: yes
andtrust_cert: yes
is what should be tested - it's like a fixture in testing