feat: Add AWS Athena support #383

svdimchenko · 2023-09-26T08:50:34Z

This is a:

new functionality

Description & motivation

Implement AWS Athena support. This PR depends on corresponding PR in dbt-athena-community repo, which implements required macroses.

Checklist

svdimchenko · 2023-10-19T07:16:39Z

@graciegoheen hi there! Could you please support what should be done extra to add dbt-athena support for this awesome package ?

macros/recursive_dag.sql

b-per · 2023-10-19T17:52:17Z

Also, would you be able to run the 2 integration-tests projects on an Athena connection and share a screenshot showing that they both pass? We most likely won't be able to add this to the automated CI we run on each PR.

Here is how those are run on CI. This is simply a dbt build run on each folder.

README.md

dave-connors-3 · 2023-12-12T15:10:09Z

hey @svdimchenko -- are you able to add the macro dispatch to this PR, and add show passing integration tests per @b-per 's last comment?

svdimchenko · 2023-12-12T16:04:25Z

hey @svdimchenko -- are you able to add the macro dispatch to this PR, and add show passing integration tests per @b-per 's last comment?

Hi @dave-connors-3 ! Sorry for some delay from my side. I'll try to finalise this PR during this week according to comments discussed.

svdimchenko · 2024-01-08T16:00:12Z

hey @svdimchenko -- are you able to add the macro dispatch to this PR, and add show passing integration tests per @b-per 's last comment?

here are results for integration_tests

❯ dbt clean
15:14:49  Running with dbt=1.7.4
15:14:49  Checking /Users/serhii.dimchenko/src/github/forks/dbt-project-evaluator/integration_tests/dbt_packages/*
15:14:49  Cleaned /Users/serhii.dimchenko/src/github/forks/dbt-project-evaluator/integration_tests/dbt_packages/*
15:14:49  Checking /Users/serhii.dimchenko/src/github/forks/dbt-project-evaluator/integration_tests/target/*
15:14:49  Cleaned /Users/serhii.dimchenko/src/github/forks/dbt-project-evaluator/integration_tests/target/*
15:14:49  Finished cleaning all paths.
❯ dbt deps
15:14:55  Running with dbt=1.7.4
15:14:56  Installing ../
15:14:56  Installed from <local @ ../>
15:14:56  Installing exclude_package/
15:14:56  Installed from <local @ exclude_package/>
15:14:56  Installing dbt-labs/dbt_utils
15:14:56  Installed from version 1.1.1
15:14:56  Up to date!
❯ dbt build --target athena --profiles-dir . --fail-fast
15:15:05  Running with dbt=1.7.4
15:15:06  Registered adapter: athena=1.7.1
15:15:06  Unable to do partial parsing because saved manifest not found. Starting full parse.
15:15:10  Found 62 models, 68 tests, 26 seeds, 8 sources, 1 exposure, 2 metrics, 578 macros, 0 groups, 1 semantic model
15:15:10  
15:15:14  Concurrency: 4 threads (target='athena')
15:15:14  
15:15:14  1 of 143 START sql table model sandbox_serhii_dimchenko.base_exposure_relationships  [RUN]
15:15:14  2 of 143 START sql table model sandbox_serhii_dimchenko.base_metric_relationships  [RUN]
15:15:14  3 of 143 START sql table model sandbox_serhii_dimchenko.base_node_relationships  [RUN]
15:15:14  4 of 143 START sql table model sandbox_serhii_dimchenko.stg_exposures .......... [RUN]
15:15:25  1 of 143 OK created sql table model sandbox_serhii_dimchenko.base_exposure_relationships  [OK 0 in 11.89s]
15:15:25  5 of 143 START sql table model sandbox_serhii_dimchenko.stg_metrics ............ [RUN]
15:15:26  4 of 143 OK created sql table model sandbox_serhii_dimchenko.stg_exposures ..... [OK 0 in 12.10s]
15:15:26  6 of 143 START sql view model sandbox_serhii_dimchenko.stg_naming_convention_folders  [RUN]
15:15:28  2 of 143 OK created sql table model sandbox_serhii_dimchenko.base_metric_relationships  [OK 0 in 14.12s]
15:15:28  7 of 143 START sql view model sandbox_serhii_dimchenko.stg_naming_convention_prefixes  [RUN]
15:15:30  6 of 143 OK created sql view model sandbox_serhii_dimchenko.stg_naming_convention_folders  [OK -1 in 3.88s]
15:15:30  8 of 143 START sql table model sandbox_serhii_dimchenko.stg_nodes .............. [RUN]
15:15:31  7 of 143 OK created sql view model sandbox_serhii_dimchenko.stg_naming_convention_prefixes  [OK -1 in 3.76s]
15:15:31  9 of 143 START sql table model sandbox_serhii_dimchenko.stg_sources ............ [RUN]
15:15:32  3 of 143 OK created sql table model sandbox_serhii_dimchenko.base_node_relationships  [OK 0 in 18.11s]
15:15:32  10 of 143 START sql table model sandbox_serhii_dimchenko.metricflow_time_spine . [RUN]
15:15:37  5 of 143 OK created sql table model sandbox_serhii_dimchenko.stg_metrics ....... [OK 0 in 11.48s]
15:15:37  11 of 143 START sql view model sandbox_serhii_dimchenko.excluded_model ......... [RUN]
15:15:43  9 of 143 OK created sql table model sandbox_serhii_dimchenko.stg_sources ....... [OK 0 in 11.33s]
15:15:43  12 of 143 START seed file sandbox_serhii_dimchenko.dbt_project_evaluator_exceptions  [RUN]
15:15:43  10 of 143 OK created sql table model sandbox_serhii_dimchenko.metricflow_time_spine  [OK 1 in 11.33s]
15:15:43  13 of 143 START seed file sandbox_serhii_dimchenko.test_fct_chained_views_dependencies  [RUN]
15:15:45  11 of 143 OK created sql view model sandbox_serhii_dimchenko.excluded_model .... [OK -1 in 7.53s]
15:15:45  14 of 143 START seed file sandbox_serhii_dimchenko.test_fct_direct_join_to_source  [RUN]
15:15:47  8 of 143 OK created sql table model sandbox_serhii_dimchenko.stg_nodes ......... [OK 0 in 17.04s]
15:15:47  15 of 143 START seed file sandbox_serhii_dimchenko.test_fct_documentation_coverage  [RUN]
15:15:59  12 of 143 OK loaded seed file sandbox_serhii_dimchenko.dbt_project_evaluator_exceptions  [INSERT 1 in 15.77s]
15:15:59  16 of 143 START seed file sandbox_serhii_dimchenko.test_fct_duplicate_sources .. [RUN]
15:15:59  13 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_chained_views_dependencies  [INSERT 13 in 15.93s]
15:15:59  17 of 143 START seed file sandbox_serhii_dimchenko.test_fct_exposure_parents_materializations  [RUN]
15:16:00  14 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_direct_join_to_source  [INSERT 1 in 15.81s]
15:16:00  18 of 143 START seed file sandbox_serhii_dimchenko.test_fct_exposures_dependent_on_private_models  [RUN]
15:16:03  15 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_documentation_coverage  [INSERT 1 in 16.58s]
15:16:03  19 of 143 START seed file sandbox_serhii_dimchenko.test_fct_hard_coded_references  [RUN]
15:16:15  16 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_duplicate_sources  [INSERT 1 in 15.97s]
15:16:15  20 of 143 START seed file sandbox_serhii_dimchenko.test_fct_missing_primary_key_tests  [RUN]
15:16:15  17 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_exposure_parents_materializations  [INSERT 2 in 15.72s]
15:16:15  21 of 143 START seed file sandbox_serhii_dimchenko.test_fct_model_directories .. [RUN]
15:16:16  18 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_exposures_dependent_on_private_models  [INSERT 4 in 15.71s]
15:16:16  22 of 143 START seed file sandbox_serhii_dimchenko.test_fct_model_fanout ....... [RUN]
15:16:18  19 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_hard_coded_references  [INSERT 1 in 15.25s]
15:16:18  23 of 143 START seed file sandbox_serhii_dimchenko.test_fct_model_naming_conventions  [RUN]
15:16:30  20 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_missing_primary_key_tests  [INSERT 10 in 15.50s]
15:16:30  24 of 143 START seed file sandbox_serhii_dimchenko.test_fct_multiple_sources_joined  [RUN]
15:16:31  21 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_model_directories  [INSERT 6 in 16.54s]
15:16:31  25 of 143 START seed file sandbox_serhii_dimchenko.test_fct_public_models_without_contract  [RUN]
15:16:32  22 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_model_fanout ... [INSERT 1 in 15.67s]
15:16:32  26 of 143 START seed file sandbox_serhii_dimchenko.test_fct_rejoining_of_upstream_concepts  [RUN]
15:16:34  23 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_model_naming_conventions  [INSERT 4 in 15.54s]
15:16:34  27 of 143 START seed file sandbox_serhii_dimchenko.test_fct_root_models ........ [RUN]
15:16:46  24 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_multiple_sources_joined  [INSERT 1 in 15.53s]
15:16:46  28 of 143 START seed file sandbox_serhii_dimchenko.test_fct_source_directories . [RUN]
15:16:47  25 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_public_models_without_contract  [INSERT 1 in 15.45s]
15:16:47  29 of 143 START seed file sandbox_serhii_dimchenko.test_fct_source_fanout ...... [RUN]
15:16:48  26 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_rejoining_of_upstream_concepts  [INSERT 1 in 15.75s]
15:16:48  30 of 143 START seed file sandbox_serhii_dimchenko.test_fct_staging_dependent_on_staging  [RUN]
15:16:49  27 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_root_models .... [INSERT 1 in 15.35s]
15:16:49  31 of 143 START seed file sandbox_serhii_dimchenko.test_fct_test_coverage ...... [RUN]
15:17:02  28 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_source_directories  [INSERT 1 in 16.00s]
15:17:02  32 of 143 START seed file sandbox_serhii_dimchenko.test_fct_test_directories ... [RUN]
15:17:02  29 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_source_fanout .. [INSERT 2 in 15.45s]
15:17:02  33 of 143 START seed file sandbox_serhii_dimchenko.test_fct_undocumented_models  [RUN]
15:17:03  30 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_staging_dependent_on_staging  [INSERT 1 in 15.20s]
15:17:03  34 of 143 START seed file sandbox_serhii_dimchenko.test_fct_undocumented_public_models  [RUN]
15:17:05  31 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_test_coverage .. [INSERT 1 in 15.56s]
15:17:05  35 of 143 START seed file sandbox_serhii_dimchenko.test_fct_undocumented_source_tables  [RUN]
15:17:18  32 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_test_directories  [INSERT 2 in 16.25s]
15:17:18  36 of 143 START seed file sandbox_serhii_dimchenko.test_fct_undocumented_sources  [RUN]
15:17:18  33 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_undocumented_models  [INSERT 11 in 16.10s]
15:17:18  37 of 143 START seed file sandbox_serhii_dimchenko.test_fct_unused_sources ..... [RUN]
15:17:19  34 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_undocumented_public_models  [INSERT 1 in 16.35s]
15:17:19  38 of 143 START sql view model sandbox_serhii_dimchenko.stg_exposure_relationships  [RUN]
15:17:21  35 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_undocumented_source_tables  [INSERT 5 in 16.26s]
15:17:21  39 of 143 START test not_null_stg_exposures_unique_id .......................... [RUN]
15:17:23  38 of 143 OK created sql view model sandbox_serhii_dimchenko.stg_exposure_relationships  [OK -1 in 3.90s]
15:17:23  40 of 143 START test unique_stg_exposures_unique_id ............................ [RUN]
15:17:24  39 of 143 PASS not_null_stg_exposures_unique_id ................................ [PASS in 3.00s]
15:17:24  41 of 143 START sql view model sandbox_serhii_dimchenko.stg_metric_relationships  [RUN]
15:17:26  40 of 143 PASS unique_stg_exposures_unique_id .................................. [PASS in 2.89s]
15:17:26  42 of 143 START test accepted_values_stg_naming_convention_folders_model_type___var_model_types_  [RUN]
15:17:28  41 of 143 OK created sql view model sandbox_serhii_dimchenko.stg_metric_relationships  [OK -1 in 3.79s]
15:17:28  43 of 143 START test not_null_stg_naming_convention_folders_folder_name ........ [RUN]
15:17:29  42 of 143 PASS accepted_values_stg_naming_convention_folders_model_type___var_model_types_  [PASS in 2.78s]
15:17:29  44 of 143 START test unique_stg_naming_convention_folders_folder_name .......... [RUN]
15:17:31  43 of 143 PASS not_null_stg_naming_convention_folders_folder_name .............. [PASS in 2.97s]
15:17:31  45 of 143 START test accepted_values_stg_naming_convention_prefixes_model_type___var_model_types_  [RUN]
15:17:32  44 of 143 PASS unique_stg_naming_convention_folders_folder_name ................ [PASS in 2.87s]
15:17:32  46 of 143 START test not_null_stg_naming_convention_prefixes_unique_id ......... [RUN]
15:17:34  45 of 143 PASS accepted_values_stg_naming_convention_prefixes_model_type___var_model_types_  [PASS in 2.94s]
15:17:34  47 of 143 START test unique_stg_naming_convention_prefixes_unique_id ........... [RUN]
15:17:34  46 of 143 PASS not_null_stg_naming_convention_prefixes_unique_id ............... [PASS in 2.76s]
15:17:34  48 of 143 START sql table model sandbox_serhii_dimchenko.stg_node_relationships  [RUN]
15:17:34  37 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_unused_sources . [INSERT 2 in 16.05s]
15:17:34  36 of 143 OK loaded seed file sandbox_serhii_dimchenko.test_fct_undocumented_sources  [INSERT 1 in 16.50s]
15:17:34  49 of 143 START test not_null_stg_metrics_unique_id ............................ [RUN]
15:17:34  50 of 143 START test unique_stg_metrics_unique_id .............................. [RUN]
15:17:37  47 of 143 PASS unique_stg_naming_convention_prefixes_unique_id ................. [PASS in 2.86s]
15:17:37  51 of 143 START test sample_custom_test ........................................ [RUN]
15:17:37  50 of 143 PASS unique_stg_metrics_unique_id .................................... [PASS in 3.11s]
15:17:37  52 of 143 START test unique_stg_model_1_id ..................................... [RUN]
15:17:38  49 of 143 PASS not_null_stg_metrics_unique_id .................................. [PASS in 3.23s]
15:17:38  53 of 143 START test not_null_stg_model_2_id ................................... [RUN]
15:17:40  51 of 143 PASS sample_custom_test .............................................. [PASS in 2.92s]
15:17:40  54 of 143 START test unique_stg_model_2_id ..................................... [RUN]
15:17:40  52 of 143 PASS unique_stg_model_1_id ........................................... [PASS in 2.82s]
15:17:40  55 of 143 START test dbt_utils_unique_combination_of_columns_stg_model_3_id__color  [RUN]
15:17:40  53 of 143 PASS not_null_stg_model_2_id ......................................... [PASS in 2.78s]
15:17:40  56 of 143 START test not_null_stg_sources_unique_id ............................ [RUN]
15:17:42  54 of 143 PASS unique_stg_model_2_id ........................................... [PASS in 2.77s]
15:17:42  57 of 143 START test unique_stg_sources_unique_id .............................. [RUN]
15:17:43  55 of 143 PASS dbt_utils_unique_combination_of_columns_stg_model_3_id__color ... [PASS in 2.93s]
15:17:43  58 of 143 START test not_null_metricflow_time_spine_date_day ................... [RUN]
15:17:43  56 of 143 PASS not_null_stg_sources_unique_id .................................. [PASS in 2.91s]
15:17:43  59 of 143 START test unique_metricflow_time_spine_date_day ..................... [RUN]
15:17:45  48 of 143 OK created sql table model sandbox_serhii_dimchenko.stg_node_relationships  [OK 224 in 11.09s]
15:17:45  60 of 143 START test not_null_stg_nodes_unique_id .............................. [RUN]
15:17:45  57 of 143 PASS unique_stg_sources_unique_id .................................... [PASS in 2.91s]
15:17:45  61 of 143 START test unique_stg_nodes_unique_id ................................ [RUN]
15:17:46  59 of 143 PASS unique_metricflow_time_spine_date_day ........................... [PASS in 2.94s]
15:17:46  62 of 143 START test not_null_stg_exposure_relationships_unique_id ............. [RUN]
15:17:46  58 of 143 PASS not_null_metricflow_time_spine_date_day ......................... [PASS in 3.02s]
15:17:46  63 of 143 START test unique_stg_exposure_relationships_unique_id ............... [RUN]
15:17:48  61 of 143 PASS unique_stg_nodes_unique_id ...................................... [PASS in 2.93s]
15:17:48  64 of 143 START test not_null_stg_metric_relationships_unique_id ............... [RUN]
15:17:48  60 of 143 PASS not_null_stg_nodes_unique_id .................................... [PASS in 3.09s]
15:17:48  65 of 143 START test unique_stg_metric_relationships_unique_id ................. [RUN]
15:17:49  63 of 143 PASS unique_stg_exposure_relationships_unique_id ..................... [PASS in 2.75s]
15:17:49  66 of 143 START sql table model sandbox_serhii_dimchenko.fct_model_6 ........... [RUN]
15:17:49  62 of 143 PASS not_null_stg_exposure_relationships_unique_id ................... [PASS in 2.82s]
15:17:49  67 of 143 START test not_null_stg_node_relationships_unique_id ................. [RUN]
15:17:51  64 of 143 PASS not_null_stg_metric_relationships_unique_id ..................... [PASS in 2.80s]
15:17:51  68 of 143 START test unique_stg_node_relationships_unique_id ................... [RUN]
15:17:51  65 of 143 PASS unique_stg_metric_relationships_unique_id ....................... [PASS in 2.83s]
15:17:51  69 of 143 START sql table model sandbox_serhii_dimchenko.int_all_graph_resources  [RUN]
15:17:52  67 of 143 PASS not_null_stg_node_relationships_unique_id ....................... [PASS in 2.97s]
15:17:52  70 of 143 START test not_null_int_model_4_id ................................... [RUN]
15:17:54  68 of 143 PASS unique_stg_node_relationships_unique_id ......................... [PASS in 2.87s]
15:17:54  71 of 143 START test unique_int_model_4_id ..................................... [RUN]
15:17:55  70 of 143 PASS not_null_int_model_4_id ......................................... [PASS in 2.77s]
15:17:55  72 of 143 START test not_null_stg_model_4_concat_coalesce_fake_column_for_testing_  [RUN]
15:17:57  71 of 143 PASS unique_int_model_4_id ........................................... [PASS in 2.69s]
15:17:57  73 of 143 START test not_null_stg_model_4_id ................................... [RUN]
15:17:58  72 of 143 PASS not_null_stg_model_4_concat_coalesce_fake_column_for_testing_ ... [PASS in 2.74s]
15:17:58  74 of 143 START test relationships_stg_model_4_id__id__ref_stg_model_2_ ........ [RUN]
15:17:59  66 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_model_6 ...... [OK 1 in 10.27s]
15:17:59  75 of 143 START test unique_stg_model_4_id ..................................... [RUN]
15:18:00  73 of 143 PASS not_null_stg_model_4_id ......................................... [PASS in 2.85s]
15:18:00  76 of 143 START sql table model sandbox_serhii_dimchenko.report_1 .............. [RUN]
15:18:01  74 of 143 PASS relationships_stg_model_4_id__id__ref_stg_model_2_ .............. [PASS in 2.93s]
15:18:01  77 of 143 START sql table model sandbox_serhii_dimchenko.report_2 .............. [RUN]
15:18:02  75 of 143 PASS unique_stg_model_4_id ........................................... [PASS in 2.82s]
15:18:02  78 of 143 START sql table model sandbox_serhii_dimchenko.dim_model_7 ........... [RUN]
15:18:08  69 of 143 OK created sql table model sandbox_serhii_dimchenko.int_all_graph_resources  [OK 124 in 16.28s]
15:18:08  79 of 143 START test dbt_utils_expression_is_true_int_all_graph_resources_model_type___base_  [RUN]
15:18:09  76 of 143 OK created sql table model sandbox_serhii_dimchenko.report_1 ......... [OK 1 in 9.69s]
15:18:09  80 of 143 START test is_not_empty_string_int_all_graph_resources_on_schema_change  [RUN]
15:18:10  79 of 143 PASS dbt_utils_expression_is_true_int_all_graph_resources_model_type___base_  [PASS in 2.72s]
15:18:10  81 of 143 START test not_null_int_all_graph_resources_resource_id .............. [RUN]
15:18:12  78 of 143 OK created sql table model sandbox_serhii_dimchenko.dim_model_7 ...... [OK 1 in 9.51s]
15:18:12  82 of 143 START test unique_int_all_graph_resources_resource_id ................ [RUN]
15:18:12  80 of 143 PASS is_not_empty_string_int_all_graph_resources_on_schema_change .... [PASS in 2.86s]
15:18:13  81 of 143 PASS not_null_int_all_graph_resources_resource_id .................... [PASS in 2.99s]
15:18:14  77 of 143 OK created sql table model sandbox_serhii_dimchenko.report_2 ......... [OK 1 in 13.81s]
15:18:14  82 of 143 PASS unique_int_all_graph_resources_resource_id ...................... [PASS in 2.72s]
15:18:14  83 of 143 START sql table model sandbox_serhii_dimchenko.fct_documentation_coverage  [RUN]
15:18:14  84 of 143 START sql table model sandbox_serhii_dimchenko.fct_duplicate_sources . [RUN]
15:18:14  85 of 143 START sql table model sandbox_serhii_dimchenko.fct_hard_coded_references  [RUN]
15:18:14  86 of 143 START sql view model sandbox_serhii_dimchenko.fct_model_naming_conventions  [RUN]
15:18:20  86 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_model_naming_conventions  [OK -1 in 5.73s]
15:18:20  87 of 143 START sql view model sandbox_serhii_dimchenko.fct_public_models_without_contract  [RUN]
15:18:25  84 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_duplicate_sources  [OK 1 in 10.93s]
15:18:25  88 of 143 START sql view model sandbox_serhii_dimchenko.fct_source_directories . [RUN]
15:18:26  87 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_public_models_without_contract  [OK -1 in 5.80s]
15:18:26  89 of 143 START sql view model sandbox_serhii_dimchenko.fct_undocumented_models  [RUN]
15:18:26  83 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_documentation_coverage  [OK 1 in 11.72s]
15:18:26  90 of 143 START sql view model sandbox_serhii_dimchenko.fct_undocumented_public_models  [RUN]
15:18:27  85 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_hard_coded_references  [OK 1 in 12.53s]
15:18:27  91 of 143 START sql view model sandbox_serhii_dimchenko.fct_undocumented_source_tables  [RUN]
15:18:31  88 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_source_directories  [OK -1 in 5.69s]
15:18:31  92 of 143 START sql view model sandbox_serhii_dimchenko.fct_undocumented_sources  [RUN]
15:18:31  89 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_models  [OK -1 in 5.49s]
15:18:31  93 of 143 START sql view model sandbox_serhii_dimchenko.int_direct_relationships  [RUN]
15:18:31  90 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_public_models  [OK -1 in 5.36s]
15:18:31  94 of 143 START test equality_fct_model_naming_conventions ..................... [RUN]
15:18:32  91 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_source_tables  [OK -1 in 5.50s]
15:18:32  95 of 143 START test equality_fct_duplicate_sources ............................ [RUN]
15:18:35  95 of 143 PASS equality_fct_duplicate_sources .................................. [PASS in 2.80s]
15:18:35  96 of 143 START test dbt_utils_equality_test_fct_public_models_without_contract_ref_fct_public_models_without_contract_  [RUN]
15:18:35  93 of 143 OK created sql view model sandbox_serhii_dimchenko.int_direct_relationships  [OK -1 in 3.93s]
15:18:35  97 of 143 START test equality_fct_documentation_coverage ....................... [RUN]
15:18:36  94 of 143 PASS equality_fct_model_naming_conventions ........................... [PASS in 4.04s]
15:18:36  98 of 143 START test equality_fct_hard_coded_references ........................ [RUN]
15:18:36  92 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_sources  [OK -1 in 5.44s]
15:18:36  99 of 143 START test equality_fct_source_directories ........................... [RUN]
15:18:38  97 of 143 PASS equality_fct_documentation_coverage ............................. [PASS in 3.05s]
15:18:38  100 of 143 START test equality_fct_undocumented_models ......................... [RUN]
15:18:39  98 of 143 PASS equality_fct_hard_coded_references .............................. [PASS in 3.00s]
15:18:39  101 of 143 START test dbt_utils_equality_test_fct_undocumented_public_models_ref_fct_undocumented_public_models_  [RUN]
15:18:40  96 of 143 PASS dbt_utils_equality_test_fct_public_models_without_contract_ref_fct_public_models_without_contract_  [PASS in 4.94s]
15:18:40  102 of 143 START test equality_fct_undocumented_source_tables .................. [RUN]
15:18:40  99 of 143 PASS equality_fct_source_directories ................................. [PASS in 3.86s]
15:18:40  103 of 143 START test not_null_int_direct_relationships_unique_id .............. [RUN]
15:18:43  100 of 143 PASS equality_fct_undocumented_models ............................... [PASS in 4.85s]
15:18:43  104 of 143 START test unique_int_direct_relationships_unique_id ................ [RUN]
15:18:44  101 of 143 PASS dbt_utils_equality_test_fct_undocumented_public_models_ref_fct_undocumented_public_models_  [PASS in 5.08s]
15:18:44  105 of 143 START test equality_fct_undocumented_sources ........................ [RUN]
15:18:44  103 of 143 PASS not_null_int_direct_relationships_unique_id .................... [PASS in 3.81s]
15:18:45  102 of 143 PASS equality_fct_undocumented_source_tables ........................ [PASS in 4.72s]
15:18:48  104 of 143 PASS unique_int_direct_relationships_unique_id ...................... [PASS in 5.17s]
15:18:48  106 of 143 START sql view model sandbox_serhii_dimchenko.fct_test_directories .. [RUN]
15:18:48  107 of 143 START sql view model sandbox_serhii_dimchenko.int_all_dag_relationships  [RUN]
15:18:48  108 of 143 START sql view model sandbox_serhii_dimchenko.int_model_test_summary  [RUN]
15:18:49  105 of 143 PASS equality_fct_undocumented_sources .............................. [PASS in 4.93s]
15:18:53  108 of 143 OK created sql view model sandbox_serhii_dimchenko.int_model_test_summary  [OK -1 in 4.08s]
15:18:53  109 of 143 START test not_null_int_model_test_summary_resource_name ............ [RUN]
15:18:53  110 of 143 START test unique_int_model_test_summary_resource_name .............. [RUN]
15:18:54  107 of 143 OK created sql view model sandbox_serhii_dimchenko.int_all_dag_relationships  [OK -1 in 5.13s]
15:18:54  111 of 143 START test not_null_int_all_dag_relationships_path .................. [RUN]
15:18:54  106 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_test_directories  [OK -1 in 5.76s]
15:18:54  112 of 143 START test unique_int_all_dag_relationships_path .................... [RUN]
15:18:58  109 of 143 PASS not_null_int_model_test_summary_resource_name .................. [PASS in 5.29s]
15:18:58  113 of 143 START test equality_fct_test_directories ............................ [RUN]
15:18:58  110 of 143 PASS unique_int_model_test_summary_resource_name .................... [PASS in 5.33s]
15:18:58  114 of 143 START sql view model sandbox_serhii_dimchenko.fct_missing_primary_key_tests  [RUN]
15:19:00  112 of 143 PASS unique_int_all_dag_relationships_path .......................... [PASS in 5.98s]
15:19:00  115 of 143 START sql table model sandbox_serhii_dimchenko.fct_test_coverage .... [RUN]
15:19:01  111 of 143 PASS not_null_int_all_dag_relationships_path ........................ [PASS in 7.18s]
15:19:01  116 of 143 START sql view model sandbox_serhii_dimchenko.fct_chained_views_dependencies  [RUN]
15:19:04  114 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_missing_primary_key_tests  [OK -1 in 6.47s]
15:19:04  117 of 143 START sql table model sandbox_serhii_dimchenko.fct_direct_join_to_source  [RUN]
15:19:05  113 of 143 PASS equality_fct_test_directories .................................. [PASS in 7.18s]
15:19:05  118 of 143 START sql view model sandbox_serhii_dimchenko.fct_exposure_parents_materializations  [RUN]
15:19:07  116 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_chained_views_dependencies  [OK -1 in 6.35s]
15:19:07  119 of 143 START sql view model sandbox_serhii_dimchenko.fct_exposures_dependent_on_private_models  [RUN]
15:19:10  118 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_exposure_parents_materializations  [OK -1 in 5.40s]
15:19:10  120 of 143 START sql table model sandbox_serhii_dimchenko.fct_marts_or_intermediate_dependent_on_source  [RUN]
15:19:13  115 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_test_coverage  [OK 1 in 13.06s]
15:19:13  121 of 143 START sql view model sandbox_serhii_dimchenko.fct_model_directories . [RUN]
15:19:14  119 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_exposures_dependent_on_private_models  [OK -1 in 6.50s]
15:19:14  122 of 143 START sql table model sandbox_serhii_dimchenko.fct_model_fanout ..... [RUN]
15:19:17  117 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_direct_join_to_source  [OK 1 in 13.05s]
15:19:17  123 of 143 START sql table model sandbox_serhii_dimchenko.fct_multiple_sources_joined  [RUN]
15:19:20  121 of 143 OK created sql view model sandbox_serhii_dimchenko.fct_model_directories  [OK -1 in 6.35s]
15:19:20  124 of 143 START sql table model sandbox_serhii_dimchenko.fct_rejoining_of_upstream_concepts  [RUN]
15:19:23  120 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_marts_or_intermediate_dependent_on_source  [OK 2 in 12.22s]
15:19:23  125 of 143 START sql table model sandbox_serhii_dimchenko.fct_root_models ...... [RUN]
15:19:26  122 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_model_fanout  [OK 1 in 12.16s]
15:19:26  126 of 143 START sql table model sandbox_serhii_dimchenko.fct_source_fanout .... [RUN]
15:19:29  123 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_multiple_sources_joined  [OK 1 in 11.16s]
15:19:29  127 of 143 START sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_marts_or_intermediate  [RUN]
15:19:35  125 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_root_models . [OK 1 in 12.48s]
15:19:35  128 of 143 START sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_staging  [RUN]
15:19:37  124 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_rejoining_of_upstream_concepts  [OK 1 in 17.52s]
15:19:37  129 of 143 START sql table model sandbox_serhii_dimchenko.fct_unused_sources ... [RUN]
15:19:38  126 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_source_fanout  [OK 2 in 12.43s]
15:19:38  130 of 143 START test equality_fct_missing_primary_key_tests ................... [RUN]
15:19:40  127 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_marts_or_intermediate  [OK 1 in 11.00s]
15:19:40  131 of 143 START test equality_fct_chained_views_dependencies .................. [RUN]
15:19:44  130 of 143 PASS equality_fct_missing_primary_key_tests ......................... [PASS in 5.85s]
15:19:44  132 of 143 START test dbt_utils_equality_test_fct_exposure_parents_materializations_ref_fct_exposure_parents_materializations_  [RUN]
15:19:46  128 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_staging  [OK 1 in 11.22s]
15:19:46  133 of 143 START test equality_fct_test_coverage ............................... [RUN]
15:19:49  131 of 143 PASS equality_fct_chained_views_dependencies ........................ [PASS in 9.28s]
15:19:49  134 of 143 START test dbt_utils_equality_test_fct_exposures_dependent_on_private_models_ref_fct_exposures_dependent_on_private_models_  [RUN]
15:19:49  129 of 143 OK created sql table model sandbox_serhii_dimchenko.fct_unused_sources  [OK 2 in 12.19s]
15:19:49  135 of 143 START test equality_fct_direct_join_to_source ....................... [RUN]
15:19:50  133 of 143 PASS equality_fct_test_coverage ..................................... [PASS in 3.78s]
15:19:50  136 of 143 START test equality_fct_model_directories ........................... [RUN]
15:19:52  132 of 143 PASS dbt_utils_equality_test_fct_exposure_parents_materializations_ref_fct_exposure_parents_materializations_  [PASS in 7.97s]
15:19:52  137 of 143 START test equality_fct_model_fanout ................................ [RUN]
15:19:53  135 of 143 PASS equality_fct_direct_join_to_source ............................. [PASS in 3.89s]
15:19:53  138 of 143 START test equality_fct_multiple_sources_joined ..................... [RUN]
15:19:55  137 of 143 PASS equality_fct_model_fanout ...................................... [PASS in 2.69s]
15:19:55  139 of 143 START test equality_fct_root_models ................................. [RUN]
15:19:57  138 of 143 PASS equality_fct_multiple_sources_joined ........................... [PASS in 3.77s]
15:19:57  140 of 143 START test equality_fct_rejoining_of_upstream_concepts .............. [RUN]
15:19:58  134 of 143 PASS dbt_utils_equality_test_fct_exposures_dependent_on_private_models_ref_fct_exposures_dependent_on_private_models_  [PASS in 9.19s]
15:19:58  141 of 143 START test equality_fct_source_fanout ............................... [RUN]
15:19:59  139 of 143 PASS equality_fct_root_models ....................................... [PASS in 3.81s]
15:19:59  142 of 143 START test equality_fct_staging_dependent_on_staging ................ [RUN]
15:20:00  136 of 143 PASS equality_fct_model_directories ................................. [PASS in 9.51s]
15:20:00  143 of 143 START test equality_fct_unused_sources .............................. [RUN]
15:20:00  140 of 143 PASS equality_fct_rejoining_of_upstream_concepts .................... [PASS in 3.03s]
15:20:01  142 of 143 PASS equality_fct_staging_dependent_on_staging ...................... [PASS in 2.82s]
15:20:02  141 of 143 PASS equality_fct_source_fanout ..................................... [PASS in 3.92s]
15:20:03  143 of 143 PASS equality_fct_unused_sources .................................... [PASS in 3.81s]
15:20:03  
15:20:03  Finished running 28 table models, 21 view models, 26 seeds, 68 tests in 0 hours 4 minutes and 53.89 seconds (293.89s).
15:20:04  
15:20:04  Completed successfully
15:20:04  
15:20:04  Done. PASS=143 WARN=0 ERROR=0 SKIP=0 TOTAL=143

svdimchenko · 2024-01-08T16:00:47Z

hey @svdimchenko -- are you able to add the macro dispatch to this PR, and add show passing integration tests per @b-per 's last comment?

and here are results for integrations_tests_2

❯ dbt deps
15:26:06  Running with dbt=1.7.4
15:26:06  Updating lock file in file path: /Users/serhii.dimchenko/src/github/forks/dbt-project-evaluator/integration_tests_2/package-lock.yml
15:26:06  Installing ../
15:26:06  Installed from <local @ ../>
15:26:06  Installing dbt-labs/dbt_utils
15:26:07  Installed from version 1.1.1
15:26:07  Up to date!
❯ dbt build --target athena --profiles-dir . --fail-fast
15:27:36  Running with dbt=1.7.4
15:27:36  Registered adapter: athena=1.7.1
15:27:36  Unable to do partial parsing because saved manifest not found. Starting full parse.

### List of issues raised by dbt_project_evaluator ###


15:27:39  Found 57 models, 1 operation, 1 seed, 27 tests, 5 sources, 0 exposures, 0 metrics, 576 macros, 0 groups, 0 semantic models
15:27:39  
15:27:43  Concurrency: 4 threads (target='athena')
15:27:43  
15:27:43  1 of 72 START sql table model sandbox_serhii_dimchenko.base_exposure_relationships  [RUN]
15:27:43  2 of 72 START sql table model sandbox_serhii_dimchenko.base_metric_relationships  [RUN]
15:27:43  3 of 72 START sql table model sandbox_serhii_dimchenko.base_node_relationships . [RUN]
15:27:43  4 of 72 START sql table model sandbox_serhii_dimchenko.stg_exposures ........... [RUN]
15:27:53  2 of 72 OK created sql table model sandbox_serhii_dimchenko.base_metric_relationships  [OK 0 in 10.42s]
15:27:53  1 of 72 OK created sql table model sandbox_serhii_dimchenko.base_exposure_relationships  [OK 0 in 10.42s]
15:27:53  5 of 72 START sql table model sandbox_serhii_dimchenko.stg_metrics ............. [RUN]
15:27:53  6 of 72 START sql view model sandbox_serhii_dimchenko.stg_naming_convention_folders  [RUN]
15:27:54  4 of 72 OK created sql table model sandbox_serhii_dimchenko.stg_exposures ...... [OK 0 in 10.64s]
15:27:54  7 of 72 START sql view model sandbox_serhii_dimchenko.stg_naming_convention_prefixes  [RUN]
15:27:57  3 of 72 OK created sql table model sandbox_serhii_dimchenko.base_node_relationships  [OK 0 in 13.89s]
15:27:57  8 of 72 START sql table model sandbox_serhii_dimchenko.stg_nodes ............... [RUN]
15:27:59  6 of 72 OK created sql view model sandbox_serhii_dimchenko.stg_naming_convention_folders  [OK -1 in 5.21s]
15:27:59  9 of 72 START sql table model sandbox_serhii_dimchenko.stg_sources ............. [RUN]
15:27:59  7 of 72 OK created sql view model sandbox_serhii_dimchenko.stg_naming_convention_prefixes  [OK -1 in 5.44s]
15:27:59  10 of 72 START seed file sandbox_serhii_dimchenko.dbt_project_evaluator_exceptions  [RUN]
15:28:04  5 of 72 OK created sql table model sandbox_serhii_dimchenko.stg_metrics ........ [OK 0 in 10.92s]
15:28:04  11 of 72 START sql view model sandbox_serhii_dimchenko.stg_metric_relationships  [RUN]
15:28:09  11 of 72 OK created sql view model sandbox_serhii_dimchenko.stg_metric_relationships  [OK -1 in 4.18s]
15:28:09  12 of 72 START sql view model sandbox_serhii_dimchenko.stg_exposure_relationships  [RUN]
15:28:11  8 of 72 OK created sql table model sandbox_serhii_dimchenko.stg_nodes .......... [OK 0 in 13.78s]
15:28:11  13 of 72 START sql table model sandbox_serhii_dimchenko.stg_node_relationships . [RUN]
15:28:12  9 of 72 OK created sql table model sandbox_serhii_dimchenko.stg_sources ........ [OK 0 in 13.03s]
15:28:12  14 of 72 START sql table model sandbox_serhii_dimchenko.fct_model_6 ............ [RUN]
15:28:12  12 of 72 OK created sql view model sandbox_serhii_dimchenko.stg_exposure_relationships  [OK -1 in 3.82s]
15:28:12  15 of 72 START sql table model sandbox_serhii_dimchenko.int_all_graph_resources  [RUN]
15:28:14  10 of 72 OK loaded seed file sandbox_serhii_dimchenko.dbt_project_evaluator_exceptions  [INSERT 0 in 15.02s]
15:28:21  13 of 72 OK created sql table model sandbox_serhii_dimchenko.stg_node_relationships  [OK 128 in 10.58s]
15:28:21  14 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_model_6 ....... [OK 1 in 9.44s]
15:28:26  15 of 72 OK created sql table model sandbox_serhii_dimchenko.int_all_graph_resources  [OK 20 in 14.08s]
15:28:26  16 of 72 START sql view model sandbox_serhii_dimchenko.fct_documentation_coverage  [RUN]
15:28:26  17 of 72 START sql table model sandbox_serhii_dimchenko.fct_duplicate_sources .. [RUN]
15:28:26  18 of 72 START sql table model sandbox_serhii_dimchenko.fct_hard_coded_references  [RUN]
15:28:26  19 of 72 START sql view model sandbox_serhii_dimchenko.fct_model_naming_conventions  [RUN]
15:28:35  19 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_model_naming_conventions  [OK -1 in 8.17s]
15:28:35  20 of 72 START sql view model sandbox_serhii_dimchenko.fct_public_models_without_contract  [RUN]
15:28:40  20 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_public_models_without_contract  [OK -1 in 4.90s]
15:28:40  21 of 72 START sql view model sandbox_serhii_dimchenko.fct_source_directories .. [RUN]
15:28:40  16 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_documentation_coverage  [OK -1 in 13.96s]
15:28:40  22 of 72 START sql view model sandbox_serhii_dimchenko.fct_undocumented_models . [RUN]
15:28:43  17 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_duplicate_sources  [OK 0 in 16.92s]
15:28:43  18 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_hard_coded_references  [OK 0 in 16.92s]
15:28:43  23 of 72 START sql view model sandbox_serhii_dimchenko.fct_undocumented_public_models  [RUN]
15:28:43  24 of 72 START sql view model sandbox_serhii_dimchenko.fct_undocumented_source_tables  [RUN]
15:28:44  21 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_source_directories  [OK -1 in 4.34s]
15:28:44  25 of 72 START sql view model sandbox_serhii_dimchenko.fct_undocumented_sources  [RUN]
15:28:45  22 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_models  [OK -1 in 4.51s]
15:28:45  26 of 72 START sql view model sandbox_serhii_dimchenko.int_direct_relationships  [RUN]
15:28:48  23 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_public_models  [OK -1 in 4.32s]
15:28:48  27 of 72 START test is_empty_fct_model_naming_conventions_ ..................... [RUN]
15:28:48  24 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_source_tables  [OK -1 in 4.34s]
15:28:48  28 of 72 START test is_empty_fct_public_models_without_contract_ ............... [RUN]
15:28:49  25 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_undocumented_sources  [OK -1 in 5.01s]
15:28:49  29 of 72 START test valid_documentation_coverage ............................... [RUN]
15:28:49  26 of 72 OK created sql view model sandbox_serhii_dimchenko.int_direct_relationships  [OK -1 in 4.38s]
15:28:49  30 of 72 START test is_empty_fct_duplicate_sources_ ............................ [RUN]
15:28:51  28 of 72 PASS is_empty_fct_public_models_without_contract_ ..................... [PASS in 3.03s]
15:28:51  31 of 72 START test is_empty_fct_hard_coded_references_ ........................ [RUN]
15:28:52  29 of 72 WARN 1 valid_documentation_coverage ................................... [WARN 1 in 2.84s]
15:28:52  32 of 72 START test is_empty_fct_source_directories_ ........................... [RUN]
15:28:52  27 of 72 WARN 4 is_empty_fct_model_naming_conventions_ ......................... [WARN 4 in 4.12s]
15:28:52  33 of 72 START test is_empty_fct_undocumented_models_ .......................... [RUN]
15:28:52  30 of 72 PASS is_empty_fct_duplicate_sources_ .................................. [PASS in 2.82s]
15:28:52  34 of 72 START test is_empty_fct_undocumented_public_models_ ................... [RUN]
15:28:54  31 of 72 PASS is_empty_fct_hard_coded_references_ .............................. [PASS in 3.14s]
15:28:54  35 of 72 START test is_empty_fct_undocumented_source_tables_ ................... [RUN]
15:28:55  32 of 72 WARN 1 is_empty_fct_source_directories_ ............................... [WARN 1 in 2.95s]
15:28:55  36 of 72 START test is_empty_fct_undocumented_sources_ ......................... [RUN]
15:28:55  33 of 72 WARN 14 is_empty_fct_undocumented_models_ ............................. [WARN 14 in 2.90s]
15:28:55  37 of 72 START sql view model sandbox_serhii_dimchenko.fct_test_directories .... [RUN]
15:28:55  34 of 72 PASS is_empty_fct_undocumented_public_models_ ......................... [PASS in 3.27s]
15:28:55  38 of 72 START sql view model sandbox_serhii_dimchenko.int_all_dag_relationships  [RUN]
15:28:57  35 of 72 WARN 4 is_empty_fct_undocumented_source_tables_ ....................... [WARN 4 in 2.84s]
15:28:57  39 of 72 START sql view model sandbox_serhii_dimchenko.int_model_test_summary .. [RUN]
15:28:58  36 of 72 WARN 2 is_empty_fct_undocumented_sources_ ............................. [WARN 2 in 3.12s]
15:28:59  37 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_test_directories  [OK -1 in 4.14s]
15:28:59  40 of 72 START test is_empty_fct_test_directories_ ............................. [RUN]
15:29:00  38 of 72 OK created sql view model sandbox_serhii_dimchenko.int_all_dag_relationships  [OK -1 in 4.10s]
15:29:00  41 of 72 START sql view model sandbox_serhii_dimchenko.fct_chained_views_dependencies  [RUN]
15:29:00  42 of 72 START sql table model sandbox_serhii_dimchenko.fct_direct_join_to_source  [RUN]
15:29:02  39 of 72 OK created sql view model sandbox_serhii_dimchenko.int_model_test_summary  [OK -1 in 5.44s]
15:29:02  43 of 72 START sql view model sandbox_serhii_dimchenko.fct_exposure_parents_materializations  [RUN]
15:29:04  40 of 72 PASS is_empty_fct_test_directories_ ................................... [PASS in 5.55s]
15:29:04  44 of 72 START sql view model sandbox_serhii_dimchenko.fct_exposures_dependent_on_private_models  [RUN]
15:29:05  41 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_chained_views_dependencies  [OK -1 in 4.96s]
15:29:05  45 of 72 START sql table model sandbox_serhii_dimchenko.fct_marts_or_intermediate_dependent_on_source  [RUN]
15:29:06  43 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_exposure_parents_materializations  [OK -1 in 3.86s]
15:29:06  46 of 72 START sql view model sandbox_serhii_dimchenko.fct_model_directories ... [RUN]
15:29:10  44 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_exposures_dependent_on_private_models  [OK -1 in 5.24s]
15:29:10  47 of 72 START sql table model sandbox_serhii_dimchenko.fct_model_fanout ....... [RUN]
15:29:11  46 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_model_directories  [OK -1 in 5.24s]
15:29:11  48 of 72 START sql table model sandbox_serhii_dimchenko.fct_multiple_sources_joined  [RUN]
15:29:13  42 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_direct_join_to_source  [OK 2 in 13.59s]
15:29:13  49 of 72 START sql table model sandbox_serhii_dimchenko.fct_rejoining_of_upstream_concepts  [RUN]
15:29:18  45 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_marts_or_intermediate_dependent_on_source  [OK 2 in 13.87s]
15:29:18  50 of 72 START sql table model sandbox_serhii_dimchenko.fct_root_models ........ [RUN]
15:29:24  48 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_multiple_sources_joined  [OK 1 in 12.67s]
15:29:24  51 of 72 START sql table model sandbox_serhii_dimchenko.fct_source_fanout ...... [RUN]
15:29:25  47 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_model_fanout .. [OK 1 in 14.94s]
15:29:25  52 of 72 START sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_marts_or_intermediate  [RUN]
15:29:30  49 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_rejoining_of_upstream_concepts  [OK 0 in 16.39s]
15:29:30  53 of 72 START sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_staging  [RUN]
15:29:32  50 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_root_models ... [OK 1 in 14.03s]
15:29:32  54 of 72 START sql table model sandbox_serhii_dimchenko.fct_unused_sources ..... [RUN]
15:29:38  52 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_marts_or_intermediate  [OK 1 in 13.53s]
15:29:38  55 of 72 START sql view model sandbox_serhii_dimchenko.fct_missing_primary_key_tests  [RUN]
15:29:39  51 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_source_fanout . [OK 2 in 15.00s]
15:29:39  56 of 72 START sql view model sandbox_serhii_dimchenko.fct_test_coverage ....... [RUN]
15:29:43  53 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_staging_dependent_on_staging  [OK 1 in 13.07s]
15:29:43  57 of 72 START test valid_chained_views_dependencies ........................... [RUN]
15:29:43  55 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_missing_primary_key_tests  [OK -1 in 4.83s]
15:29:43  58 of 72 START test is_empty_fct_exposure_parents_materializations_ ............ [RUN]
15:29:45  54 of 72 OK created sql table model sandbox_serhii_dimchenko.fct_unused_sources  [OK 1 in 12.75s]
15:29:45  59 of 72 START test is_empty_fct_exposures_dependent_on_private_models_ ........ [RUN]
15:29:46  57 of 72 PASS valid_chained_views_dependencies ................................. [PASS in 3.84s]
15:29:46  60 of 72 START test is_empty_fct_model_directories_ ............................ [RUN]
15:29:47  56 of 72 OK created sql view model sandbox_serhii_dimchenko.fct_test_coverage .. [OK -1 in 7.81s]
15:29:47  61 of 72 START test is_empty_fct_direct_join_to_source_ ........................ [RUN]
15:29:50  61 of 72 WARN 2 is_empty_fct_direct_join_to_source_ ............................ [WARN 2 in 2.78s]
15:29:50  62 of 72 START test is_empty_fct_marts_or_intermediate_dependent_on_source_ .... [RUN]
15:29:50  58 of 72 PASS is_empty_fct_exposure_parents_materializations_ .................. [PASS in 7.19s]
15:29:50  63 of 72 START test is_empty_fct_multiple_sources_joined_ ...................... [RUN]
15:29:51  60 of 72 WARN 4 is_empty_fct_model_directories_ ................................ [WARN 4 in 4.85s]
15:29:51  64 of 72 START test is_empty_fct_model_fanout_ ................................. [RUN]
15:29:53  59 of 72 PASS is_empty_fct_exposures_dependent_on_private_models_ .............. [PASS in 7.35s]
15:29:53  65 of 72 START test is_empty_fct_rejoining_of_upstream_concepts_ ............... [RUN]
15:29:53  62 of 72 WARN 2 is_empty_fct_marts_or_intermediate_dependent_on_source_ ........ [WARN 2 in 2.91s]
15:29:53  66 of 72 START test is_empty_fct_root_models_ .................................. [RUN]
15:29:54  63 of 72 WARN 1 is_empty_fct_multiple_sources_joined_ .......................... [WARN 1 in 3.85s]
15:29:54  67 of 72 START test is_empty_fct_staging_dependent_on_marts_or_intermediate_ ... [RUN]
15:29:54  64 of 72 WARN 1 is_empty_fct_model_fanout_ ..................................... [WARN 1 in 2.90s]
15:29:54  68 of 72 START test is_empty_fct_source_fanout_ ................................ [RUN]
15:29:55  66 of 72 WARN 1 is_empty_fct_root_models_ ...................................... [WARN 1 in 2.74s]
15:29:55  69 of 72 START test is_empty_fct_staging_dependent_on_staging_ ................. [RUN]
15:29:56  65 of 72 PASS is_empty_fct_rejoining_of_upstream_concepts_ ..................... [PASS in 3.05s]
15:29:56  70 of 72 START test is_empty_fct_missing_primary_key_tests_ .................... [RUN]
15:29:57  68 of 72 WARN 2 is_empty_fct_source_fanout_ .................................... [WARN 2 in 2.78s]
15:29:57  67 of 72 WARN 1 is_empty_fct_staging_dependent_on_marts_or_intermediate_ ....... [WARN 1 in 2.91s]
15:29:57  71 of 72 START test is_empty_fct_unused_sources_ ............................... [RUN]
15:29:57  72 of 72 START test valid_test_coverage ........................................ [RUN]
15:29:58  69 of 72 WARN 1 is_empty_fct_staging_dependent_on_staging_ ..................... [WARN 1 in 2.87s]
15:30:00  71 of 72 WARN 1 is_empty_fct_unused_sources_ ................................... [WARN 1 in 2.91s]
15:30:00  70 of 72 WARN 14 is_empty_fct_missing_primary_key_tests_ ....................... [WARN 14 in 4.83s]
15:30:03  72 of 72 WARN 1 valid_test_coverage ............................................ [WARN 1 in 6.32s]
15:30:03  
15:30:03  Running 1 on-run-end hook

### List of issues raised by dbt_project_evaluator ###

-- valid_documentation_coverage --
|                measured_at | total_models | documented_models | documentation_coverage_pct | base_documentation_coverage_pct | staging_documentation_coverage_pct | ... |
| -------------------------- | ------------ | ----------------- | -------------------------- | ------------------------------- | ---------------------------------- | --- |
| 2024-01-08 15:30:05.300000 |           14 |                 0 |                          0 |                                 |                                  0 | ... |


-- is_empty_fct_model_naming_conventions_ --
| resource_name | prefix  | model_type | appropriate_prefixes |
| ------------- | ------- | ---------- | -------------------- |
| report_2      | report_ | other      | rpt_                 |
| report_1      | report_ | other      | rpt_                 |
| model_8       | model_  | marts      | dim_, fct_           |
| report_3      | report_ | other      | rpt_                 |


-- is_empty_fct_source_directories_ --
| resource_name    | resource_type | current_file_path                  | change_file_path_to                |
| ---------------- | ------------- | ---------------------------------- | ---------------------------------- |
| source_2.table_3 | source        | models/staging/source_1/source.yml | models/staging/source_2/source.yml |


-- is_empty_fct_undocumented_models_ --
| resource_name | model_type   |
| ------------- | ------------ |
| report_2      | other        |
| report_1      | other        |
| stg_model_4   | staging      |
| stg_model_3   | staging      |
| stg_model_1   | staging      |
| int_model_5   | intermediate |
| stg_model_5   | staging      |
| stg_model_2   | staging      |
| model_8       | marts        |
| dim_model_7   | marts        |
| fct_model_6   | marts        |
| fct_model_9   | marts        |
| int_model_4   | intermediate |
| report_3      | other        |


-- is_empty_fct_undocumented_source_tables_ --
| resource_name    |
| ---------------- |
| source_1.table_2 |
| source_1.table_4 |
| source_2.table_3 |
| source_1.table_5 |


-- is_empty_fct_undocumented_sources_ --
| source_name |
| ----------- |
| source_1    |
| source_2    |


-- is_empty_fct_direct_join_to_source_ --
| parent           | parent_resource_type | child       | child_resource_type | distance |
| ---------------- | -------------------- | ----------- | ------------------- | -------- |
| source_1.table_2 | source               | int_model_4 | model               |        1 |
| stg_model_1      | model                | int_model_4 | model               |        1 |


-- is_empty_fct_model_directories_ --
| resource_name | resource_type | model_type   | current_file_path                         | change_file_path_to                         |
| ------------- | ------------- | ------------ | ----------------------------------------- | ------------------------------------------- |
| int_model_5   | model         | intermediate | models/marts/int_model_5.sql              | models/.../intermediate/.../int_model_5.sql |
| dim_model_7   | model         | marts        | models/marts/intermediate/dim_model_7.sql | models/.../marts/.../dim_model_7.sql        |
| int_model_4   | model         | intermediate | models/marts/int_model_4.sql              | models/.../intermediate/.../int_model_4.sql |
| stg_model_3   | model         | staging      | models/staging/source_1/stg_model_3.sql   | models/staging/source_2/stg_model_3.sql     |


-- is_empty_fct_marts_or_intermediate_dependent_on_source_ --
| parent           | parent_resource_type | child       | child_model_type |
| ---------------- | -------------------- | ----------- | ---------------- |
| source_1.table_2 | source               | int_model_4 | intermediate     |
| source_1.table_5 | source               | fct_model_9 | marts            |


-- is_empty_fct_multiple_sources_joined_ --
| child       | source_parents                     |
| ----------- | ---------------------------------- |
| stg_model_2 | source_1.table_1, source_1.table_2 |


-- is_empty_fct_model_fanout_ --
| parent      | parent_model_type | leaf_children                |
| ----------- | ----------------- | ---------------------------- |
| fct_model_6 | marts             | report_1, report_2, report_3 |


-- is_empty_fct_root_models_ --
| child   |
| ------- |
| model_8 |


-- is_empty_fct_source_fanout_ --
| parent           | model_children           |
| ---------------- | ------------------------ |
| source_1.table_2 | int_model_4, stg_model_2 |
| source_1.table_1 | stg_model_1, stg_model_2 |


-- is_empty_fct_staging_dependent_on_marts_or_intermediate_ --
| parent      | parent_model_type | child       | child_model_type |
| ----------- | ----------------- | ----------- | ---------------- |
| fct_model_9 | marts             | stg_model_5 | staging          |


-- is_empty_fct_staging_dependent_on_staging_ --
| parent      | parent_model_type | child       | child_model_type |
| ----------- | ----------------- | ----------- | ---------------- |
| stg_model_2 | staging           | stg_model_4 | staging          |


-- is_empty_fct_unused_sources_ --
| parent           |
| ---------------- |
| source_1.table_4 |


-- is_empty_fct_missing_primary_key_tests_ --
| resource_name | resource_type | model_type   | is_primary_key_tested | number_of_tests_on_model |
| ------------- | ------------- | ------------ | --------------------- | ------------------------ |
| report_2      | model         | other        |                 False |                        0 |
| report_1      | model         | other        |                 False |                        0 |
| stg_model_4   | model         | staging      |                 False |                        0 |
| stg_model_3   | model         | staging      |                 False |                        0 |
| stg_model_1   | model         | staging      |                 False |                        0 |
| int_model_5   | model         | intermediate |                 False |                        0 |
| stg_model_5   | model         | staging      |                 False |                        0 |
| stg_model_2   | model         | staging      |                 False |                        0 |
| model_8       | model         | marts        |                 False |                        0 |
| dim_model_7   | model         | marts        |                 False |                        0 |
| fct_model_6   | model         | marts        |                 False |                        0 |
| fct_model_9   | model         | marts        |                 False |                        0 |
| int_model_4   | model         | intermediate |                 False |                        0 |
| report_3      | model         | other        |                 False |                        0 |


-- valid_test_coverage --
|                measured_at | total_models | total_tests | tested_models | test_coverage_pct | base_test_coverage_pct | ... |
| -------------------------- | ------------ | ----------- | ------------- | ----------------- | ---------------------- | --- |
| 2024-01-08 15:30:43.274000 |           14 |           0 |             0 |                 0 |                        | ... |



15:30:46  1 of 1 START hook: dbt_project_evaluator_integration_tests_2.on-run-end.0 ...... [RUN]
15:30:46  1 of 1 OK hook: dbt_project_evaluator_integration_tests_2.on-run-end.0 ......... [OK in 0.00s]
15:30:46  
15:30:46  
15:30:46  Finished running 22 table models, 22 view models, 1 seed, 27 tests, 1 hook in 0 hours 3 minutes and 7.25 seconds (187.25s).
15:30:47  
15:30:47  Completed with 18 warnings:
15:30:47  
15:30:47  Warning in test valid_documentation_coverage (models/marts/documentation/documentation.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/documentation/documentation.yml/valid_documentation_coverage.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_model_naming_conventions_ (models/marts/structure/structure.yml)
15:30:47  Got 4 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/structure/structure.yml/is_empty_fct_model_naming_conventions_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_source_directories_ (models/marts/structure/structure.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/structure/structure.yml/is_empty_fct_source_directories_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_undocumented_models_ (models/marts/documentation/documentation.yml)
15:30:47  Got 14 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/documentation/documentation.yml/is_empty_fct_undocumented_models_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_undocumented_source_tables_ (models/marts/documentation/documentation.yml)
15:30:47  Got 4 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/documentation/documentation.yml/is_empty_fct_undocumented_source_tables_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_undocumented_sources_ (models/marts/documentation/documentation.yml)
15:30:47  Got 2 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/documentation/documentation.yml/is_empty_fct_undocumented_sources_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_direct_join_to_source_ (models/marts/dag/dag.yml)
15:30:47  Got 2 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_direct_join_to_source_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_model_directories_ (models/marts/structure/structure.yml)
15:30:47  Got 4 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/structure/structure.yml/is_empty_fct_model_directories_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_marts_or_intermediate_dependent_on_source_ (models/marts/dag/dag.yml)
15:30:47  Got 2 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_marts_or_intermediate_dependent_on_source_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_multiple_sources_joined_ (models/marts/dag/dag.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_multiple_sources_joined_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_model_fanout_ (models/marts/dag/dag.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_model_fanout_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_root_models_ (models/marts/dag/dag.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_root_models_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_source_fanout_ (models/marts/dag/dag.yml)
15:30:47  Got 2 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_source_fanout_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_staging_dependent_on_marts_or_intermediate_ (models/marts/dag/dag.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_staging_dependent_on_marts_or_intermediate_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_staging_dependent_on_staging_ (models/marts/dag/dag.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_staging_dependent_on_staging_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_unused_sources_ (models/marts/dag/dag.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/dag/dag.yml/is_empty_fct_unused_sources_.sql
15:30:47  
15:30:47  Warning in test is_empty_fct_missing_primary_key_tests_ (models/marts/tests/testing.yml)
15:30:47  Got 14 results, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/tests/testing.yml/is_empty_fct_missing_primary_key_tests_.sql
15:30:47  
15:30:47  Warning in test valid_test_coverage (models/marts/tests/testing.yml)
15:30:47  Got 1 result, configured to warn if != 0
15:30:47  
15:30:47    compiled Code at target/compiled/dbt_project_evaluator/models/marts/tests/testing.yml/valid_test_coverage.sql
15:30:47  
15:30:47  Done. PASS=54 WARN=18 ERROR=0 SKIP=0 TOTAL=72

svdimchenko · 2024-01-08T16:04:09Z

there is a problem with databricks now, but I'm not sure if it occured due to my changes...

16:01:31    Compilation Error in model metricflow_time_spine (models/utils/metricflow_time_spine.sql)
  int() argument must be a string, a bytes-like object or a number, not 'NoneType'

b-per · 2024-01-25T08:43:44Z

It is not related to the latest changes and was flagged a few days ago.

A new release of dbt-databricks was done so I will just retrigger the CI run for databricks and it should work.

graciegoheen

Thank you for all of the effort here - excited to get this shipped!

Add AWS Athena support

56c2035

svdimchenko requested review from graciegoheen, dave-connors-3 and b-per as code owners September 26, 2023 08:50

Minor fix

68c54f5

svdimchenko changed the title ~~Add AWS Athena support~~ feat: Add AWS Athena support Sep 26, 2023

svdimchenko and others added 2 commits September 26, 2023 13:34

Add limit for athena

9da9f74

Merge branch 'main' into feat/dbt-athena-support

ec6e884

graciegoheen reviewed Oct 19, 2023

View reviewed changes

macros/recursive_dag.sql Outdated Show resolved Hide resolved

graciegoheen reviewed Oct 19, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

Merge branch 'main' into feat/dbt-athena-support

c199a63

svdimchenko and others added 5 commits January 8, 2024 15:17

Merge branch 'main' into feat/dbt-athena-support

067052d

Implement separate macro for aws athena

aa8cee9

Run integration tests for athena

2863929

Add readme regarding local tests for Athena

d375015

Fixed sample profiles

52a0f27

Merge branch 'main' into feat/dbt-athena-support

fa5c7c9

graciegoheen approved these changes Feb 1, 2024

View reviewed changes

graciegoheen merged commit 5dc21d6 into dbt-labs:main Feb 1, 2024
7 checks passed

svdimchenko deleted the feat/dbt-athena-support branch February 1, 2024 16:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add AWS Athena support #383

feat: Add AWS Athena support #383

svdimchenko commented Sep 26, 2023 •

edited

Loading

svdimchenko commented Oct 19, 2023

b-per commented Oct 19, 2023

dave-connors-3 commented Dec 12, 2023

svdimchenko commented Dec 12, 2023

svdimchenko commented Jan 8, 2024

svdimchenko commented Jan 8, 2024

svdimchenko commented Jan 8, 2024

b-per commented Jan 25, 2024

graciegoheen left a comment

feat: Add AWS Athena support #383

feat: Add AWS Athena support #383

Conversation

svdimchenko commented Sep 26, 2023 • edited Loading

Description & motivation

Checklist

svdimchenko commented Oct 19, 2023

b-per commented Oct 19, 2023

dave-connors-3 commented Dec 12, 2023

svdimchenko commented Dec 12, 2023

svdimchenko commented Jan 8, 2024

svdimchenko commented Jan 8, 2024

svdimchenko commented Jan 8, 2024

b-per commented Jan 25, 2024

graciegoheen left a comment

Choose a reason for hiding this comment

svdimchenko commented Sep 26, 2023 •

edited

Loading