Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

destination acceptance tests: get normalization and dbt support from definition #21299

Conversation

alafanechere
Copy link
Contributor

@alafanechere alafanechere commented Jan 12, 2023

What

Closes #21303
In our efforts to decouple normalization version from platform version (#16671) we now expose the normalization version (and normalization/dbt support) in the destination definition and not in its spec.
#20245 changed the DestinationAcceptanceTest to retrieve the normalization configuration from the definition.
We now want to update the rest of the test to discard the retrieval of normalization/dbt support from the spec and assess normlization/dbt support from the definition.

How

  • Remove supportsDbt and supportsNormalization methods from DestinationAcceptanceTest
  • Use normalizationFromDefinition

Suggested review order

Then review the connector specific acceptance tests.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 12, 2023

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

✅ Sources (0)

Connector Version Changelog Publish
  • See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (48)

Connector Version Changelog Publish
destination-aws-datalake 0.1.1
destination-azure-blob-storage 0.1.6
destination-bigquery 1.2.9
destination-bigquery-denormalized 1.2.10
destination-cassandra 0.1.4
destination-clickhouse 0.2.1
destination-clickhouse-strict-encrypt 0.2.1 🔵
(ignored)
🔵
(ignored)
destination-csv 1.0.0
(changelog missing)
destination-databricks 0.3.1
destination-dev-null 0.2.7 🔵
(ignored)
🔵
(ignored)
destination-doris 0.1.0
destination-dynamodb 0.1.7
destination-e2e-test 0.2.4
destination-elasticsearch 0.1.6
destination-elasticsearch-strict-encrypt 0.1.6 🔵
(ignored)
🔵
(ignored)
destination-gcs 0.2.12
destination-iceberg 0.1.0
destination-jdbc 0.3.14 🔵
(ignored)
🔵
(ignored)
destination-kafka 0.1.10
destination-keen 0.2.4
destination-kinesis 0.1.5
destination-local-json 0.2.11
destination-mariadb-columnstore 0.1.7
destination-mongodb 0.1.9
destination-mongodb-strict-encrypt 0.1.9 🔵
(ignored)
🔵
(ignored)
destination-mqtt 0.1.3
destination-mssql 0.1.22
destination-mssql-strict-encrypt 0.1.22 🔵
(ignored)
🔵
(ignored)
destination-mysql 0.1.20
destination-mysql-strict-encrypt 0.1.21
(mismatch: 0.1.20)
🔵
(ignored)
🔵
(ignored)
destination-oracle 0.1.19
destination-oracle-strict-encrypt 0.1.19 🔵
(ignored)
🔵
(ignored)
destination-postgres 0.3.26
destination-postgres-strict-encrypt 0.3.26 🔵
(ignored)
🔵
(ignored)
destination-pubsub 0.2.0
destination-pulsar 0.1.3
destination-r2 0.1.0
destination-redis 0.1.4
destination-redpanda 0.1.0
destination-redshift 0.3.53
destination-rockset 0.1.4
destination-s3 0.3.18
destination-s3-glue 0.1.1
destination-scylla 0.1.3
destination-snowflake 0.4.42
(changelog missing)
destination-teradata 0.1.0
destination-tidb 0.1.0
destination-yugabytedb 0.1.0
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category Status Actionable Item
Version
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.

doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.

changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish
not in seed
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.

diff seed version
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/3901190765
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/3901190765
No Python unittests run

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3901292228
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3901292228
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@alafanechere alafanechere changed the title destination acceptance test: get normalization and dbt support from definition. destination acceptance tests: get normalization and dbt support from definition Jan 12, 2023
@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/3901699991
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/3901699991
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

/test connector=connectors/destination-elasticsearch

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-elasticsearch-strict-encrypt

🕑 connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901713965
✅ connectors/destination-elasticsearch-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901713965
No Python unittests run

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3901728188
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3901728188
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-teradata

🕑 connectors/destination-teradata https://github.com/airbytehq/airbyte/actions/runs/3901731457

@alafanechere alafanechere marked this pull request as ready for review January 12, 2023 11:19
@alafanechere alafanechere requested a review from a team as a code owner January 12, 2023 11:19
@alafanechere alafanechere requested a review from a team January 12, 2023 11:19
@alafanechere alafanechere marked this pull request as draft January 12, 2023 11:32
@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3901907513
❌ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3901907513
🐛 https://gradle.com/s/imic37b5snbbo

Build Failed

Test summary info:

Could not find result summary

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-clickhouse-strict-encrypt

🕑 connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901908586
❌ connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901908586
🐛 https://gradle.com/s/x57ywgvj737ia

Build Failed

Test summary info:

Could not find result summary

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/3901911248
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/3901911248
🐛 https://gradle.com/s/tcmffj62zw44k

Build Failed

Test summary info:

Could not find result summary

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-mysql-strict-encrypt

🕑 connectors/destination-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901911813
✅ connectors/destination-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901911813
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/3901915358
❌ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/3901915358
🐛 https://gradle.com/s/23vpddupjbq2i

Build Failed

Test summary info:

Could not find result summary

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901916009
❌ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901916009
🐛 https://gradle.com/s/wxjgznoqv55vs

Build Failed

Test summary info:

Could not find result summary

@alafanechere alafanechere temporarily deployed to more-secrets January 12, 2023 11:45 — with GitHub Actions Inactive
@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-tidb

🕑 connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/3901919304
❌ connectors/destination-tidb https://github.com/airbytehq/airbyte/actions/runs/3901919304
🐛 https://gradle.com/s/wbkz3scrvcxei

Build Failed

Test summary info:

Could not find result summary

@alafanechere alafanechere temporarily deployed to more-secrets January 12, 2023 11:45 — with GitHub Actions Inactive
@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3901921181
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3901921181
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@alafanechere
Copy link
Contributor Author

/test connector=connectors/destination-postgres-strict-encrypt

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/3901922902

@alafanechere
Copy link
Contributor Author

alafanechere commented Jan 12, 2023

/test connector=connectors/destination-oracle-strict-encrypt

🕑 connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901924395
❌ connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3901924395
🐛 https://gradle.com/s/tdgtda3th7rke

Build Failed

Test summary info:

Could not find result summary

@alafanechere
Copy link
Contributor Author

Hey @airbytehq/jdbc-connectors could you please let me know if the failing tests could be related to my PR?

@alafanechere alafanechere marked this pull request as ready for review January 12, 2023 14:18
@alafanechere alafanechere temporarily deployed to more-secrets January 17, 2023 09:30 — with GitHub Actions Inactive
@alafanechere alafanechere temporarily deployed to more-secrets January 17, 2023 09:31 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Jan 17, 2023

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage 26.73% 🍏

@ryankfu
Copy link
Contributor

ryankfu commented Jan 17, 2023

@alafanechere your changes shouldn't be the reason that the integration tests are failing, that said Ed and I created two different integration test runs on main to verify if the integration tests were already failing

Copy link
Contributor

@edgao edgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I think these changes look fine. E.g. one of the failing tests is the recently-added column selection test, which is also failing on master.

Copy link
Contributor

@evantahler evantahler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2023-01-17 at 4 03 34 PM

The best kind of PR!

Approval contigent on the tests passing, of course.

@alafanechere alafanechere temporarily deployed to more-secrets January 18, 2023 16:35 — with GitHub Actions Inactive
@alafanechere alafanechere temporarily deployed to more-secrets January 18, 2023 16:36 — with GitHub Actions Inactive
@alafanechere alafanechere merged commit 6c2b9a1 into master Jan 18, 2023
@alafanechere alafanechere deleted the augustin/destination-acceptance-test/normalization-dbt-from-def branch January 18, 2023 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment