Extract a test to verify Databricks and OSS Delta TPCH tables and merge connector tests into one #17137
+118
−47
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ebyhr
added
the
no-release-notes
ebyhr
commented
Apr 20, 2023
Comment on lines
+73
to
+75
| assertThat(query("SELECT * FROM " + tableName)) | ||
| .skippingTypesCheck() // Delta Lake connector returns varchar, but TPCH connector returns varchar(n) | ||
| .matches("SELECT * FROM tpch.tiny." + tableName); |
There was a problem hiding this comment.
Please feel free to leave comments if we need more tests.
ebyhr
force-pushed
the
ebi/delta-connector-test
branch
from
b2ff21e to
ea9a14b
Compare
Merge two connector tests into one in Delta Lake connector.
ebyhr
force-pushed
the
ebi/delta-connector-test
branch
from
ea9a14b to
bfce114
Compare
findinpath
approved these changes
Apr 21, 2023
findinpath
reviewed
Apr 21, 2023
| bucketName)); | ||
| }); | ||
| queryRunner.execute("CREATE SCHEMA " + SCHEMA + " WITH (location = 's3://" + BUCKET_NAME + "/" + SCHEMA + "')"); | ||
| copyTpchTables(queryRunner, "tpch", TINY_SCHEMA_NAME, queryRunner.getDefaultSession(), REQUIRED_TPCH_TABLES); |
There was a problem hiding this comment.
Adding here the logs of the related operations while running the test on my machine in case anybody is interested:
2023-04-21T06:41:49.983+0200 INFO main io.trino.testing.QueryAssertions Imported 1500 rows for customer in 4.10s
2023-04-21T06:41:50.184+0200 INFO dispatcher-query-5 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044149_00002_m52x4 :: FINISHED :: elapsed 181ms :: planning 80ms :: waiting 12ms :: scheduling 41ms :: running 40ms :: finishing 20ms :: begin 2023-04-21T06:41:49.994+02:00 :: end 2023-04-21T06:41:50.175+02:00
2023-04-21T06:41:50.637+0200 INFO dispatcher-query-1 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044150_00003_m52x4 :: FINISHED :: elapsed 357ms :: planning 71ms :: waiting 226ms :: scheduling 264ms :: running 12ms :: finishing 10ms :: begin 2023-04-21T06:41:50.273+02:00 :: end 2023-04-21T06:41:50.630+02:00
2023-04-21T06:41:50.639+0200 INFO main io.trino.testing.QueryAssertions Running import for nation
2023-04-21T06:41:51.416+0200 INFO dispatcher-query-5 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044150_00004_m52x4 :: FINISHED :: elapsed 766ms :: planning 134ms :: waiting 139ms :: scheduling 151ms :: running 44ms :: finishing 437ms :: begin 2023-04-21T06:41:50.645+02:00 :: end 2023-04-21T06:41:51.411+02:00
2023-04-21T06:41:51.419+0200 INFO main io.trino.testing.QueryAssertions Imported 25 rows for nation in 779.30ms
2023-04-21T06:41:51.498+0200 INFO dispatcher-query-6 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044151_00005_m52x4 :: FINISHED :: elapsed 68ms :: planning 23ms :: waiting 2ms :: scheduling 12ms :: running 22ms :: finishing 11ms :: begin 2023-04-21T06:41:51.423+02:00 :: end 2023-04-21T06:41:51.491+02:00
2023-04-21T06:41:51.776+0200 INFO dispatcher-query-3 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044151_00006_m52x4 :: FINISHED :: elapsed 268ms :: planning 36ms :: waiting 197ms :: scheduling 216ms :: running 7ms :: finishing 9ms :: begin 2023-04-21T06:41:51.504+02:00 :: end 2023-04-21T06:41:51.772+02:00
2023-04-21T06:41:51.778+0200 INFO main io.trino.testing.QueryAssertions Running import for orders
2023-04-21T06:41:52.775+0200 INFO dispatcher-query-1 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044151_00007_m52x4 :: FINISHED :: elapsed 987ms :: planning 169ms :: waiting 120ms :: scheduling 137ms :: running 119ms :: finishing 562ms :: begin 2023-04-21T06:41:51.784+02:00 :: end 2023-04-21T06:41:52.771+02:00
2023-04-21T06:41:52.776+0200 INFO main io.trino.testing.QueryAssertions Imported 15000 rows for orders in 997.91ms
2023-04-21T06:41:52.860+0200 INFO dispatcher-query-6 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044152_00008_m52x4 :: FINISHED :: elapsed 75ms :: planning 22ms :: waiting 2ms :: scheduling 13ms :: running 31ms :: finishing 9ms :: begin 2023-04-21T06:41:52.780+02:00 :: end 2023-04-21T06:41:52.855+02:00
2023-04-21T06:41:53.136+0200 INFO dispatcher-query-0 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044152_00009_m52x4 :: FINISHED :: elapsed 264ms :: planning 43ms :: waiting 187ms :: scheduling 207ms :: running 6ms :: finishing 8ms :: begin 2023-04-21T06:41:52.867+02:00 :: end 2023-04-21T06:41:53.131+02:00
2023-04-21T06:41:53.137+0200 INFO main io.trino.testing.QueryAssertions Running import for region
2023-04-21T06:41:53.782+0200 INFO dispatcher-query-0 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044153_00010_m52x4 :: FINISHED :: elapsed 638ms :: planning 137ms :: waiting 121ms :: scheduling 130ms :: running 19ms :: finishing 352ms :: begin 2023-04-21T06:41:53.140+02:00 :: end 2023-04-21T06:41:53.778+02:00
2023-04-21T06:41:53.783+0200 INFO main io.trino.testing.QueryAssertions Imported 5 rows for region in 646.23ms
2023-04-21T06:41:53.840+0200 INFO dispatcher-query-0 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044153_00011_m52x4 :: FINISHED :: elapsed 50ms :: planning 17ms :: waiting 2ms :: scheduling 9ms :: running 15ms :: finishing 9ms :: begin 2023-04-21T06:41:53.787+02:00 :: end 2023-04-21T06:41:53.837+02:00
2023-04-21T06:41:54.039+0200 INFO dispatcher-query-5 io.trino.event.QueryMonitor TIMELINE: Query 20230421_044153_00012_m52x4 :: FINISHED :: elapsed 190ms :: planning 29ms :: waiting 131ms :: scheduling 144ms :: running 9ms :: finishing 8ms :: begin 2023-04-21T06:41:53.845+02:00 :: end 2023-04-21T06:41:54.035+02:00
2023-04-21T06:41:54.040+0200 INFO main io.trino.testing.QueryAssertions Loading from tpch.tiny complete in 8.15s
|
Also worth mentioning, on this PR: on master |
findepi
approved these changes
Apr 21, 2023
|
thank you |
|
CI hit #15187 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Extract a test to verify Databricks and OSS Delta TPCH tables and merge connector tests into one
Release notes
(x) This is not user-visible or docs only and no release notes are required.