Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable incorrect pushdown in Mongo connector #3053

Merged
merged 5 commits into from
Mar 16, 2020
Merged

Disable incorrect pushdown in Mongo connector #3053

merged 5 commits into from
Mar 16, 2020

Conversation

findepi
Copy link
Member

@findepi findepi commented Mar 10, 2020

Fixes #1781

@findepi findepi requested a review from ebyhr March 10, 2020 09:16
@cla-bot cla-bot bot added the cla-signed label Mar 10, 2020
@ebyhr
Copy link
Member

ebyhr commented Mar 10, 2020

Could you confirm the following CI failure?

[ERROR] Failures: 
[ERROR]   TestMongoSession.testBuildQuery:52 Maps do not have the same size:1 != 2
[ERROR]   TestMongoSession.testBuildQueryNull:103 Maps do not have the same size:0 != 1
[ERROR]   TestMongoSession.testBuildQueryOr:90 Maps do not have the same size:0 != 1
[ERROR]   TestMongoSession.testBuildQueryStringType:66 Maps do not have the same size:1 != 2

@findepi findepi force-pushed the mongo-pushdown branch 3 times, most recently from 806bd0c to 6c1e7b5 Compare March 10, 2020 16:52
@findepi
Copy link
Member Author

findepi commented Mar 10, 2020

@ebyhr thanks, updated

@findepi
Copy link
Member Author

findepi commented Mar 10, 2020

CI failed -- #3049

@findepi findepi force-pushed the mongo-pushdown branch 4 times, most recently from 47d5fa3 to 46e4b2d Compare March 11, 2020 13:12
@findepi
Copy link
Member Author

findepi commented Mar 11, 2020

CI failed -- checkout failed. This PR is cursed.

@findepi findepi force-pushed the mongo-pushdown branch 3 times, most recently from 0ce1962 to 547d117 Compare March 11, 2020 22:26
@findepi
Copy link
Member Author

findepi commented Mar 12, 2020

Finally green. Please review.

kokosing
kokosing reviewed Mar 12, 2020
View reviewed changes
Copy link
Member

@kokosing kokosing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good modulo:

  • improve commit message to mention that pushdown is disabled only for not supported types
  • restore previous behaviour for types that are still supported
  • test that exposes the bug that you are fixing

@findepi findepi force-pushed the mongo-pushdown branch 3 times, most recently from 2372961 to c59e746 Compare March 12, 2020 16:38
@findepi
Copy link
Member Author

findepi commented Mar 12, 2020

improve commit message to mention that pushdown is disabled only for not supported types

it says "disable incorrect pushdown", not "disable pushdown"

@findepi
Copy link
Member Author

findepi commented Mar 12, 2020

added test

@findepi findepi force-pushed the mongo-pushdown branch 3 times, most recently from f9f3d57 to 11f6aa7 Compare March 13, 2020 22:21
@findepi
Copy link
Member Author

findepi commented Mar 13, 2020

AC

@findepi findepi mentioned this pull request Mar 14, 2020
Closed
@findepi findepi force-pushed the mongo-pushdown branch 2 times, most recently from 5555d0a to d126838 Compare March 14, 2020 19:32
findepi added 4 commits March 14, 2020 21:11
@findepi
Start MongoQueryRunner on 8080
819ffe7
@findepi
Fix types in test
d1ef2ab 
COL1 is of type BIGINT, so should not be used with varchar-typed domain.
Add another varchar column.
@findepi
Fix Mongo type mappings
c7d2249
@findepi
Disable incorrect pushdown in Mongo connector
c8e55be
@findepi
Copy link
Member Author

findepi commented Mar 15, 2020

@kokosing ready for re-review

kokosing
kokosing approved these changes Mar 15, 2020
View reviewed changes
Copy link
Member

@kokosing kokosing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing a test!

presto-testing/src/main/java/io/prestosql/testing/AbstractTestDistributedQueries.java

String tableName = "test_data_mapping_smoke_test_" + prestoTypeName.replaceAll("[^a-zA-Z0-9]", "_");

Runnable setup = () -> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you might want to use io.prestosql.testing.sql.TestTable.randomTableSuffix. Sometimes ATDQ is run in parallel for some connectors with several different configurations and then such test becomes flaky due table name conflict.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

presto-testing/src/main/java/io/prestosql/testing/AbstractTestDistributedQueries.java
if (dataMappingTestSetup.isUnsupportedType()) {
String typeNameBase = prestoTypeName.replaceFirst("\\(.*", "");
String expectedMessagePart = format("(%1$s.*not (yet )?supported)|((?i)unsupported.*%1$s)|((?i)not supported.*%1$s)", Pattern.quote(typeNameBase));
assertThatThrownBy(setup::run)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/setup::run/setup?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ThrowingCallable is not assignable from Runnable

presto-testing/src/main/java/io/prestosql/testing/AbstractTestDistributedQueries.java
String createTable = format("CREATE TABLE %s(id bigint, value %s)", tableName, prestoTypeName);
assertUpdate(createTable);
assertUpdate(
format("INSERT INTO %s VALUES (10000, NULL), (10001, %s), (99999, %s)", tableName, sampleValueLiteral, highValueLiteral),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

10000 -> 1, 10001 -> 2, 99999 -> 3

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That way it is a bit easier. It is just nit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO contrary -- 10001 or 99999 are easier to find than 1,2

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This depends on what you are optimizing for.

@findepi findepi merged commit 92b118f into trinodb:master Mar 16, 2020
@findepi findepi deleted the mongo-pushdown branch March 16, 2020 08:07
@findepi findepi added this to the 331 milestone Mar 16, 2020
@findepi findepi mentioned this pull request Mar 16, 2020
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kokosing kokosing kokosing approved these changes

@ebyhr ebyhr Awaiting requested review from ebyhr

Assignees
No one assigned
Labels
cla-signed
Milestone
331
Development

Successfully merging this pull request may close these issues.

Pull MongoDB data using Presto - date comparison in the where clause
3 participants
@findepi @ebyhr @kokosing