Implement dereference pushdown for MongoDB connector #17710

krvikash · 2023-05-31T10:44:27Z

Description

This PR implements dereference pushdown for Mongodb connector(similar to #17085).

This adds significant performance improvements for queries accessing nested fields inside struct/row columns. They have been optimized through the pushdown of dereference expressions. With this feature, the query execution prunes structural data eagerly, extracting the necessary fields.

More Details about dereference pushdown: https://trino.io/blog/2020/08/14/dereference-pushdown.html

Note: This PR merges the work of #14467 and #16790

Additional context and related issues

The feature is enabled by default.

The feature can be disabled by setting mongodb.projection-pushdown-enabled configuration property or mongodb.projection_pushdown_enabled session property to false.

Release notes

(X) Release notes are required, with the following suggested text:

# Mongodb
* Improve read performance for tables with row (struct) columns when only subset of fields is needed by a query. ({issue}`issuenumber`)

krvikash · 2023-06-05T05:57:54Z

rebased to resolve conflicts

docs/src/main/sphinx/connector/mongodb.rst

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoPageSource.java

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java

krvikash · 2023-06-06T08:50:46Z

Addressed comments and added TestMongoComplexTypePredicatePushDown for predicate pushdown.

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoClientConfig.java

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoColumnHandle.java

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

Praveen2112 · 2023-06-06T09:28:34Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoSession.java

+     * Creates a set of sufficient columns for the input projected columns. For example,
+     * if input {@param columns} include columns "a.b" and "a.b.c", then they will be projected from a single column "a.b".
+     */


Should the engine performs this optimization i.e when we were about to project columns like a.b and a.b.c there should be a project operator which ensures the plan like

tableScan(with column a.b) -> project (with two column a.b and a.b.c) ?

cc: @kasiafi , @findepi , @hashhar , @martint

Praveen2112 · 2023-06-06T09:34:40Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoPageSource.java

+        if (mongoColumnHandle.isBaseColumn()) {
+            return value;
+        }
+        if (value instanceof DBRef dbRefValue) {


Is it possible for the engine to send only DBRef to the underlying MongoDB and it could apply the projection pushdown at its end ?

After the projection pushdown, MongoPageSource will have all dereferenced columns.

I could not find a way to send only DBRef here. Let me know if you have some suggestions.

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoSession.java

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/BaseMongoConnectorSmokeTest.java

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java

lib/trino-plugin-toolkit/src/main/java/io/trino/plugin/base/projection/ApplyProjectionUtil.java

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java

Praveen2112 · 2023-06-26T07:15:37Z

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java

@@ -0,0 +1,295 @@
+/*


Can we have the plan assertion as a part of ConnectorTest/ConnectorSmokeTest

This is similar to what TestDeltaLakeProjectionPushdownPlans and other connectors' Projection Pushdown Plan tests.

Praveen2112 · 2023-06-26T07:18:34Z

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java

+                                table -> {
+                                    MongoTableHandle mongoTableHandle = (MongoTableHandle) table;
+                                    TupleDomain<ColumnHandle> constraint = mongoTableHandle.getConstraint();
+                                    return mongoTableHandle.getProjectedColumns().equals(ImmutableSet.of(column0Handle, columnY))


We don't have to do an exact equals here - We need to ensure that - we project till the parent column.

#17710 (comment)

Praveen2112 · 2023-06-28T07:38:26Z

Can we squash the fixup commits

krvikash · 2023-06-28T07:45:04Z

Can we squash the fixup commits

Done

Name Base class test relevant so that it can be use for non-file format based connector test.

Praveen2112

% comments

...no-plugin-toolkit/src/test/java/io/trino/plugin/base/projection/TestApplyProjectionUtil.java

lib/trino-plugin-toolkit/src/main/java/io/trino/plugin/base/projection/ApplyProjectionUtil.java

Praveen2112 · 2023-06-28T07:43:28Z

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java

+                .setCatalogSessionProperty(CATALOG, "projection_pushdown_enabled", "false")
+                .build();
+
+        getQueryRunner().execute("CREATE TABLE " + tableName + " (col0) AS SELECT CAST(row(5, 6) AS row(a bigint, b bigint)) AS col0 WHERE false");


Can we replace them with TestTable

This test class uses LocalQueryRunner which does not allow DROP TABLE. TestTable execute 'DROP TABLE' after executing on close. So can not use TestTable here.

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoTableHandle.java

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoColumnHandle.java

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoPageSource.java

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoSession.java

Praveen2112 · 2023-06-28T10:00:03Z

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

+
+    private static boolean isPushDownSupported(ConnectorExpression connectorExpression)
+    {
+        if (!(connectorExpression instanceof FieldDereference fieldDereference)) {


This is a bit tricky - now it means we support non FieldDereference pushdown expression too.

Updated method to return false for connectorExpression other than Variable and FieldDereference.

krvikash · 2023-06-28T20:39:46Z

Thanks, @Praveen2112 for reviewing. I have addressed the comments.

Praveen2112

Minor nits.

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

lib/trino-plugin-toolkit/src/main/java/io/trino/plugin/base/projection/ApplyProjectionUtil.java

plugin/trino-mongodb/src/main/java/io/trino/plugin/mongodb/MongoMetadata.java

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java

Co-authored-by: praveenkrishna <[email protected]> Co-authored-by: Mateusz "Serafin" Gajewski <[email protected]>

krvikash · 2023-06-30T07:47:57Z

Thanks, @Praveen2112 for reviewing. Addressed comments.

krvikash · 2023-06-30T09:12:06Z

CI (https://github.com/trinodb/trino/actions/runs/5420554711/jobs/9854904606?pr=17710) is failing. Same issue #10631

krvikash · 2023-07-04T06:17:17Z

Thank you all for reviewing the PR and merging it.

Praveen2112 · 2023-07-04T06:18:22Z

Thanks for working on this.

cla-bot bot added the cla-signed label May 31, 2023

github-actions bot added delta-lake Delta Lake connector docs hive Hive connector iceberg Iceberg connector mongodb MongoDB connector tests:hive labels May 31, 2023

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from b38b744 to ad5cd4b Compare May 31, 2023 11:19

krvikash mentioned this pull request May 31, 2023

Calculate Physical Input size for query stats in Mongo DB connector #17713

Open

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch 2 times, most recently from 305397a to 83d9bd8 Compare June 1, 2023 13:45

krvikash marked this pull request as ready for review June 1, 2023 13:48

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch 2 times, most recently from 6cc226e to 851554e Compare June 1, 2023 16:31

krvikash requested review from atanasenko, huberty89, ebyhr, kokosing and Praveen2112 June 5, 2023 05:51

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from 851554e to 8baaf45 Compare June 5, 2023 05:56

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from 8baaf45 to 777382a Compare June 5, 2023 06:13

krvikash mentioned this pull request Jun 5, 2023

Add simple projection pushdown support to MongoDB #17618

Closed

kokosing reviewed Jun 5, 2023

View reviewed changes

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from 777382a to 03877e3 Compare June 6, 2023 08:49

ebyhr removed their request for review June 6, 2023 08:51

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from 03877e3 to 7f8da8b Compare June 6, 2023 09:02

Praveen2112 reviewed Jun 6, 2023

View reviewed changes

ebyhr reviewed Jun 6, 2023

View reviewed changes

plugin/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoConnectorTest.java Show resolved Hide resolved

Praveen2112 reviewed Jun 26, 2023

View reviewed changes

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch 3 times, most recently from e6e59ac to 8866ee9 Compare June 26, 2023 10:59

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from 8866ee9 to 4d3f73f Compare June 28, 2023 07:44

krvikash and others added 5 commits June 28, 2023 13:38

Make variable arrangement consistent in MongoTableHandle

a1297bc

Move ProjectionPushDown util methods to plugin toolkit

b55bf89

Rename BaseTestFileFormatComplexTypesPredicatePushDown

b80e56d

Name Base class test relevant so that it can be use for non-file format based connector test.

Verify testProjectionPushdown with projection pushdown disabled

d1472d1

Add test for dotted field and dollar prefixed field in mongo connector

29a6995

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from 4d3f73f to 7f7cac3 Compare June 28, 2023 08:08

Praveen2112 reviewed Jun 28, 2023

View reviewed changes

Praveen2112 approved these changes Jun 30, 2023

View reviewed changes

Praveen2112 reviewed Jun 30, 2023

View reviewed changes

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java Outdated Show resolved Hide resolved

...in/trino-mongodb/src/test/java/io/trino/plugin/mongodb/TestMongoProjectionPushdownPlans.java Outdated Show resolved Hide resolved

Implement dereference pushdown for MongoDB connector

8a77330

Co-authored-by: praveenkrishna <[email protected]> Co-authored-by: Mateusz "Serafin" Gajewski <[email protected]>

krvikash force-pushed the krvikash/mongodb-dereference-pushdown branch from 06b3390 to 8a77330 Compare June 30, 2023 07:46

ebyhr approved these changes Jun 30, 2023

View reviewed changes

Praveen2112 merged commit dd4bcb0 into trinodb:master Jul 4, 2023

github-actions bot added this to the 421 milestone Jul 4, 2023

krvikash deleted the krvikash/mongodb-dereference-pushdown branch July 4, 2023 06:17

colebow mentioned this pull request Jul 5, 2023

Add Trino 421 release notes #18075

Merged

krvikash mentioned this pull request Jul 12, 2023

When selecting both a dereference field and a dotted field with the same name in MongoDB, the dereference field returns NULL #18243

Open

ebyhr mentioned this pull request Apr 12, 2024

Support pushdown on row type column in MongoDB #3765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement dereference pushdown for MongoDB connector #17710

Implement dereference pushdown for MongoDB connector #17710

krvikash commented May 31, 2023 •

edited

Loading

krvikash commented Jun 5, 2023

krvikash commented Jun 6, 2023

Praveen2112 Jun 6, 2023

Praveen2112 Jun 6, 2023

krvikash Jun 8, 2023

Praveen2112 Jun 26, 2023

krvikash Jun 26, 2023

Praveen2112 Jun 26, 2023

krvikash Jun 26, 2023

Praveen2112 commented Jun 28, 2023

krvikash commented Jun 28, 2023

Praveen2112 left a comment

Praveen2112 Jun 28, 2023

krvikash Jun 28, 2023

Praveen2112 Jun 28, 2023

krvikash Jun 28, 2023

krvikash commented Jun 28, 2023

Praveen2112 left a comment

krvikash commented Jun 30, 2023

krvikash commented Jun 30, 2023

krvikash commented Jul 4, 2023

Praveen2112 commented Jul 4, 2023

Implement dereference pushdown for MongoDB connector #17710

Implement dereference pushdown for MongoDB connector #17710

Conversation

krvikash commented May 31, 2023 • edited Loading

Description

Additional context and related issues

Release notes

krvikash commented Jun 5, 2023

krvikash commented Jun 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Praveen2112 commented Jun 28, 2023

krvikash commented Jun 28, 2023

Praveen2112 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krvikash commented Jun 28, 2023

Praveen2112 left a comment

Choose a reason for hiding this comment

krvikash commented Jun 30, 2023

krvikash commented Jun 30, 2023

krvikash commented Jul 4, 2023

Praveen2112 commented Jul 4, 2023

krvikash commented May 31, 2023 •

edited

Loading