planner, executor: index join enhancement #8471

winoros · 2018-11-27T09:52:25Z

What problem does this PR solve?

Originally, we use constant 1 to build fake eq condition when try to build index join. It worked before, but now there're some cases it cannot satisfy. fixes Fail to choose index join in when index has time column. #8244
We want that the case like select * from t, tt where t.a=tt.a and tt.b >= t.b and tt.b < t.b+20 can use tt.b >= t.b and tt.b < t.b+20 to build index join and access the inner table.

What is changed and how it works?

We analyze the join key and filters as the following procedures:

find the join key with index' column. To see that whether it can be used.
find the filters that is in form column eq constant or column in (constant list).
find the filters that is in form of column from inner table compareOp expression only contain outer table's column and not a constant value.
3.1 union first two part, construct the template ranges that will be used in execution phase.
if 3 find nothing. We call ranger to build range for the column.
4.1 union this three part, construct the template ranges that will be used in execution phase.

When in execution phase, if there's re expressions found in 3. We will calculate it, build range for it and append it to the tail of the template range.

Check List

Tests

Unit test // test in planner is added.
Integration test
Manual test

Code changes

Has exported function/method change
Has exported variable/fields change

Side effects

Increased code complexity

Related changes

Need to update the documentation
Need to be included in the release note

This change is

winoros · 2018-11-27T11:28:13Z

/run-common-test
/run-sqllogic-test

winoros · 2018-11-28T08:25:17Z

planner/core/exhaust_physical_plans.go

 	}
+	if len(nextColCmpFilterManager.OpType) == 0 {


We can let the manager hold the constant range too.
But this will lead to this situation: If there's no filter need to be computed at runtime. We can actually set the range information in planner. But if we use the manager hold the constant range, the constant range will be set during execution phase.

But this can save some codes.

Which one do you prefer?

planner/core/exhaust_physical_plans.go

winoros · 2018-11-28T08:27:51Z

planner/core/exhaust_physical_plans.go

-	access, eqConds, remained, keyOff2IdxOff := p.buildFakeEqCondsForIndexJoin(innerJoinKeys, idxCols, colLengths, innerPlan.pushedDownConds)
+// ColWithCompareOps is used in index join to handle the column with compare functions(>=, >, <, <=).
+// It stores the compare functions and build ranges in execution phase.
+type ColWithCompareOps struct {


Maybe we have a better name for this struct.

executor/builder.go

winoros · 2018-11-28T08:29:06Z

executor/index_lookup_join.go

-	indexRanges   []*ranger.Range
-	keyOff2IdxOff []int
+	indexRanges           []*ranger.Range
+	nextColCompareFilters *plannercore.ColWithCompareOps


Maybe we have a better name for this variable.

winoros · 2018-11-28T08:30:05Z

We need to modify the explain result of index join.
The main change now is okay to review.
Unit-test will be added later

winoros · 2018-12-11T09:33:45Z

When adding unit-test, found some incorrect result when index have prefix-string column.
It needs some time to solve it.

winoros · 2018-12-13T11:37:58Z

/run-all-tests

winoros · 2018-12-17T12:29:31Z

UPD: The failed case is caused by the incorrect behavior of MutRow, not the pr itself.

winoros · 2018-12-18T07:25:21Z

The failed case is fixed by #8725

zz-jason

LGTM

alivxxx · 2019-04-28T09:36:52Z

@winoros Please fix the conflicts.

alivxxx · 2019-04-28T11:11:02Z

planner/core/exhaust_physical_plans.go

+	// If the filter contains index column covered by join keys, it will be useless since we always use join key to build range for that index column..
+	for _, innerFilter := range innerPlan.pushedDownConds {
+		affectedCols := expression.ExtractColumns(innerFilter)
+		if expression.ColumnSliceIsIntersect(affectedCols, ijHelper.curPossibleUsedKeys) {


Seems that curPossibleUsedKeys is always empty here because we build them in removeUselessEqAndInFunc, which is after this.

Removed this for loop.

winoros · 2019-04-28T12:29:55Z

./run-all-tests

alivxxx

LGTM

winoros added sig/planner SIG: Planner sig/execution SIG execution labels Nov 27, 2018

planner, executor: index join enhancement

5bdb46f

winoros force-pushed the indexjoin-enhancement branch from 6fd3dbf to 5bdb46f Compare November 27, 2018 09:53

tiny fix

029c6fd

winoros force-pushed the indexjoin-enhancement branch from 7e053bf to 029c6fd Compare November 27, 2018 11:14

winoros added 2 commits November 27, 2018 21:03

fix sqllogic-test

92570d6

fix sqllogic test again

3f4c9b9

winoros force-pushed the indexjoin-enhancement branch from a2685c7 to 3f4c9b9 Compare November 28, 2018 06:01

add comment

dbe54c9

winoros commented Nov 28, 2018

View reviewed changes

fix explain result

519f562

winoros mentioned this pull request Dec 12, 2018

ranger: some code clean up #8663

Merged

winoros added 6 commits December 13, 2018 15:58

fix string method

3a0b74c

Merge branch 'master' into indexjoin-enhancement

a67e026

add unit test and fix things found in it

7d5c4df

Merge branch 'master' into indexjoin-enhancement

ff45b3f

more clear fix and remove unused changes

16196a1

fix unit-test and explain test

d9d51d8

winoros force-pushed the indexjoin-enhancement branch from 4e082a3 to d9d51d8 Compare December 13, 2018 11:31

winoros changed the title ~~[WIP] planner, executor: index join enhancement~~ planner, executor: index join enhancement Dec 13, 2018

winoros mentioned this pull request Feb 12, 2019

range info for the inner child of index join in the "explain" result is not correct #8058

Closed

winoros requested review from zz-jason and eurekaka February 18, 2019 07:29

winoros added 3 commits April 16, 2019 15:02

update to be compatiable with issue#9562

c42ed2c

update logger

ba8c208

add some comment

7631437

winoros force-pushed the indexjoin-enhancement branch from b13e136 to 7631437 Compare April 16, 2019 07:13

fix unit test

dca3ff4

winoros requested review from zz-jason and qw4990 and removed request for zz-jason April 16, 2019 07:32

qw4990 requested review from qw4990 and removed request for qw4990 April 16, 2019 09:07

Merge branch 'master' into indexjoin-enhancement

19a06f3

winoros force-pushed the indexjoin-enhancement branch from 6ff3cc1 to 19a06f3 Compare April 22, 2019 03:40

fix explaintest

bfbd3b4

winoros added the priority/release-blocker This issue blocks a release. Please solve it ASAP. label Apr 28, 2019

zz-jason reviewed Apr 28, 2019

View reviewed changes

zz-jason added the status/LGT1 Indicates that a PR has LGTM 1. label Apr 28, 2019

alivxxx reviewed Apr 28, 2019

View reviewed changes

winoros added 3 commits April 28, 2019 20:20

address comments

662ef3d

Merge branch 'master' into indexjoin-enhancement

1c69103

fix unit test

cba3da9

winoros requested a review from alivxxx April 28, 2019 13:55

alivxxx approved these changes Apr 29, 2019

View reviewed changes

alivxxx added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Apr 29, 2019

alivxxx merged commit 9d74d64 into pingcap:master Apr 29, 2019

winoros deleted the indexjoin-enhancement branch May 7, 2019 08:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner, executor: index join enhancement #8471

planner, executor: index join enhancement #8471

winoros commented Nov 27, 2018 •

edited

Loading

winoros commented Nov 27, 2018

winoros Nov 28, 2018 •

edited

Loading

winoros Nov 28, 2018

winoros Nov 28, 2018

winoros commented Nov 28, 2018 •

edited

Loading

winoros commented Dec 11, 2018

winoros commented Dec 13, 2018

winoros commented Dec 17, 2018

winoros commented Dec 18, 2018

zz-jason left a comment

alivxxx commented Apr 28, 2019

alivxxx Apr 28, 2019

winoros Apr 28, 2019 •

edited

Loading

winoros commented Apr 28, 2019

alivxxx left a comment

planner, executor: index join enhancement #8471

planner, executor: index join enhancement #8471

Conversation

winoros commented Nov 27, 2018 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

winoros commented Nov 27, 2018

winoros Nov 28, 2018 • edited Loading

Choose a reason for hiding this comment

winoros Nov 28, 2018

Choose a reason for hiding this comment

winoros Nov 28, 2018

Choose a reason for hiding this comment

winoros commented Nov 28, 2018 • edited Loading

winoros commented Dec 11, 2018

winoros commented Dec 13, 2018

winoros commented Dec 17, 2018

winoros commented Dec 18, 2018

zz-jason left a comment

Choose a reason for hiding this comment

alivxxx commented Apr 28, 2019

alivxxx Apr 28, 2019

Choose a reason for hiding this comment

winoros Apr 28, 2019 • edited Loading

Choose a reason for hiding this comment

winoros commented Apr 28, 2019

alivxxx left a comment

Choose a reason for hiding this comment

winoros commented Nov 27, 2018 •

edited

Loading

winoros Nov 28, 2018 •

edited

Loading

winoros commented Nov 28, 2018 •

edited

Loading

winoros Apr 28, 2019 •

edited

Loading