planner: support specify table partition in optimize hint #17638

lzmhhh123 · 2020-06-03T11:46:50Z

What problem does this PR solve?

Issue Number: close #17113

Problem Summary: support specify table parition in optimize hint

What is changed and how it works?

What's Changed: support specify table parition in optimize hint

Related changes

PR to update pingcap/docs/pingcap/docs-cn: after this PR merge, we should update the docs.
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM

Release note

support specify table partition in optimize hint

…on_in_table_hints

lzmhhh123 · 2020-06-03T11:55:39Z

/run-unit-test

codecov · 2020-06-03T12:02:55Z

Codecov Report

Merging #17638 into master will increase coverage by 0.6366%.
The diff coverage is 79.1081%.

@@               Coverage Diff                @@
##             master     #17638        +/-   ##
================================================
+ Coverage   79.4228%   80.0594%   +0.6366%     
================================================
  Files           524        526         +2     
  Lines        142143     145994      +3851     
================================================
+ Hits         112894     116882      +3988     
+ Misses        20101      19969       -132     
+ Partials       9148       9143         -5

…on_in_table_hints

XuHuaiyu · 2020-06-05T05:50:41Z

planner/core/logical_plan_builder.go

-	if hintInfo.ifPreferTiFlash(alias) {
-		if ds.preferStoreType != 0 {
+	if hintTbl := hintInfo.ifPreferTiFlash(alias); hintTbl != nil {
+		if ds.preferStoreType != 0 && ds.tableInfo.Partition == nil {


Add a comment for why we do not raise an error when ds.tableInfo.Partition != nil

XuHuaiyu · 2020-06-05T05:56:36Z

planner/core/logical_plans.go

@@ -502,7 +503,8 @@ type DataSource struct {
 	// it is converted from statisticTable, and used for IO/network cost estimating.
 	TblColHists *statistics.HistColl
 	//preferStoreType means the DataSource is enforced to which storage.
-	preferStoreType int
+	preferStoreType           int
+	preferStoreTypePartitions map[int][]model.CIStr


preferPartitions is enough?

Add a comment for why this is a map

XuHuaiyu · 2020-06-05T06:03:22Z

planner/core/planbuilder.go

+		case TiDBMergeJoin, HintSMJ, TiDBIndexNestedLoopJoin, HintINLJ, HintINLHJ, HintINLMJ, TiDBHashJoin, HintHJ:
+			if len(tableInfo.partitions) > 0 {
+				ctx.GetSessionVars().StmtCtx.AppendWarning(
+					errors.New(fmt.Sprintf("%s hint do not support specify partitions", hintName)))


Do we need to invoke restore2StorageHint here?

Do we need to extend restore2StorageHint?

The error message can be refined to Optimizer Hint %s is inapplicable on specified partitions

XuHuaiyu

Do we need to raise a warning for sql like if the partition is unknown in the hint:
select /*+ use_index(t partition(p_non_exist)) */ from t partition(p1,p2)

planner/core/testdata/integration_suite_in.json

lzmhhh123 · 2020-06-05T07:56:56Z

Do we need to raise a warning for sql like if the partition is unknown in the hint:
select /*+ use_index(t partition(p_non_exist)) */ from t partition(p1,p2)

OK, I try to support it.

lzmhhh123 · 2020-06-05T08:41:18Z

/rebuild

….com/lzmhhh123/tidb into dev/support_partition_in_table_hints

XuHuaiyu · 2020-06-05T08:37:17Z

planner/core/planbuilder.go

+		if len(table.partitions) > 0 {
+			buffer.WriteString(" PARTITION(")
+			for j, partition := range table.partitions {
+				buffer.WriteString(partition.L)


This should be moved to line 262

XuHuaiyu · 2020-06-05T08:48:31Z

planner/core/rule_partition_processor.go

+	for _, p := range partitions {
+		if !partitionSet.Exist(p.L) {
+			ds.ctx.GetSessionVars().StmtCtx.AppendWarning(
+				errors.New(fmt.Sprintf("partition `%s` in hint can't hit any source", p.L)))


Can we return the partition name back to checkHintsApplicable, and build a more comprehensible error message like Unknow partition %s in optimizer hint %s

lzmhhh123 · 2020-06-05T09:29:03Z

/run-unit-test

tiancaiamao · 2020-06-18T15:27:18Z

LGTM

…on_in_table_hints

tiancaiamao · 2020-06-19T14:41:33Z

@XuHuaiyu

XuHuaiyu

LGTM

XuHuaiyu · 2020-06-22T03:07:06Z

/merge

ti-srebot · 2020-06-22T03:07:09Z

Sorry @XuHuaiyu, you don't have permission to trigger auto merge event on this branch. You are not a committer for the related sigs:planner(slack).

XuHuaiyu · 2020-06-22T03:07:18Z

/run-all-tests

Signed-off-by: ti-srebot <[email protected]>

ti-srebot · 2020-06-22T03:22:22Z

cherry pick to release-4.0 in PR #18157

…18157)

planner: support specify table parition in optimize hint

7922bf1

lzmhhh123 requested review from tiancaiamao, eurekaka, winoros, XuHuaiyu and zz-jason June 3, 2020 11:47

lzmhhh123 added 3 commits June 3, 2020 19:48

fix error check

3d745c6

Merge remote-tracking branch 'origin/master' into dev/support_partiti…

5b63ef6

…on_in_table_hints

fix fmt

2c18291

lzmhhh123 added sig/planner SIG: Planner needs-cherry-pick-4.0 labels Jun 3, 2020

lzmhhh123 added this to the v4.0.1 milestone Jun 5, 2020

Merge remote-tracking branch 'origin/master' into dev/support_partiti…

6985351

…on_in_table_hints

XuHuaiyu reviewed Jun 5, 2020

View reviewed changes

planner/core/testdata/integration_suite_in.json Show resolved Hide resolved

lzmhhh123 added 3 commits June 5, 2020 16:21

address comments

99bbad5

fix warning for join hint

6eacc8c

Merge branch 'master' into dev/support_partition_in_table_hints

839a9bb

lzmhhh123 added 2 commits June 5, 2020 16:44

fix fmt

36a48aa

Merge branch 'dev/support_partition_in_table_hints' of https://github…

20d4b01

….com/lzmhhh123/tidb into dev/support_partition_in_table_hints

XuHuaiyu reviewed Jun 5, 2020

View reviewed changes

lzmhhh123 added 2 commits June 5, 2020 17:15

address comments

30cac51

fix fmt

23b2567

lzmhhh123 removed request for tiancaiamao and zz-jason June 5, 2020 09:34

bb7133 modified the milestones: v4.0.1, v4.0.2 Jun 6, 2020

tiancaiamao added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 18, 2020

Merge remote-tracking branch 'origin/master' into dev/support_partiti…

17ce498

…on_in_table_hints

zz-jason modified the milestones: v4.0.2, v4.0.3 Jun 19, 2020

lzmhhh123 requested a review from a team as a code owner June 19, 2020 09:46

zz-jason modified the milestones: v4.0.3, v5.0-alpha Jun 19, 2020

XuHuaiyu approved these changes Jun 22, 2020

View reviewed changes

XuHuaiyu modified the milestones: v5.0-alpha, v4.0.3 Jun 22, 2020

Merge branch 'master' into dev/support_partition_in_table_hints

8589254

XuHuaiyu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 22, 2020

lzmhhh123 merged commit 3e5db05 into pingcap:master Jun 22, 2020

lzmhhh123 deleted the dev/support_partition_in_table_hints branch June 22, 2020 03:21

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jun 22, 2020

cherry pick pingcap#17638 to release-4.0

b206c66

Signed-off-by: ti-srebot <[email protected]>

ti-srebot mentioned this pull request Jun 22, 2020

planner: support specify table partition in optimize hint (#17638) #18157

Merged

SunRunAway pushed a commit that referenced this pull request Sep 9, 2020

planner: support specify table partition in optimize hint (#17638) (#…

88f98fc

…18157)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: support specify table partition in optimize hint #17638

planner: support specify table partition in optimize hint #17638

lzmhhh123 commented Jun 3, 2020

lzmhhh123 commented Jun 3, 2020

codecov bot commented Jun 3, 2020 •

edited

Loading

XuHuaiyu Jun 5, 2020

XuHuaiyu Jun 5, 2020

XuHuaiyu Jun 5, 2020

XuHuaiyu Jun 5, 2020 •

edited

Loading

XuHuaiyu left a comment

lzmhhh123 commented Jun 5, 2020

lzmhhh123 commented Jun 5, 2020

XuHuaiyu Jun 5, 2020

XuHuaiyu Jun 5, 2020

lzmhhh123 commented Jun 5, 2020

tiancaiamao commented Jun 18, 2020

tiancaiamao commented Jun 19, 2020

XuHuaiyu left a comment

XuHuaiyu commented Jun 22, 2020

ti-srebot commented Jun 22, 2020

XuHuaiyu commented Jun 22, 2020

ti-srebot commented Jun 22, 2020

planner: support specify table partition in optimize hint #17638

planner: support specify table partition in optimize hint #17638

Conversation

lzmhhh123 commented Jun 3, 2020

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

lzmhhh123 commented Jun 3, 2020

codecov bot commented Jun 3, 2020 • edited Loading

Codecov Report

XuHuaiyu Jun 5, 2020

Choose a reason for hiding this comment

XuHuaiyu Jun 5, 2020

Choose a reason for hiding this comment

XuHuaiyu Jun 5, 2020

Choose a reason for hiding this comment

XuHuaiyu Jun 5, 2020 • edited Loading

Choose a reason for hiding this comment

XuHuaiyu left a comment

Choose a reason for hiding this comment

lzmhhh123 commented Jun 5, 2020

lzmhhh123 commented Jun 5, 2020

XuHuaiyu Jun 5, 2020

Choose a reason for hiding this comment

XuHuaiyu Jun 5, 2020

Choose a reason for hiding this comment

lzmhhh123 commented Jun 5, 2020

tiancaiamao commented Jun 18, 2020

tiancaiamao commented Jun 19, 2020

XuHuaiyu left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Jun 22, 2020

ti-srebot commented Jun 22, 2020

XuHuaiyu commented Jun 22, 2020

ti-srebot commented Jun 22, 2020

codecov bot commented Jun 3, 2020 •

edited

Loading

XuHuaiyu Jun 5, 2020 •

edited

Loading