Enhancement/optimize edge all predicate #5481

czpmango · 2023-04-06T08:17:23Z

What type of PR is this?

bug
feature
enhancement

What problem(s) does this PR solve?

Push edge all predicate down to the storage layer to optimize performance.

Checklist:

Tests:

Unit test(positive and negative cases)
Function test
Performance test
N/A

Affects:

Documentation affected (Please add the label if documentation needs to be modified.)
Incompatibility (If it breaks the compatibility, please describe it and add the label.）
If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
Performance impacted: Consumes more CPU/Memory

jievince

LGTM

yixinglu · 2023-04-07T01:43:39Z

plz solve the conflicts of this PR

czpmango · 2023-04-07T01:55:47Z

plz solve the conflicts of this PR

Ok.

[Optimizer]Embed edge all predicate into Traverse Move rules fix expr util small change fix compile rename rule Split opt rule Push edge all filter fmt smll fix Add tck fix tck fmt Add tck Add tck Add tck tmp fix fix Fix tck

Shylock-Hg · 2023-04-07T07:42:52Z

src/graph/executor/query/TraverseExecutor.cpp

@@ -62,8 +62,12 @@ Status TraverseExecutor::buildRequestVids() {
    auto vidType = SchemaUtil::propTypeToValueType(metaVidType.get_type());
    for (; iter->valid(); iter->next()) {
      const auto& vid = src->eval(ctx(iter));
-      DCHECK_EQ(vid.type(), vidType)


Why delete this?

Should check why vid type is unexpected.

yixinglu

good job!

maybe you should test some scenarios which could not push the all predicates down, such as:

all( i in e where i.likeness>95 AND true);
all(i in e WHERE i.likeness>95 AND v.player.name =="");
all(i in e WHERE i.likeness>95 OR i.likeness < 50);
...

yixinglu · 2023-04-07T07:56:42Z

src/graph/visitor/ExtractFilterExprVisitor.cpp

+      canBePushed_ = false;
+      return;
+    }
+  }


could you extract these checks into following functions, such as:

bool checkEdgeSchema(const std::string &sym) const; bool checkTagSchema(const std::string &sym) const;

if so, you could be able to use these in following way:

if (!checkTagSchema(expr->sym())) return; if (!checkEdgeSchema(expr->sym()) return;

yixinglu · 2023-04-07T08:02:03Z

tests/tck/features/optimizer/EmbedEdgeAllPredIntoTraverseRule.feature

+      | 15 | Filter         | 11           |                | {"condition": "((n.player.age>0)==true)"} |
+      | 11 | AppendVertices | 14           |                |                                           |
+      | 14 | Filter         | 13           |                | {"condition": "(size($e)>0)"}             |
+      | 13 | Traverse       | 1            |                | {"edge filter": "(*.likeness>90)"}        |


why the predicate is sometimes in edge filter and sometimes in filter fields of Traverse?

0 step filter cannot push to storage layer.

yixinglu · 2023-04-07T08:03:40Z

tests/tck/features/optimizer/EmbedEdgeAllPredIntoTraverseRule.feature

+      | 53 | Filter         | 52           |                | {"condition": "(id($friend)!=\"Tony Parker\")"}                              |
+      | 52 | AppendVertices | 59           |                |                                                                              |
+      | 59 | Filter         | 60           |                | {"condition": "(id($person)==\"Tony Parker\")"}                              |
+      | 60 | Traverse       | 2            |                | {"filter": "((like.likeness>0) AND (like.likeness>1))"}                      |


we could merge the condition into like.likeness>1 later.

Yep. Expression normalization is not optimized yet.

czpmango · 2023-04-07T08:23:03Z

all(i in e WHERE i.likeness>95 AND v.player.name =="");

ACK.

* refactor traverse output (#5464) * refactor traverse output * fix pruneproperties error & none_direct_dst * fix test error * fix shortest path * Change the compaction filter logic to let periodic compaction go through custom compaction filter, to gc expired data (#5447) * Push filter down cross join (#5473) * fix comment * push down filter through cross join --------- Co-authored-by: Sophie <[email protected]> * Fix shortest path crash (#5472) * fix crash of geo (#5475) * fix crash of geo * change log(fatal) to log(error) * fix miss arg $GITHUB_OUTPUT (#5478) * Split optimizer rules (#5470) Fix compile small rename Fix tck Fix tck fmt Fix tck Fix tck * Enhancement/optimize edge all predicate (#5481) * fix eval contains filter on storaged (#5485) * fix eval contains filter on storaged * add tck case * add tck case * fix tck * fix lint * fix lint * Fix expression util function (#5487) fmt Co-authored-by: Sophie <[email protected]> * fix ContainsFilter random fail (#5489) * Fixed graphd startup issue (#5493) * fix prunproperties (#5494) * stop the pushing down of not expressions that are not rewritten to proper forms. (#5502) * Fix edge all predicate with rank function (#5503) Co-authored-by: Sophie <[email protected]> * rewrite param in subgraph & path (#5500) * check param in subgraph * rewrite param in path --------- Co-authored-by: Sophie <[email protected]> * Fix concurrent bug about session count (#5496) * Fix regex expression (#5507) * Update requirements.txt (#5512) Solidified tomli version to solve centos7 compatibility issues * Update cluster id (#5514) --------- Co-authored-by: jimingquan <[email protected]> Co-authored-by: Ryan <[email protected]> Co-authored-by: Yee <[email protected]> Co-authored-by: jie.wang <[email protected]> Co-authored-by: George <[email protected]> Co-authored-by: kyle.cao <[email protected]> Co-authored-by: codesigner <[email protected]> Co-authored-by: dutor <[email protected]> Co-authored-by: Cheng Xuntao <[email protected]> Co-authored-by: Yichen Wang <[email protected]>

czpmango added the ready-for-testing PR: ready for the CI test label Apr 6, 2023

czpmango force-pushed the enhancement/optimize-edge-all-predicate branch 2 times, most recently from 4c5d5b1 to 6c86d3c Compare April 6, 2023 09:49

czpmango marked this pull request as draft April 6, 2023 10:52

jievince previously approved these changes Apr 6, 2023

View reviewed changes

czpmango dismissed jievince’s stale review via f4133f8 April 7, 2023 01:57

czpmango force-pushed the enhancement/optimize-edge-all-predicate branch 2 times, most recently from f4133f8 to 9916f04 Compare April 7, 2023 02:35

czpmango marked this pull request as ready for review April 7, 2023 06:00

czpmango force-pushed the enhancement/optimize-edge-all-predicate branch from bfc7a1c to fe5fb95 Compare April 7, 2023 06:00

[Optimizer]Embed edge all predicate into Traverse

08002ff

[Optimizer]Embed edge all predicate into Traverse Move rules fix expr util small change fix compile rename rule Split opt rule Push edge all filter fmt smll fix Add tck fix tck fmt Add tck Add tck Add tck tmp fix fix Fix tck

czpmango force-pushed the enhancement/optimize-edge-all-predicate branch from d1848d5 to 08002ff Compare April 7, 2023 07:11

czpmango requested review from yixinglu, Shylock-Hg and jievince April 7, 2023 07:18

Shylock-Hg reviewed Apr 7, 2023

View reviewed changes

jievince previously approved these changes Apr 7, 2023

View reviewed changes

yixinglu reviewed Apr 7, 2023

View reviewed changes

Add some test cases

348be3b

czpmango dismissed jievince’s stale review via 348be3b April 7, 2023 08:41

yixinglu approved these changes Apr 7, 2023

View reviewed changes

yixinglu requested a review from jievince April 7, 2023 09:39

jievince approved these changes Apr 7, 2023

View reviewed changes

Sophie-Xie merged commit 0659aa2 into vesoft-inc:master Apr 7, 2023

wey-gu mentioned this pull request Apr 8, 2023

Weekly Report 2023-04-07 vesoft-inc/nebula-community#395

Closed

Sophie-Xie mentioned this pull request Apr 10, 2023

[Optimizer] Optimize variable-length edge predicate execution #5441

Closed

Sophie-Xie pushed a commit that referenced this pull request Apr 13, 2023

Enhancement/optimize edge all predicate (#5481)

313f5eb

Sophie-Xie added the cherry-pick-v3.5 label Apr 13, 2023

Sophie-Xie added the already-picked-3.5 label Apr 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement/optimize edge all predicate #5481

Enhancement/optimize edge all predicate #5481

czpmango commented Apr 6, 2023 •

edited

Loading

jievince left a comment

yixinglu commented Apr 7, 2023

czpmango commented Apr 7, 2023

Shylock-Hg Apr 7, 2023

czpmango Apr 7, 2023

Shylock-Hg Apr 7, 2023

czpmango Apr 7, 2023

yixinglu left a comment

yixinglu Apr 7, 2023

yixinglu Apr 7, 2023

czpmango Apr 7, 2023

yixinglu Apr 7, 2023

czpmango Apr 7, 2023

czpmango commented Apr 7, 2023

Enhancement/optimize edge all predicate #5481

Enhancement/optimize edge all predicate #5481

Conversation

czpmango commented Apr 6, 2023 • edited Loading

What type of PR is this?

What problem(s) does this PR solve?

Checklist:

jievince left a comment

Choose a reason for hiding this comment

yixinglu commented Apr 7, 2023

czpmango commented Apr 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yixinglu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

czpmango commented Apr 7, 2023

czpmango commented Apr 6, 2023 •

edited

Loading