Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optimizer rule for pushing limit into the fulltext index scan #5575

Merged
merged 7 commits into from
Jun 6, 2023

Conversation

yixinglu
Copy link
Contributor

@yixinglu yixinglu commented Jun 5, 2023

What type of PR is this?

  • bug
  • feature
  • enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:

Support the optimization for pushing the limit into the fulltext index scan, such as:

      LOOKUP ON tag2 WHERE prefix(tag2.prop2,"nebula")
      YIELD id(vertex) as id, tag2.prop1 as prop1, tag2.prop2 as prop2 |
      LIMIT 1,1
Execution Plan (optimize time 2609 us)

-----+-------------------+--------------+----------------+--------------------------------------------------
| id | name              | dependencies | profiling data | operator info                                   |
-----+-------------------+--------------+----------------+--------------------------------------------------
|  8 | Project           | 9            |                | outputVar: {                                    |
|    |                   |              |                |   "colNames": [                                 |
|    |                   |              |                |     "id",                                       |
|    |                   |              |                |     "prop1",                                    |
|    |                   |              |                |     "prop2"                                     |
|    |                   |              |                |   ],                                            |
|    |                   |              |                |   "name": "__Limit_4",                          |
|    |                   |              |                |   "type": "DATASET"                             |
|    |                   |              |                | }                                               |
|    |                   |              |                | inputVar: __Limit_7                             |
|    |                   |              |                | columns: [                                      |
|    |                   |              |                |   "id(VERTEX) AS id",                           |
|    |                   |              |                |   "tag2.prop1 AS prop1",                        |
|    |                   |              |                |   "tag2.prop2 AS prop2"                         |
|    |                   |              |                | ]                                               |
-----+-------------------+--------------+----------------+--------------------------------------------------
|  9 | Limit             | 11           |                | outputVar: {                                    |
|    |                   |              |                |   "colNames": [],                               |
|    |                   |              |                |   "name": "__Limit_7",                          |
|    |                   |              |                |   "type": "DATASET"                             |
|    |                   |              |                | }                                               |
|    |                   |              |                | inputVar: __GetVertices_10                      |
|    |                   |              |                | offset: 1                                       |
|    |                   |              |                | count: 1                                        |
-----+-------------------+--------------+----------------+--------------------------------------------------
| 11 | GetVertices       | 12           |                | outputVar: {                                    |
|    |                   |              |                |   "colNames": [],                               |
|    |                   |              |                |   "type": "DATASET",                            |
|    |                   |              |                |   "name": "__GetVertices_10"                    |
|    |                   |              |                | }                                               |
|    |                   |              |                | inputVar: __FulltextIndexScan_12                |
|    |                   |              |                | space: 6                                        |
|    |                   |              |                | dedup: 0                                        |
|    |                   |              |                | limit: 2                                        |
|    |                   |              |                | filter:                                         |
|    |                   |              |                | orderBy: []                                     |
|    |                   |              |                | src: COLUMN[0]                                  |
|    |                   |              |                | props: [                                        |
|    |                   |              |                |   {                                             |
|    |                   |              |                |     "tagId": 7,                                 |
|    |                   |              |                |     "props": [                                  |
|    |                   |              |                |       "_vid",                                   |
|    |                   |              |                |       "_tag",                                   |
|    |                   |              |                |       "prop1",                                  |
|    |                   |              |                |       "prop2"                                   |
|    |                   |              |                |     ]                                           |
|    |                   |              |                |   }                                             |
|    |                   |              |                | ]                                               |
|    |                   |              |                | exprs:                                          |
-----+-------------------+--------------+----------------+--------------------------------------------------
| 12 | FulltextIndexScan | 0            |                | outputVar: {                                    |
|    |                   |              |                |   "colNames": [],                               |
|    |                   |              |                |   "name": "__FulltextIndexScan_12",             |
|    |                   |              |                |   "type": "DATASET"                             |
|    |                   |              |                | }                                               |
|    |                   |              |                | inputVar:                                       |
|    |                   |              |                | space: 0                                        |
|    |                   |              |                | dedup: 0                                        |
|    |                   |              |                | limit: 2                                        |
|    |                   |              |                | filter:                                         |
|    |                   |              |                | orderBy: []                                     |
|    |                   |              |                | isEdge: 0                                       |
|    |                   |              |                | searchExpr: PREFIX(tag2.prop2, "nebula", 10000) |
|    |                   |              |                | index: nebula_index_tag2_prop2                  |
-----+-------------------+--------------+----------------+--------------------------------------------------
|  0 | Start             |              |                | outputVar: {                                    |
|    |                   |              |                |   "colNames": [],                               |
|    |                   |              |                |   "name": "__Start_0",                          |
|    |                   |              |                |   "type": "DATASET"                             |
|    |                   |              |                | }                                               |
-----+-------------------+--------------+----------------+--------------------------------------------------

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

  • Unit test(positive and negative cases)
  • Function test
  • Performance test
  • N/A

Affects:

  • Documentation affected (Please add the label if documentation needs to be modified.)
  • Incompatibility (If it breaks the compatibility, please describe it and add the label.)
  • If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
  • Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

@yixinglu yixinglu added the ready-for-testing PR: ready for the CI test label Jun 5, 2023
@yixinglu yixinglu mentioned this pull request Jun 5, 2023
11 tasks
@yixinglu yixinglu marked this pull request as ready for review June 6, 2023 06:06
@cangfengzhs
Copy link
Contributor

How to support offset? I don't find it in operator info

@yixinglu
Copy link
Contributor Author

yixinglu commented Jun 6, 2023

offset has been supported in LIMIT sentence, such as LIMIT 1, 2, FulltextIndexScan only has limit field and need not to keep the offset which is in Limit plan node.

@yixinglu
Copy link
Contributor Author

yixinglu commented Jun 6, 2023

pls hold on this PR, i need to add more rules to support offset in fulltextIndexScan

@yixinglu yixinglu merged commit 88dde49 into vesoft-inc:master Jun 6, 2023
@yixinglu yixinglu deleted the es-ft branch June 6, 2023 07:10
yixinglu added a commit to yixinglu/nebula that referenced this pull request Sep 14, 2023
* Ft search (vesoft-inc#5567)

* Add optimizer rule for pushing limit into the fulltext index scan (vesoft-inc#5575)

* Merge limit and full text index scan rule (vesoft-inc#5577)

* Support to return score from es query (vesoft-inc#5580)

* modify ft search (vesoft-inc#5584)

* modify ft search

* fix bug

* address some comment

* fix bug

* fix conflict

---------

Co-authored-by: hs.zhang <[email protected]>
Co-authored-by: Yee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-for-testing PR: ready for the CI test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants