feat: support clean read base on fast mode #5371

hongyunyan · 2022-07-14T12:05:04Z

What problem does this PR solve?

Issue Number: ref #5252

Problem Summary:

What is changed and how it works?

enable clean read(optimization handle column) on fast mode.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

…yunyan_fast_mode_non_schema_part

…lean_read_base_on_fast_mode

…hub.com/hongyunyan/tiflash into hongyunyan_clean_read_base_on_fast_mode

…lean_read_base_on_fast_mode

ti-chi-bot · 2022-07-14T12:05:07Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

hongyunyan · 2022-07-15T00:59:36Z

/run-all-tests

hongyunyan · 2022-07-15T01:03:03Z

/rebuild

hongyunyan · 2022-07-15T01:12:10Z

/run-all-tests

sre-bot · 2022-07-15T01:23:35Z

Coverage for changed files

Filename                                               Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DMDecoratorStreams.h                                        48                 8    83.33%          13                 6    53.85%          98                20    79.59%          22                 3    86.36%
File/DMFileBlockInputStream.cpp                             12                 2    83.33%           2                 0   100.00%          45                 6    86.67%          12                 3    75.00%
File/DMFileBlockInputStream.h                               16                 1    93.75%          16                 1    93.75%          54                 1    98.15%           0                 0         -
File/DMFileReader.cpp                                      227                14    93.83%          12                 0   100.00%         445                27    93.93%         162                19    88.27%
File/DMFileReader.h                                          9                 0   100.00%           3                 0   100.00%           7                 0   100.00%           4                 0   100.00%
Segment.cpp                                                689               210    69.52%          47                 3    93.62%        1169               263    77.50%         366               192    47.54%
StableValueSpace.cpp                                       107                25    76.64%          15                 1    93.33%         277                66    76.17%          74                25    66.22%
StableValueSpace.h                                          18                 9    50.00%          14                 5    64.29%          35                26    25.71%           4                 4     0.00%
tests/gtest_dm_delta_merge_store_for_fast_mode.cpp        1544               325    78.95%          12                 0   100.00%        1366                 5    99.63%         734               192    73.84%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                     2670               594    77.75%         134                16    88.06%        3496               414    88.16%        1378               438    68.21%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18556      9572             48.42%    210528  96354        54.23%

full coverage report (for internal network access only)

hongyunyan · 2022-07-15T01:40:07Z

@lidezhu @JaySon-Huang

lidezhu

LGTM

JaySon-Huang

Bascially LGTM

dbms/src/Storages/DeltaMerge/Segment.cpp

dbms/src/Storages/DeltaMerge/File/DMFileReader.cpp

dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store_for_fast_mode.cpp

JaySon-Huang

LGTM

dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store_for_fast_mode.cpp

…for_fast_mode.cpp

JaySon-Huang · 2022-07-15T10:35:25Z

/merge

ti-chi-bot · 2022-07-15T10:35:25Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-07-15T10:35:27Z

This pull request has been accepted and is ready to merge.

Commit hash: 9746854

ti-chi-bot · 2022-07-15T10:49:15Z

@hongyunyan: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-07-15T11:09:59Z

Coverage for changed files

Filename                                               Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DMDecoratorStreams.h                                        48                 8    83.33%          13                 6    53.85%          98                20    79.59%          22                 3    86.36%
File/DMFileBlockInputStream.cpp                             12                 2    83.33%           2                 0   100.00%          45                 6    86.67%          12                 3    75.00%
File/DMFileBlockInputStream.h                               16                 1    93.75%          16                 1    93.75%          54                 1    98.15%           0                 0         -
File/DMFileReader.cpp                                      227                14    93.83%          12                 0   100.00%         445                27    93.93%         162                19    88.27%
File/DMFileReader.h                                          9                 0   100.00%           3                 0   100.00%           7                 0   100.00%           4                 0   100.00%
Segment.cpp                                                689               210    69.52%          47                 3    93.62%        1169               263    77.50%         366               192    47.54%
StableValueSpace.cpp                                       107                25    76.64%          15                 1    93.33%         277                66    76.17%          74                25    66.22%
StableValueSpace.h                                          18                 9    50.00%          14                 5    64.29%          35                26    25.71%           4                 4     0.00%
tests/gtest_dm_delta_merge_store_for_fast_mode.cpp        1531               325    78.77%          12                 0   100.00%        1394                 5    99.64%         734               192    73.84%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                     2657               594    77.64%         134                16    88.06%        3524               414    88.25%        1378               438    68.21%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18593      9573             48.51%    211774  96395        54.48%

full coverage report (for internal network access only)

hongyunyan added 30 commits July 5, 2022 16:42

add fast mode logic

b29ee6b

fix code format

c60185e

Merge branch 'master' of https://github.com/pingcap/tiflash into hong…

14266f7

…yunyan_fast_mode_non_schema_part

compatible old code

0dff68b

fix bug

3039aab

Merge branch 'master' into hongyunyan_fast_mode_non_schema_part

732ab10

fix test

7898ae8

add test for fast mode

e8c52be

Merge branch 'master' into hongyunyan_fast_mode_non_schema_part

67aed53

fix format

e01ad7c

support dtworkload in fast mode

377734f

remove useless code

acb5080

fix delete column error

94a3bc6

support clean read in fast mode

506bc15

fix comments

49b6cf9

update fast mode

5a8fd0d

Merge branch 'hongyunyan_fast_mode_non_schema_part' into hongyunyan_c…

29cf500

…lean_read_base_on_fast_mode

for test

d23e0e8

Merge branch 'hongyunyan_clean_read_base_on_fast_mode' of https://git…

ef60b22

…hub.com/hongyunyan/tiflash into hongyunyan_clean_read_base_on_fast_mode

include delele handle

191ac02

support handle optimization

47a053b

hongyunyan_clean_read_base_on_fast_mode

f641530

fix comment

6648b68

Merge branch 'hongyunyan_fast_mode_non_schema_part' into hongyunyan_c…

3d64809

…lean_read_base_on_fast_mode

fix bug

1187896

merge master

391ebd9

fix bug

97e7191

merge conflict

f2ce510

merge master

dd56b02

add test

15ea2f4

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 14, 2022

hongyunyan added 2 commits July 14, 2022 20:06

fix format

620060f

change name

d48eed7

lidezhu approved these changes Jul 15, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 15, 2022

JaySon-Huang reviewed Jul 15, 2022

View reviewed changes

dbms/src/Storages/DeltaMerge/Segment.cpp Show resolved Hide resolved

dbms/src/Storages/DeltaMerge/File/DMFileReader.cpp Outdated Show resolved Hide resolved

dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store_for_fast_mode.cpp Show resolved Hide resolved

fix comments

242a4f8

JaySon-Huang approved these changes Jul 15, 2022

View reviewed changes

dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store_for_fast_mode.cpp Outdated Show resolved Hide resolved

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 15, 2022

Update dbms/src/Storages/DeltaMerge/tests/gtest_dm_delta_merge_store_…

9746854

…for_fast_mode.cpp

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 15, 2022

ti-chi-bot added 2 commits July 15, 2022 18:35

Merge branch 'master' into hongyunyan_clean_read_base_on_fast_mode

cf8d717

Merge branch 'master' into hongyunyan_clean_read_base_on_fast_mode

8d14cbf

ti-chi-bot merged commit f9c001c into pingcap:master Jul 15, 2022

JaySon-Huang mentioned this pull request Jul 22, 2022

Simplify comparing the result from inputstream #5452

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support clean read base on fast mode #5371

feat: support clean read base on fast mode #5371

hongyunyan commented Jul 14, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Jul 14, 2022 •

edited

Loading

hongyunyan commented Jul 15, 2022

hongyunyan commented Jul 15, 2022

hongyunyan commented Jul 15, 2022

sre-bot commented Jul 15, 2022

hongyunyan commented Jul 15, 2022

lidezhu left a comment

JaySon-Huang left a comment

JaySon-Huang left a comment

JaySon-Huang commented Jul 15, 2022

ti-chi-bot commented Jul 15, 2022

ti-chi-bot commented Jul 15, 2022

ti-chi-bot commented Jul 15, 2022

sre-bot commented Jul 15, 2022

feat: support clean read base on fast mode #5371

feat: support clean read base on fast mode #5371

Conversation

hongyunyan commented Jul 14, 2022 • edited by JaySon-Huang Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jul 14, 2022 • edited Loading

hongyunyan commented Jul 15, 2022

hongyunyan commented Jul 15, 2022

hongyunyan commented Jul 15, 2022

sre-bot commented Jul 15, 2022

Coverage for changed files

Coverage summary

hongyunyan commented Jul 15, 2022

lidezhu left a comment

Choose a reason for hiding this comment

JaySon-Huang left a comment

Choose a reason for hiding this comment

JaySon-Huang left a comment

Choose a reason for hiding this comment

JaySon-Huang commented Jul 15, 2022

ti-chi-bot commented Jul 15, 2022

ti-chi-bot commented Jul 15, 2022

ti-chi-bot commented Jul 15, 2022

sre-bot commented Jul 15, 2022

Coverage for changed files

Coverage summary

hongyunyan commented Jul 14, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Jul 14, 2022 •

edited

Loading