Test: Refine dbgFuncCoprocessor #6088

ywqzzy · 2022-10-08T09:23:40Z

What problem does this PR solve?

Issue Number: ref #4609

Problem Summary:

What is changed and how it works?

Clean up dbgFuncCoprocessor, make it only contains API for DBGInvoker.
Add dbgFuncCoprocessorUtils.
Separate code in dbgFuncCoprocessor into dbgQueryCompiler, which convert raw query into DAGRequests.
Separate code in dbgFuncCoprocessor into dbgQueryExecutor, which take DAGRequests as input then convert them into BlockInputStreams or SelectResponse.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ti-chi-bot · 2022-10-08T09:23:41Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Lloyd-Pottiger
SeaRise

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ywqzzy · 2022-10-08T09:23:51Z

/run-all-tests

ywqzzy · 2022-10-08T09:40:43Z

/run-all-tests

sre-bot · 2022-10-08T09:42:52Z

Coverage for changed files

Filename                                              Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Columns/ColumnUtils.cpp                                     7                 7     0.00%           1                 1     0.00%          13                13     0.00%           4                 4     0.00%
Core/BlockUtils.cpp                                        23                23     0.00%           2                 2     0.00%          48                48     0.00%          14                14     0.00%
DataStreams/UniqRawResReformatBlockOutputStream.h          17                17     0.00%           4                 4     0.00%          33                33     0.00%           8                 8     0.00%
Debug/dbgFuncCoprocessor.cpp                               52                52     0.00%           6                 6     0.00%         117               117     0.00%          36                36     0.00%
Debug/dbgFuncCoprocessorUtils.cpp                         101                92     8.91%          10                 7    30.00%         164               140    14.63%          56                49    12.50%
Debug/dbgQueryCompiler.cpp                                148                74    50.00%           9                 2    77.78%         289               124    57.09%         116                66    43.10%
Debug/dbgQueryCompiler.h                                   25                 2    92.00%           4                 0   100.00%          49                 2    95.92%          14                 2    85.71%
Debug/dbgQueryExecutor.cpp                                 69                56    18.84%          11                 6    45.45%         224               140    37.50%          48                40    16.67%
Server/MockComputeClient.h                                  8                 2    75.00%           3                 0   100.00%          20                 6    70.00%           4                 2    50.00%
Storages/tests/gtest_filter_parser.cpp                    202                44    78.22%          12                 0   100.00%         402                 2    99.50%          34                26    23.53%
TestUtils/ExecutorTestUtils.h                               3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/MPPTaskTestUtils.cpp                             37                 5    86.49%          13                 2    84.62%         115                19    83.48%          18                 3    83.33%
TestUtils/MPPTaskTestUtils.h                                6                 1    83.33%           3                 0   100.00%          13                 3    76.92%           2                 1    50.00%
TestUtils/mockExecutor.cpp                                 74                 3    95.95%          44                 1    97.73%         290                24    91.72%          34                 2    94.12%
TestUtils/mockExecutor.h                                   14                 4    71.43%          14                 4    71.43%          32                 6    81.25%           0                 0         -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                     786               383    51.27%         139                36    74.10%        1827               690    62.23%         388               253    34.79%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18393      7489             59.28%    215458  77198        64.17%

full coverage report (for internal network access only)

sre-bot · 2022-10-08T10:00:57Z

Coverage for changed files

Filename                                              Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Columns/ColumnUtils.cpp                                     7                 7     0.00%           1                 1     0.00%          13                13     0.00%           4                 4     0.00%
Core/BlockUtils.cpp                                        23                23     0.00%           2                 2     0.00%          48                48     0.00%          14                14     0.00%
DataStreams/UniqRawResReformatBlockOutputStream.h          17                17     0.00%           4                 4     0.00%          33                33     0.00%           8                 8     0.00%
Debug/dbgFuncCoprocessor.cpp                               52                52     0.00%           6                 6     0.00%         117               117     0.00%          36                36     0.00%
Debug/dbgFuncCoprocessorUtils.cpp                          55                46    16.36%           8                 5    37.50%         115                91    20.87%          46                39    15.22%
Debug/dbgQueryCompiler.cpp                                148                74    50.00%           9                 2    77.78%         289               124    57.09%         116                66    43.10%
Debug/dbgQueryCompiler.h                                   25                 2    92.00%           4                 0   100.00%          49                 2    95.92%          14                 2    85.71%
Debug/dbgQueryExecutor.cpp                                115               102    11.30%          13                 8    38.46%         273               189    30.77%          58                50    13.79%
Server/MockComputeClient.h                                  8                 2    75.00%           3                 0   100.00%          20                 6    70.00%           4                 2    50.00%
Storages/tests/gtest_filter_parser.cpp                    202                44    78.22%          12                 0   100.00%         402                 2    99.50%          34                26    23.53%
TestUtils/ExecutorTestUtils.h                               3                 1    66.67%           3                 1    66.67%          18                13    27.78%           0                 0         -
TestUtils/MPPTaskTestUtils.cpp                             37                 5    86.49%          13                 2    84.62%         115                19    83.48%          18                 3    83.33%
TestUtils/MPPTaskTestUtils.h                                6                 1    83.33%           3                 0   100.00%          13                 3    76.92%           2                 1    50.00%
TestUtils/mockExecutor.cpp                                 74                 3    95.95%          44                 1    97.73%         290                24    91.72%          34                 2    94.12%
TestUtils/mockExecutor.h                                   14                 4    71.43%          14                 4    71.43%          32                 6    81.25%           0                 0         -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                     786               383    51.27%         139                36    74.10%        1827               690    62.23%         388               253    34.79%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18393      7487             59.29%    215458  77168        64.18%

full coverage report (for internal network access only)

dbms/src/Core/BlockUtils.cpp

dbms/src/Debug/dbgFuncCoprocessorUtils.cpp

dbms/src/Core/BlockUtils.cpp

dbms/src/Debug/dbgFuncCoprocessorUtils.cpp

…ne_dbgFuncCoprocessor

JaySon-Huang · 2022-10-12T06:18:14Z

dbms/src/Core/BlockUtils.cpp

+    return true;
+}
+
+String formatBlockData(const Block & block)


I think getColumnsContent do the same jobs as this function

tiflash/dbms/src/TestUtils/FunctionTestUtils.cpp

Lines 445 to 449 in e2cde35

String getColumnsContent(const ColumnsWithTypeAndName & cols, size_t begin, size_t end)

{

const size_t col_num = cols.size();

if (col_num <= 0)

return "";

JaySon-Huang · 2022-10-12T06:21:27Z

dbms/src/Core/BlockUtils.cpp

+
+namespace DB
+{
+bool blockEqual(const Block & expected, const Block & actual, String & unequal_msg)


It has a "gtest-specified" version in FunctionTestUtils.cpp 😂

It has a "gtest-specified" version in FunctionTestUtils.cpp 😂

I will not modify the code, since dbgNaturalDag that use the utils is not used for now.

dbms/src/Core/BlockUtils.cpp

ywqzzy · 2022-10-25T07:56:51Z

/run-all-tests

ywqzzy · 2022-10-25T09:00:30Z

/run-all-tests

…ne_dbgFuncCoprocessor

ywqzzy · 2022-10-25T10:08:13Z

/run-all-tests

ywqzzy · 2022-10-26T02:38:18Z

/run-all-tests

ywqzzy · 2022-10-26T03:09:31Z

/merge

ti-chi-bot · 2022-10-26T03:09:32Z

@ywqzzy: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-10-26T03:09:34Z

This pull request has been accepted and is ready to merge.

Commit hash: 19c3b3a

SeaRise · 2022-10-31T06:15:19Z

/run-build-release comment=true

ywqzzy added 3 commits October 8, 2022 14:56

refine.

f7f6b3f

clean up.

d2ca2f0

split dbgfuncCoprocessor

4510c30

refine.

40909ab

ti-chi-bot removed the do-not-merge/needs-linked-issue label Oct 8, 2022

refine.

fbf072c

ywqzzy marked this pull request as ready for review October 8, 2022 10:17

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 8, 2022

Lloyd-Pottiger reviewed Oct 9, 2022

View reviewed changes

dbms/src/Core/BlockUtils.cpp Outdated Show resolved Hide resolved

dbms/src/Debug/dbgFuncCoprocessorUtils.cpp Outdated Show resolved Hide resolved

address comments

951a8f3

ywqzzy force-pushed the refine_dbgFuncCoprocessor branch from f433a06 to 951a8f3 Compare October 9, 2022 03:42

u

915c5c4

ywqzzy requested a review from Lloyd-Pottiger October 9, 2022 03:47

Lloyd-Pottiger reviewed Oct 9, 2022

View reviewed changes

dbms/src/Core/BlockUtils.cpp Outdated Show resolved Hide resolved

dbms/src/Debug/dbgFuncCoprocessorUtils.cpp Show resolved Hide resolved

dbms/src/Debug/dbgFuncCoprocessorUtils.cpp Outdated Show resolved Hide resolved

ywqzzy added 2 commits October 11, 2022 19:29

Merge branch 'master' of https://github.com/pingcap/tiflash into refi…

9934882

…ne_dbgFuncCoprocessor

fix

173a78b

JaySon-Huang reviewed Oct 12, 2022

View reviewed changes

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 12, 2022

SeaRise self-requested a review October 25, 2022 06:19

SeaRise approved these changes Oct 25, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 25, 2022

Lloyd-Pottiger approved these changes Oct 25, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 25, 2022

Merge branch 'master' of https://github.com/pingcap/tiflash into refi…

85f0ee3

…ne_dbgFuncCoprocessor

ywqzzy added 2 commits October 26, 2022 10:31

u

d46d377

fix.

c6ab143

ywqzzy force-pushed the refine_dbgFuncCoprocessor branch from 4f5d608 to c6ab143 Compare October 26, 2022 02:38

Merge branch 'master' into refine_dbgFuncCoprocessor

19c3b3a

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Oct 26, 2022

ti-chi-bot merged commit 8ae3dd7 into pingcap:master Oct 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test: Refine dbgFuncCoprocessor #6088

Test: Refine dbgFuncCoprocessor #6088

ywqzzy commented Oct 8, 2022 •

edited

Loading

ti-chi-bot commented Oct 8, 2022 •

edited

Loading

ywqzzy commented Oct 8, 2022

ywqzzy commented Oct 8, 2022

sre-bot commented Oct 8, 2022

sre-bot commented Oct 8, 2022

JaySon-Huang Oct 12, 2022

JaySon-Huang Oct 12, 2022

ywqzzy Oct 25, 2022

ywqzzy commented Oct 25, 2022

ywqzzy commented Oct 25, 2022

ywqzzy commented Oct 25, 2022

ywqzzy commented Oct 26, 2022

ywqzzy commented Oct 26, 2022

ti-chi-bot commented Oct 26, 2022

ti-chi-bot commented Oct 26, 2022

SeaRise commented Oct 31, 2022

	String getColumnsContent(const ColumnsWithTypeAndName & cols, size_t begin, size_t end)
	{
	const size_t col_num = cols.size();
	if (col_num <= 0)
	return "";

Test: Refine dbgFuncCoprocessor #6088

Test: Refine dbgFuncCoprocessor #6088

Conversation

ywqzzy commented Oct 8, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Oct 8, 2022 • edited Loading

ywqzzy commented Oct 8, 2022

ywqzzy commented Oct 8, 2022

sre-bot commented Oct 8, 2022

Coverage for changed files

Coverage summary

sre-bot commented Oct 8, 2022

Coverage for changed files

Coverage summary

JaySon-Huang Oct 12, 2022

Choose a reason for hiding this comment

JaySon-Huang Oct 12, 2022

Choose a reason for hiding this comment

ywqzzy Oct 25, 2022

Choose a reason for hiding this comment

ywqzzy commented Oct 25, 2022

ywqzzy commented Oct 25, 2022

ywqzzy commented Oct 25, 2022

ywqzzy commented Oct 26, 2022

ywqzzy commented Oct 26, 2022

ti-chi-bot commented Oct 26, 2022

ti-chi-bot commented Oct 26, 2022

SeaRise commented Oct 31, 2022

ywqzzy commented Oct 8, 2022 •

edited

Loading

ti-chi-bot commented Oct 8, 2022 •

edited

Loading