Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve performance of count_data #1091

Merged
merged 13 commits into from
Aug 30, 2022

Conversation

GehaFearless
Copy link
Contributor

@GehaFearless GehaFearless commented Aug 1, 2022

What problem does this PR solve?

issue: #1090
same job as before: #728

When we precisely count data for a large table, it will cost minutes or hours.
However, it's unnecessarily return key-values from server to client.

What is changed and how does it work?

Actually, we just need the count of data. So we just need transfer the count of data from server to client, but not the detailed data.

In my test, it will 2x on onebox faster than before.
for my bench on onebox:
1> before
wecom-temp-d7e2a1e011357a0ca4c3ad44d910e69e

2> this commit
image

Tests
  • Unit test
  • Manual test (add detailed scripts or steps below)
Related changes
  • Need to update the documentation
  • Need to be included in the release note

@empiredan empiredan changed the title improve performance of count_data feat: improve performance of count_data Aug 1, 2022
src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved
src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved
src/shell/commands/data_operations.cpp Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/shell/command_helper.h Outdated Show resolved Hide resolved
src/shell/command_helper.h Outdated Show resolved Hide resolved
@GehaFearless GehaFearless force-pushed the lgh_count_data branch 2 times, most recently from f4c5af1 to 1be846c Compare August 3, 2022 11:44
idl/rrdb.thrift Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
idl/rrdb.thrift Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
@GehaFearless GehaFearless force-pushed the lgh_count_data branch 3 times, most recently from 8d2fe21 to bb901e0 Compare August 11, 2022 03:40
@acelyc111
Copy link
Member

Could you give some performance comparation with the old version ?

src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
@GehaFearless GehaFearless force-pushed the lgh_count_data branch 3 times, most recently from 2e0e6fd to dad6442 Compare August 15, 2022 14:06
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/test/function_test/test_scan.cpp Outdated Show resolved Hide resolved
src/shell/commands/data_operations.cpp Show resolved Hide resolved
idl/rrdb.thrift Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_client_impl.h Outdated Show resolved Hide resolved
src/test/function_test/test_scan.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_client_impl.h Outdated Show resolved Hide resolved
src/client_lib/pegasus_client_impl.h Outdated Show resolved Hide resolved
acelyc111
acelyc111 previously approved these changes Aug 24, 2022
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/client_lib/pegasus_scanner_impl.cpp Outdated Show resolved Hide resolved
src/shell/commands/data_operations.cpp Outdated Show resolved Hide resolved
@empiredan empiredan merged commit 7e26c9f into apache:master Aug 30, 2022
@empiredan empiredan mentioned this pull request Aug 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants