Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](shuffle) Reduce memory consumption in data stream sender #41676

Merged
merged 2 commits into from
Oct 11, 2024
Merged

[Improvement](shuffle) Reduce memory consumption in data stream sender #41676

merged 2 commits into from
Oct 11, 2024

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Before:
image
image

After:
image

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Oct 10, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/runtime/vdata_stream_recvr.cpp Outdated
@@ -405,9 +395,9 @@ Status VDataStreamRecvr::add_block(const PBlock& pblock, int sender_id, int be_n
wait_for_worker, time_to_find_recvr);
}

void VDataStreamRecvr::add_block(Block* block, int sender_id, bool use_move) {
void VDataStreamRecvr::add_block(Block* block, int sender_id) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'add_block' can be made const [readability-make-member-function-const]

Suggested change
void VDataStreamRecvr::add_block(Block* block, int sender_id) {
void VDataStreamRecvr::add_block(Block* block, int sender_id) const {

be/src/vec/runtime/vdata_stream_recvr.h:88:

-     void add_block(Block* block, int sender_id);
+     void add_block(Block* block, int sender_id) const;

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.41% (9661/25825)
Line Coverage: 28.69% (80175/279456)
Region Coverage: 28.12% (41447/147402)
Branch Coverage: 24.72% (21107/85390)
Coverage Report: http://coverage.selectdb-in.cc/coverage/656146a35845ffad277388df9735524395cfa526_656146a35845ffad277388df9735524395cfa526/report/index.html

HappenLee
HappenLee previously approved these changes Oct 11, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.40% (9660/25827)
Line Coverage: 28.67% (80136/279497)
Region Coverage: 28.11% (41445/147457)
Branch Coverage: 24.71% (21105/85420)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bd85447159141deb697f2958cef258a369b49b0e_bd85447159141deb697f2958cef258a369b49b0e/report/index.html

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@Gabriel39 Gabriel39 merged commit af95818 into apache:master Oct 11, 2024
24 of 27 checks passed
@Gabriel39 Gabriel39 mentioned this pull request Oct 12, 2024
Merged
Gabriel39 added a commit that referenced this pull request Oct 12, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (#41740)
cce5425 
Follow-up for #41676
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024
@Gabriel39
[Improvement](shuffle) Reduce memory consumption in data stream sender (
227cc8c 
apache#41676)
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
4be7ee5 
)

Follow-up for apache#41676
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 14, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
58c9653 
)

Follow-up for apache#41676
dataroaring pushed a commit that referenced this pull request Oct 15, 2024
@Gabriel39
[pick](branch-3.0) pick #39982 #40576 #41555 #41592 #41676 #41740 (#4…
54b4fa3 
…1808)
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 15, 2024
@Gabriel39
[Improvement](shuffle) Reduce memory consumption in data stream sender (
43bb8b1 
apache#41676)
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Oct 15, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
e90a846 
)

Follow-up for apache#41676
@Gabriel39 Gabriel39 mentioned this pull request Oct 15, 2024
Merged
Gabriel39 added a commit that referenced this pull request Oct 15, 2024
@Gabriel39
[pick](branch-2.1) pick #41676 #41740 #41857 (#41904)
b185dfc 
pick #41676 #41740 #41857
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
@Gabriel39
[followup](memory) Block should be moved by hash shuffling (apache#41740
272a8ae 
)

Follow-up for apache#41676
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@Mryange Mryange Mryange approved these changes

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@HappenLee HappenLee HappenLee left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Gabriel39 @doris-robot @BiteTheDDDDt @HappenLee @Mryange