Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline: support fine grained shuffle #6934

Merged
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3e5c3fb
update partA
SeaRise Mar 2, 2023
60fcb90
part2
SeaRise Mar 2, 2023
884aefd
udpate
SeaRise Mar 2, 2023
d2520b6
fix comment
SeaRise Mar 2, 2023
c1362bd
add ut
SeaRise Mar 2, 2023
8aabbe8
fix
SeaRise Mar 2, 2023
700e04a
fix comment
SeaRise Mar 2, 2023
f704069
fix
SeaRise Mar 3, 2023
6035999
update
SeaRise Mar 3, 2023
63ca1c0
update
SeaRise Mar 3, 2023
e1220b1
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 3, 2023
4b7f61d
merge master
SeaRise Mar 3, 2023
06c8004
add fullstack test
SeaRise Mar 3, 2023
f6b9936
update
SeaRise Mar 3, 2023
88b64c8
fix comment
SeaRise Mar 3, 2023
688cc41
add more comments
SeaRise Mar 6, 2023
6aef338
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 7, 2023
d7effec
merge master and fix
SeaRise Mar 7, 2023
57a172c
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 13, 2023
d86bfeb
replace 1024 with maxFineGrainedStreamCount
SeaRise Mar 13, 2023
dac6717
refine
SeaRise Mar 14, 2023
c1cba4e
u
SeaRise Mar 14, 2023
cd049c9
replace magic number
SeaRise Mar 14, 2023
15356ab
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 14, 2023
08c3138
Update Pipeline.cpp
SeaRise Mar 14, 2023
0c891de
add more unit tests
SeaRise Mar 14, 2023
c65200c
Update ExecutorTestUtils.cpp
SeaRise Mar 14, 2023
4ae0cc5
address comment
SeaRise Mar 15, 2023
6f02e0e
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
ti-chi-bot Mar 20, 2023
3906d8b
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
ti-chi-bot Mar 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
  • Loading branch information
SeaRise authored Mar 7, 2023
commit 6aef338cfa7ac5a71e7c923c8eae812704cf7a56
22 changes: 17 additions & 5 deletions dbms/src/Flash/Planner/Plans/PhysicalMockTableScan.cpp
Original file line number Diff line number Diff line change
@@ -117,11 +117,23 @@ void PhysicalMockTableScan::buildPipelineExecGroup(
Context & /*context*/,
size_t /*concurrency*/)
{
group_builder.init(mock_streams.size());
size_t i = 0;
group_builder.transform([&](auto & builder) {
builder.setSourceOp(std::make_unique<BlockInputStreamSourceOp>(exec_status, log->identifier(), mock_streams[i++]));
});
if (context.mockStorage()->useDeltaMerge())
{
auto source_ops = context.mockStorage()->getSourceOpsFromDeltaMerge(exec_status, context, table_id, concurrency);
group_builder.init(source_ops.size());
size_t i = 0;
group_builder.transform([&](auto & builder) {
builder.setSourceOp(std::move(source_ops[i++]));
});
}
else
{
group_builder.init(mock_streams.size());
size_t i = 0;
group_builder.transform([&](auto & builder) {
builder.setSourceOp(std::make_unique<BlockInputStreamSourceOp>(exec_status, log->identifier(), mock_streams[i++]));
});
}
}

void PhysicalMockTableScan::finalize(const Names & parent_require)
You are viewing a condensed version of this merge commit. You can view the full changes here.