Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipeline: support fine grained shuffle #6934

Merged
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3e5c3fb
update partA
SeaRise Mar 2, 2023
60fcb90
part2
SeaRise Mar 2, 2023
884aefd
udpate
SeaRise Mar 2, 2023
d2520b6
fix comment
SeaRise Mar 2, 2023
c1362bd
add ut
SeaRise Mar 2, 2023
8aabbe8
fix
SeaRise Mar 2, 2023
700e04a
fix comment
SeaRise Mar 2, 2023
f704069
fix
SeaRise Mar 3, 2023
6035999
update
SeaRise Mar 3, 2023
63ca1c0
update
SeaRise Mar 3, 2023
e1220b1
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 3, 2023
4b7f61d
merge master
SeaRise Mar 3, 2023
06c8004
add fullstack test
SeaRise Mar 3, 2023
f6b9936
update
SeaRise Mar 3, 2023
88b64c8
fix comment
SeaRise Mar 3, 2023
688cc41
add more comments
SeaRise Mar 6, 2023
6aef338
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 7, 2023
d7effec
merge master and fix
SeaRise Mar 7, 2023
57a172c
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 13, 2023
d86bfeb
replace 1024 with maxFineGrainedStreamCount
SeaRise Mar 13, 2023
dac6717
refine
SeaRise Mar 14, 2023
c1cba4e
u
SeaRise Mar 14, 2023
cd049c9
replace magic number
SeaRise Mar 14, 2023
15356ab
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
SeaRise Mar 14, 2023
08c3138
Update Pipeline.cpp
SeaRise Mar 14, 2023
0c891de
add more unit tests
SeaRise Mar 14, 2023
c65200c
Update ExecutorTestUtils.cpp
SeaRise Mar 14, 2023
4ae0cc5
address comment
SeaRise Mar 15, 2023
6f02e0e
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
ti-chi-bot Mar 20, 2023
3906d8b
Merge branch 'master' into support_fine_grained_shuffle_for_pipeline
ti-chi-bot Mar 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
replace 1024 with maxFineGrainedStreamCount
  • Loading branch information
SeaRise committed Mar 13, 2023
commit d86bfebe0985d1710593a6e76ad968a84a77f892
2 changes: 2 additions & 0 deletions dbms/src/Flash/Coprocessor/FineGrainedShuffle.h
Original file line number Diff line number Diff line change
@@ -21,6 +21,8 @@ namespace DB
{
static constexpr std::string_view enableFineGrainedShuffleExtraInfo = "enable fine grained shuffle";

static constexpr size_t maxFineGrainedStreamCount = 1024;

inline bool enableFineGrainedShuffle(uint64_t stream_count)
{
return stream_count > 0;
4 changes: 2 additions & 2 deletions dbms/src/Flash/Planner/Plans/PhysicalExchangeSender.cpp
Original file line number Diff line number Diff line change
@@ -65,7 +65,7 @@ void PhysicalExchangeSender::buildBlockInputStreamImpl(DAGPipeline & pipeline, C
{
extra_info = String(enableFineGrainedShuffleExtraInfo);
RUNTIME_CHECK(exchange_type == tipb::ExchangeType::Hash, ExchangeType_Name(exchange_type));
RUNTIME_CHECK(fine_grained_shuffle.stream_count <= 1024, fine_grained_shuffle.stream_count);
RUNTIME_CHECK(fine_grained_shuffle.stream_count <= maxFineGrainedStreamCount, fine_grained_shuffle.stream_count);
}
pipeline.transform([&](auto & stream) {
// construct writer
@@ -96,7 +96,7 @@ void PhysicalExchangeSender::buildPipelineExecGroup(
if (fine_grained_shuffle.enable())
{
RUNTIME_CHECK(exchange_type == tipb::ExchangeType::Hash, ExchangeType_Name(exchange_type));
RUNTIME_CHECK(fine_grained_shuffle.stream_count <= 1024, fine_grained_shuffle.stream_count);
RUNTIME_CHECK(fine_grained_shuffle.stream_count <= maxFineGrainedStreamCount, fine_grained_shuffle.stream_count);
}

group_builder.transform([&](auto & builder) {