Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor dynamic filtering configs #5262

Merged
merged 2 commits into from
Sep 24, 2020

Conversation

raunaqmorarka
Copy link
Member

Moved dynamic filtering related configs to DynamicFilterConfig.
Added enable-large-dynamic-filters to allow collection of
large dynamic filters.
Removed existing configs dynamic-filtering-max-per-driver-row-count,
dynamic-filtering-max-per-driver-size,
dynamic-filtering-range-row-limit-per-driver and their session
properties. These are replaced by new properties based on
join distribution type and value of enable_large_dynamic_filters.

@raunaqmorarka raunaqmorarka force-pushed the df_config_refactor branch 2 times, most recently from 856244e to f5e09e0 Compare September 22, 2020 16:29
@raunaqmorarka raunaqmorarka force-pushed the df_config_refactor branch 3 times, most recently from f3b5974 to c364f28 Compare September 23, 2020 14:26
@raunaqmorarka
Copy link
Member Author

# Run complete. Total time: 00:07:24

Benchmark                                                  (collectionLimits)  (positionsPerPage)   Mode  Cnt      Score     Error  Units
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect               100,0                  32  thrpt   60   1332.206 ±  67.269  ops/s
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect               100,0                1024  thrpt   60  20249.422 ± 838.008  ops/s
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect            500,5000                  32  thrpt   60   1210.126 ±  14.265  ops/s
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect            500,5000                1024  thrpt   60  13043.162 ± 493.613  ops/s
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect          5000,50000                  32  thrpt   60    579.463 ±  15.016  ops/s
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect          5000,50000                1024  thrpt   60   1473.962 ±  32.937  ops/s

@raunaqmorarka
Copy link
Member Author

Results for some more permutations

Benchmark                                                  (collectionLimits)  (positionsPerPage)  Mode  Cnt  Score   Error  Units
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect            100,1000                1024  avgt   45  0.044 ± 0.001  ms/op
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect            500,5000                1024  avgt   45  0.074 ± 0.001  ms/op
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect          1000,10000                1024  avgt   45  0.138 ± 0.007  ms/op
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect           500,50000                1024  avgt   45  0.266 ± 0.007  ms/op
BenchmarkDynamicFilterSourceOperator.dynamicFilterCollect          5000,50000                1024  avgt   45  0.698 ± 0.015  ms/op

Copy link
Member

@sopel39 sopel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small comments

@@ -166,7 +170,7 @@ public void testJoinWithNonSelectiveBuildSide()
}

@Test(timeOut = 30_000)
public void testJoinLargeBuildSideNoDynamicFiltering()
public void testJoinLargeBuildSideDynamicFiltering()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe call it testJoinLargeBuildSideRangeDynamicFiltering
Also, we are missing test case for "too" large build side for DFs to be collected. It would be good to restore it, but it's not critical

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed, I'll add the too-large-no-df test as well in a different PR

Moved dynamic filtering related configs to DynamicFilterConfig.
Added enable-large-dynamic-filters to allow collection of
large dynamic filters.
Removed existing configs dynamic-filtering-max-per-driver-row-count,
dynamic-filtering-max-per-driver-size,
dynamic-filtering-range-row-limit-per-driver and their session
properties. These are replaced by new properties based on
join distribution type and value of enable_large_dynamic_filters.
@sopel39 sopel39 merged commit 38ecd82 into trinodb:master Sep 24, 2020
@sopel39
Copy link
Member

sopel39 commented Sep 24, 2020

merged thanks!

@sopel39 sopel39 mentioned this pull request Sep 24, 2020
9 tasks
@raunaqmorarka raunaqmorarka deleted the df_config_refactor branch September 24, 2020 13:56
@martint martint added this to the 342 milestone Sep 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

3 participants