Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: take_run improvements #3705

Merged
merged 4 commits into from
Feb 13, 2023
Merged

perf: take_run improvements #3705

merged 4 commits into from
Feb 13, 2023

Conversation

askoa
Copy link
Contributor

@askoa askoa commented Feb 12, 2023

Which issue does this PR close?

Closes #3701
Part of #3520

Rationale for this change

See issue description #3701.

What changes are included in this PR?

Update the take_run function to do the below
1 Take physical_indices for the given logical indices.
2 Run encode the physical_indices while keeping track of physical indices that needs to be taken.
3 take values from run_array.values based on physical indices from the step 2.
4 Build a new run array using run_ends from step 2 and values from step 3.

The performance of benchmark primitive_take_run has improved by approx 15%

Benchmark result
Primitive_run_take/(run_array_len:512, physical_array_len:64, take_len:512)
                        time:   [14.263 µs 14.289 µs 14.314 µs]
                        change: [-16.324% -15.022% -13.604%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe
primitive_run_take/(run_array_len:512, physical_array_len:128, take_len:512)
                        time:   [14.299 µs 14.320 µs 14.344 µs]
                        change: [-17.510% -16.083% -14.660%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe
primitive_run_take/(run_array_len:1024, physical_array_len:256, take_len:512)
                        time:   [14.672 µs 14.689 µs 14.706 µs]
                        change: [-17.290% -15.772% -14.314%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe
primitive_run_take/(run_array_len:1024, physical_array_len:256, take_len:1024)
                        time:   [29.943 µs 29.972 µs 30.002 µs]
                        change: [-18.529% -17.130% -15.819%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  7 (7.00%) high severe
primitive_run_take/(run_array_len:2048, physical_array_len:512, take_len:512)
                        time:   [15.864 µs 15.896 µs 15.928 µs]
                        change: [-15.176% -13.517% -11.892%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe
primitive_run_take/(run_array_len:2048, physical_array_len:512, take_len:1024)
                        time:   [34.206 µs 34.275 µs 34.347 µs]
                        change: [-15.092% -13.950% -12.743%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe
primitive_run_take/(run_array_len:4096, physical_array_len:1024, take_len:512)
                        time:   [18.066 µs 18.094 µs 18.121 µs]
                        change: [-14.902% -13.842% -12.604%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe
primitive_run_take/(run_array_len:4096, physical_array_len:1024, take_len:1024)
                        time:   [37.529 µs 37.571 µs 37.612 µs]
                        change: [-17.016% -15.749% -14.471%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  6 (6.00%) high severe

Are there any user-facing changes?

No

@github-actions github-actions bot added the arrow Changes to the arrow crate label Feb 12, 2023
Copy link
Contributor

@tustvold tustvold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, I believe this also adds support for taking from arbitrary RunEndEncoded arrays?

arrow-select/src/take.rs Outdated Show resolved Hide resolved
@tustvold
Copy link
Contributor

CI failure is unrelated and should be fixed by merging master into this branch

@askoa
Copy link
Contributor Author

askoa commented Feb 13, 2023

I believe this also adds support for taking from arbitrary RunEndEncoded arrays?

Yes, any data type supported by take will work.

@tustvold tustvold merged commit d011e6a into apache:master Feb 13, 2023
@ursabot
Copy link

ursabot commented Feb 13, 2023

Benchmark runs are scheduled for baseline = e37e379 and contender = d011e6a. d011e6a is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

take_run improvements
3 participants