Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ArrayAccessor, Iterator, Extend and benches for string run builder #3583

Closed
wants to merge 20 commits into from
Closed

Add ArrayAccessor, Iterator, Extend and benches for string run builder #3583

wants to merge 20 commits into from

Conversation

askoa
Copy link
Contributor

@askoa askoa commented Jan 22, 2023

Which issue does this PR close?

Part of #3520

Rationale for this change

See issue description

What changes are included in this PR?

Built on top of yet to be merged (1/21/2023) PRs #3553 and #3534. Hence marking this PR draft.

  • Implement Extend for the PrimitiveRunBuilder and GenericByteRunBuilder
  • Add TypedRunArray
  • Add RunArrayIter and ArrayAccessor for TypedRunArray
  • Add benchmarks for StringRunBuilder

Are there any user-facing changes?

Users will get new features for RunArray.

No breaking changes.

@github-actions github-actions bot added arrow Changes to the arrow crate parquet Changes to the parquet crate labels Jan 22, 2023
@askoa askoa changed the title Ree array more Add ArrayAccessor, Iterator and look up using binary search for REE Array Jan 22, 2023
@askoa
Copy link
Contributor Author

askoa commented Jan 22, 2023

Benchmark for building REE Array

Benchmarking string_ree_array_builder/(ree_array_len:20, logical_array_len:1000, string_len: 5): Collecting 100 samples in estimated 5.0191 s (192k iterati
string_ree_array_builder/(ree_array_len:20, logical_array_len:1000, string_len: 5)
                        time:   [22.523 µs 23.376 µs 24.348 µs]
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe
Benchmarking string_ree_array_builder/(ree_array_len:100, logical_array_len:1000, string_len: 5): Collecting 100 samples in estimated 5.1036 s (167k iterat
string_ree_array_builder/(ree_array_len:100, logical_array_len:1000, string_len: 5)
                        time:   [28.168 µs 29.445 µs 31.107 µs]
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe
Benchmarking string_ree_array_builder/(ree_array_len:100, logical_array_len:1000, string_len: 10): Collecting 100 samples in estimated 5.0983 s (212k itera
string_ree_array_builder/(ree_array_len:100, logical_array_len:1000, string_len: 10)
                        time:   [23.035 µs 23.994 µs 25.231 µs]
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe
Benchmarking string_ree_array_builder/(ree_array_len:100, logical_array_len:10000, string_len: 10): Collecting 100 samples in estimated 5.4504 s (45k itera
string_ree_array_builder/(ree_array_len:100, logical_array_len:10000, string_len: 10)
                        time:   [118.89 µs 123.77 µs 130.04 µs]
Found 10 outliers among 100 measurements (10.00%)
  2 (2.00%) high mild
  8 (8.00%) high severe
Benchmarking string_ree_array_builder/(ree_array_len:100, logical_array_len:10000, string_len: 100): Collecting 100 samples in estimated 5.2981 s (15k iter
string_ree_array_builder/(ree_array_len:100, logical_array_len:10000, string_len: 100)
                        time:   [296.52 µs 310.34 µs 328.42 µs]
Found 17 outliers among 100 measurements (17.00%)
  8 (8.00%) high mild
  9 (9.00%) high severe

@askoa askoa changed the title Add ArrayAccessor, Iterator and look up using binary search for REE Array Add ArrayAccessor, Iterator, Extend and benches for string run builder Jan 24, 2023
@askoa
Copy link
Contributor Author

askoa commented Jan 25, 2023

Will make a new PR.

@askoa askoa closed this Jan 25, 2023
@askoa askoa deleted the ree-array-more branch January 25, 2023 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant