You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a Parquet string read benchmark that compares the performance of reading Parquet files with a fixed number of characters with and without the fixed length optimization. This will test (1) single file reads (2) multi-file reads with larger files (3) multi-file reads with smaller files and (4) all of the previous with the fixed length optimization.
Here are results demonstrating ~2x speedup with using the fixed length string optimization collected on a Cray XC with a Lustre file system:
test
sec
single-file
39.388
fixed-single
20.664
scaled-five
17.283
fixed-scaled-five
8.814
five
85.332
fixed-five
43.743
scaled-ten
8.998
fixed-scaled-ten
4.794
ten
86.761
fixed-ten
45.498
The text was updated successfully, but these errors were encountered:
* Add Parquet fixed length string benchmark
Add a Parquet string read benchmark that compares the performance of
reading Parquet files with a fixed number of characters with and
without the fixed length optimization. This will test (1) single
file reads (2) multi-file reads with larger files (3) multi-file
reads with smaller files and (4) all of the previous with the fixed
length optimization.
Here are results demonstrating ~2x speedup with using the fixed
length string optimization collected on a Cray XC with a Lustre file system:
| test | sec |
|:------------------|-------:|
| single-file | 39.388 |
| fixed-single | 20.664 |
| scaled-five | 17.283 |
| fixed-scaled-five | 8.814 |
| five | 85.332 |
| fixed-five | 43.743 |
| scaled-ten | 8.998 |
| fixed-scaled-ten | 4.794 |
| ten | 86.761 |
| fixed-ten | 45.498 |
* Change default size
Add a Parquet string read benchmark that compares the performance of reading Parquet files with a fixed number of characters with and without the fixed length optimization. This will test (1) single file reads (2) multi-file reads with larger files (3) multi-file reads with smaller files and (4) all of the previous with the fixed length optimization.
Here are results demonstrating ~2x speedup with using the fixed length string optimization collected on a Cray XC with a Lustre file system:
The text was updated successfully, but these errors were encountered: