Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Benchmarks] Make partitions default to number of cores instead of 2 #8292

Merged
merged 2 commits into from
Nov 22, 2023
Merged

[Benchmarks] Make partitions default to number of cores instead of 2 #8292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2985fca
Default partitions to num cores
andygrove Nov 21, 2023
ccefd32
update test
andygrove Nov 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions benchmarks/src/sort.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,9 @@ impl RunOpt {
println!("Executing '{title}' (sorting by: {expr:?})");
rundata.start_new_case(title);
for i in 0..self.common.iterations {
let config =
SessionConfig::new().with_target_partitions(self.common.partitions);
let config = SessionConfig::new().with_target_partitions(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think SessionConfig already defaults to the number of CPUs, so we can probably just avoid setting this at all if self.common.partitions.is_none()

self.common.partitions.unwrap_or(num_cpus::get()),
);
let ctx = SessionContext::new_with_config(config);
let (rows, elapsed) =
exec_sort(&ctx, &expr, &test_file, self.common.debug).await?;
Expand Down
6 changes: 3 additions & 3 deletions benchmarks/src/tpch/run.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ impl RunOpt {
}

fn partitions(&self) -> usize {
self.common.partitions
self.common.partitions.unwrap_or(num_cpus::get())
}
}

Expand Down Expand Up @@ -325,7 +325,7 @@ mod tests {
let path = get_tpch_data_path()?;
let common = CommonOpt {
iterations: 1,
partitions: 2,
partitions: Some(2),
batch_size: 8192,
debug: false,
};
Expand Down Expand Up @@ -357,7 +357,7 @@ mod tests {
let path = get_tpch_data_path()?;
let common = CommonOpt {
iterations: 1,
partitions: 2,
partitions: Some(2),
batch_size: 8192,
debug: false,
};
Expand Down
8 changes: 4 additions & 4 deletions benchmarks/src/util/options.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ pub struct CommonOpt {
#[structopt(short = "i", long = "iterations", default_value = "3")]
pub iterations: usize,

/// Number of partitions to process in parallel
#[structopt(short = "n", long = "partitions", default_value = "2")]
pub partitions: usize,
/// Number of partitions to process in parallel. Defaults to number of available cores.
#[structopt(short = "n", long = "partitions")]
pub partitions: Option<usize>,

/// Batch size when reading CSV or Parquet files
#[structopt(short = "s", long = "batch-size", default_value = "8192")]
Expand All @@ -48,7 +48,7 @@ impl CommonOpt {
/// Modify the existing config appropriately
pub fn update_config(&self, config: SessionConfig) -> SessionConfig {
config
.with_target_partitions(self.partitions)
.with_target_partitions(self.partitions.unwrap_or(num_cpus::get()))
.with_batch_size(self.batch_size)
}
}