Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new user doc to translate logical plan to physical plan #12026

Merged
merged 11 commits into from
Aug 21, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,4 @@ workspace = true

[dependencies]
datafusion = { workspace = true }
tokio = { workspace = true }
jc4x4 marked this conversation as resolved.
Show resolved Hide resolved
32 changes: 32 additions & 0 deletions docs/source/library-user-guide/building-logical-plans.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,38 @@ Filter: person.id > Int32(500) [id:Int32;N, name:Utf8;N]
TableScan: person [id:Int32;N, name:Utf8;N]
```

## Translating Logical Plan to Physical Plan
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jc4x4 -- I have some ideas on improving this example -- I will push some suggestions to this branch


<!-- source for this example is in datafusion_docs::library_logical_plan::translate_logical_to_physical -->

```rust
// create a default table source
let schema = Schema::new(vec![
Field::new("id", DataType::Int32, true),
Field::new("name", DataType::Utf8, true),
]);
let table_provider = Arc::new(MemTable::try_new(Arc::new(schema), vec![])?);
let table_source = Arc::new(DefaultTableSource::new(table_provider));

// create a LogicalPlanBuilder for a table scan without projection or filters
let logical_plan = LogicalPlanBuilder::scan("person", table_source, None)?.build()?;

// create a physical plan using the default physical planner
let ctx = SessionContext::new();
let planner = DefaultPhysicalPlanner::default();
let physical_plan = planner.create_physical_plan(&logical_plan, &ctx.state()).await?;

// print the plan
println!("{}", DisplayableExecutionPlan::new(physical_plan.as_ref()).indent(true));
```

This example produces the following physical plan:

```
MemoryExec: partitions=0, partition_sizes=[]
```


## Table Sources

The previous example used a [LogicalTableSource], which is used for tests and documentation in DataFusion, and is also
Expand Down
32 changes: 32 additions & 0 deletions docs/src/library_logical_plan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,12 @@
// under the License.

use datafusion::arrow::datatypes::{DataType, Field, Schema, SchemaRef};
use datafusion::datasource::{DefaultTableSource, MemTable};
use datafusion::error::Result;
use datafusion::logical_expr::builder::LogicalTableSource;
use datafusion::logical_expr::{Filter, LogicalPlan, LogicalPlanBuilder, TableScan};
use datafusion::physical_plan::display::DisplayableExecutionPlan;
use datafusion::physical_planner::{DefaultPhysicalPlanner, PhysicalPlanner};
use datafusion::prelude::*;
use std::sync::Arc;

Expand Down Expand Up @@ -76,3 +79,32 @@ fn plan_builder_1() -> Result<()> {

Ok(())
}

#[tokio::test]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is confusing -- i don't think it is used anymore --- I added #12081 to remove it

async fn translate_logical_to_physical() -> Result<()> {
// create a default table source
let schema = Schema::new(vec![
Field::new("id", DataType::Int32, true),
Field::new("name", DataType::Utf8, true),
]);
let table_provider = Arc::new(MemTable::try_new(Arc::new(schema), vec![])?);
let table_source = Arc::new(DefaultTableSource::new(table_provider));

// create a LogicalPlanBuilder for a table scan without projection or filters
let logical_plan = LogicalPlanBuilder::scan("person", table_source, None)?.build()?;

// create a physical plan using the default physical planner
let ctx = SessionContext::new();
let planner = DefaultPhysicalPlanner::default();
let physical_plan = planner
.create_physical_plan(&logical_plan, &ctx.state())
.await?;

// print the plan
println!(
"{}",
DisplayableExecutionPlan::new(physical_plan.as_ref()).indent(true)
);

Ok(())
}
Loading