[SPARK-43829][CONNECT] Improve SparkConnectPlanner by reuse Dataset and avoid construct new Dataset #43473
Conversation
beliefer
force-pushed
the
SPARK-43829_new
branch
2 times, most recently
from
9c51ed4 to
13b8449
Compare
|
ping @hvanhovell cc @HyukjinKwon @zhengruifeng @ueshin |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
…nd avoid construct new Dataset
beliefer
force-pushed
the
SPARK-43829_new
branch
3 times, most recently
from
b11a9d8 to
d919877
Compare
beliefer
force-pushed
the
SPARK-43829_new
branch
from
d919877 to
b6e0183
Compare
beliefer
force-pushed
the
SPARK-43829_new
branch
from
dd0504c to
9d4c19d
Compare
| @@ -115,6 +115,49 @@ class SparkConnectPlanner( | |||
| private lazy val pythonExec = | |||
| sys.env.getOrElse("PYSPARK_PYTHON", sys.env.getOrElse("PYSPARK_DRIVER_PYTHON", "python3")) | |||
|
|
|||
| // Some relation transform need to create Dataset, then get the logical plan from the Dataset. | |||
| // This method used to reuse the Dataset instead to discard it. | |||
| def transformRelationAsDataset(rel: proto.Relation): Dataset[Row] = { | |||
There was a problem hiding this comment.
TBH I want to move away from constructing Datasets wholesale. In many cases there is no real need, and it is also expensive to do.
There was a problem hiding this comment.
Is this something you want to work on?
There was a problem hiding this comment.
TBH I want to move away from constructing Datasets wholesale. In many cases there is no real need, and it is also expensive to do.
Yes. the current implementation create man duplicate datasets and then discard them. We should reuse these datasets and reduce the overhead.
There was a problem hiding this comment.
Is this something you want to work on?
Yes.
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
What changes were proposed in this pull request?
Currently,
SparkConnectPlanner.transformRelationalways return theLogicalPlanofDataset.The
SparkConnectPlanExecution.handlePlanconstructs a newDatasetfor it.Sometimes,
SparkConnectPlanExecution.handlePlancould reuse theDatasetcreated bySparkConnectPlanner.transformRelation.This PR creates a common method
transformRelationAsDataset, so as theDatasetcould be reused in many places.Why are the changes needed?
Improve
SparkConnectPlannerby reuseDatasetand avoid construct newDataset.Does this PR introduce any user-facing change?
'No'.
Just update inner implementation.
How was this patch tested?
Exists test cases.
Was this patch authored or co-authored using generative AI tooling?
'No'.