Scala API for Optimize #960
Labels
enhancement
New feature or request
Comments
jbguerraz
pushed a commit
to jbguerraz/delta
that referenced
this issue
Jul 6, 2022
Add functions to `DeltaTable` to perform optimization.
API documentation:
```
/**
* Optimize the data layout of the table. This returns
* a [[DeltaOptimizeBuilder]] object that can be used to specify
* the partition filter to limit the scope of optimize and
* also execute different optimization techniques such as file
* compaction or order data using Z-Order curves.
*
* See the [[DeltaOptimizeBuilder]] for a full description
* of this operation.
*
* Scala example to run file compaction on a subset of
* partitions in the table:
* {{{
* deltaTable
* .optimize()
* .where("date='2021-11-18'")
* .executeCompaction();
* }}}
*
* @SInCE 1.3.0
*/
def optimize(): DeltaOptimizeBuilder
```
```
/**
* Builder class for constructing OPTIMIZE command and executing.
*
* @param sparkSession SparkSession to use for execution
* @param tableIdentifier Id of the table on which to
* execute the optimize
* @SInCE 1.3.0
*/
class DeltaOptimizeBuilder(
sparkSession: SparkSession,
tableIdentifier: String) extends AnalysisHelper {
/**
* Apply partition filter on this optimize command builder to limit
* the operation on selected partitions.
* @param partitionFilter The partition filter to apply
* @return [[DeltaOptimizeBuilder]] with partition filter applied
*/
def where(partitionFilter: String): DeltaOptimizeBuilder
/**
* Compact the small files in selected partitions.
* @return DataFrame containing the OPTIMIZE execution metrics
*/
def executeCompaction(): DataFrame
}
```
Closes delta-io#961
Fixes delta-io#960
Signed-off-by: Venki Korukanti <[email protected]>
GitOrigin-RevId: 615e215b96fb9e9b9223d3d2b429dc18dff102f4
jbguerraz
pushed a commit
to jbguerraz/delta
that referenced
this issue
Jul 6, 2022
Add functions to `DeltaTable` to perform optimization.
API documentation:
```
/**
* Optimize the data layout of the table. This returns
* a [[DeltaOptimizeBuilder]] object that can be used to specify
* the partition filter to limit the scope of optimize and
* also execute different optimization techniques such as file
* compaction or order data using Z-Order curves.
*
* See the [[DeltaOptimizeBuilder]] for a full description
* of this operation.
*
* Scala example to run file compaction on a subset of
* partitions in the table:
* {{{
* deltaTable
* .optimize()
* .where("date='2021-11-18'")
* .executeCompaction();
* }}}
*
* @SInCE 1.3.0
*/
def optimize(): DeltaOptimizeBuilder
```
```
/**
* Builder class for constructing OPTIMIZE command and executing.
*
* @param sparkSession SparkSession to use for execution
* @param tableIdentifier Id of the table on which to
* execute the optimize
* @SInCE 1.3.0
*/
class DeltaOptimizeBuilder(
sparkSession: SparkSession,
tableIdentifier: String) extends AnalysisHelper {
/**
* Apply partition filter on this optimize command builder to limit
* the operation on selected partitions.
* @param partitionFilter The partition filter to apply
* @return [[DeltaOptimizeBuilder]] with partition filter applied
*/
def where(partitionFilter: String): DeltaOptimizeBuilder
/**
* Compact the small files in selected partitions.
* @return DataFrame containing the OPTIMIZE execution metrics
*/
def executeCompaction(): DataFrame
}
```
Closes delta-io#961
Fixes delta-io#960
Signed-off-by: Venki Korukanti <[email protected]>
GitOrigin-RevId: 615e215b96fb9e9b9223d3d2b429dc18dff102f4
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add an API for calling Optimize through a DeltaTable instance.
The text was updated successfully, but these errors were encountered: