Skip to content

Commit

Permalink
Fix config format problem (#11278)
Browse files Browse the repository at this point in the history
Signed-off-by: liurenjie1024 <[email protected]>
  • Loading branch information
liurenjie1024 authored Jul 31, 2024
1 parent c9f1ab9 commit 20bff54
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 9 deletions.
6 changes: 1 addition & 5 deletions docs/additional-functionality/advanced_configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,7 @@ Name | Description | Default Value | Applicable at
<a name="sql.csv.read.float.enabled"></a>spark.rapids.sql.csv.read.float.enabled|CSV reading is not 100% compatible when reading floats.|true|Runtime
<a name="sql.decimalOverflowGuarantees"></a>spark.rapids.sql.decimalOverflowGuarantees|FOR TESTING ONLY. DO NOT USE IN PRODUCTION. Please see the decimal section of the compatibility documents for more information on this config.|true|Runtime
<a name="sql.delta.lowShuffleMerge.deletionVector.broadcast.threshold"></a>spark.rapids.sql.delta.lowShuffleMerge.deletionVector.broadcast.threshold|Currently we need to broadcast deletion vector to all executors to perform low shuffle merge. When we detect the deletion vector broadcast size is larger than this value, we will fallback to normal shuffle merge.|20971520|Runtime
<a name="sql.delta.lowShuffleMerge.enabled"></a>spark.rapids.sql.delta.lowShuffleMerge.enabled|Option to turn on the low shuffle merge for Delta Lake. Currently there are some limitations for this feature:
1. We only support Databricks Runtime 13.3 and Deltalake 2.4.
2. The file scan mode must be set to PERFILE
3. The deletion vector size must be smaller than spark.rapids.sql.delta.lowShuffleMerge.deletionVector.broadcast.threshold
|false|Runtime
<a name="sql.delta.lowShuffleMerge.enabled"></a>spark.rapids.sql.delta.lowShuffleMerge.enabled|Option to turn on the low shuffle merge for Delta Lake. Currently there are some limitations for this feature: 1. We only support Databricks Runtime 13.3 and Deltalake 2.4. 2. The file scan mode must be set to PERFILE 3. The deletion vector size must be smaller than spark.rapids.sql.delta.lowShuffleMerge.deletionVector.broadcast.threshold |false|Runtime
<a name="sql.detectDeltaCheckpointQueries"></a>spark.rapids.sql.detectDeltaCheckpointQueries|Queries against Delta Lake _delta_log checkpoint Parquet files are not efficient on the GPU. When this option is enabled, the plugin will attempt to detect these queries and fall back to the CPU.|true|Runtime
<a name="sql.detectDeltaLogQueries"></a>spark.rapids.sql.detectDeltaLogQueries|Queries against Delta Lake _delta_log JSON files are not efficient on the GPU. When this option is enabled, the plugin will attempt to detect these queries and fall back to the CPU.|true|Runtime
<a name="sql.fast.sample"></a>spark.rapids.sql.fast.sample|Option to turn on fast sample. If enable it is inconsistent with CPU sample because of GPU sample algorithm is inconsistent with CPU.|false|Runtime
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2330,11 +2330,11 @@ val SHUFFLE_COMPRESSION_LZ4_CHUNK_SIZE = conf("spark.rapids.shuffle.compression.
val ENABLE_DELTA_LOW_SHUFFLE_MERGE =
conf("spark.rapids.sql.delta.lowShuffleMerge.enabled")
.doc("Option to turn on the low shuffle merge for Delta Lake. Currently there are some " +
"limitations for this feature: \n" +
"1. We only support Databricks Runtime 13.3 and Deltalake 2.4. \n" +
s"2. The file scan mode must be set to ${RapidsReaderType.PERFILE} \n" +
"limitations for this feature: " +
"1. We only support Databricks Runtime 13.3 and Deltalake 2.4. " +
s"2. The file scan mode must be set to ${RapidsReaderType.PERFILE} " +
"3. The deletion vector size must be smaller than " +
s"${DELTA_LOW_SHUFFLE_MERGE_DEL_VECTOR_BROADCAST_THRESHOLD.key} \n")
s"${DELTA_LOW_SHUFFLE_MERGE_DEL_VECTOR_BROADCAST_THRESHOLD.key} ")
.booleanConf
.createWithDefault(false)

Expand Down

0 comments on commit 20bff54

Please sign in to comment.