Skip to content

Commit

Permalink
[SPARK-44718][SQL] Match ColumnVector memory-mode config default to O…
Browse files Browse the repository at this point in the history
…ffHeapMemoryMode config value

### What changes were proposed in this pull request?

Set the column vector default memory mode to depend on the off-heap memory mode flag. This is to prevent a user from using Vectorized-Reader with an on-heap column-vector by default when the off-heap memory mode is enabled on the cluster.

### Why are the changes needed?
Avoid the unintentional usage of on-heap memory in vectorized-reader when off-heap memory mode is enabled by the user.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Manual & existing tests.

Closes apache#42394 from majdyz/offheap-colvec-mode-default-value.

Lead-authored-by: Zamil Majdy <[email protected]>
Co-authored-by: Zamil Majdy <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit afcccb4)
  • Loading branch information
majdyz authored and Mrhs121 committed Apr 17, 2024
1 parent e017b57 commit fb4509f
Showing 1 changed file with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -477,10 +477,10 @@ object SQLConf {
val COLUMN_VECTOR_OFFHEAP_ENABLED =
buildConf("spark.sql.columnVector.offheap.enabled")
.internal()
.doc("When true, use OffHeapColumnVector in ColumnarBatch.")
.doc("When true, use OffHeapColumnVector in ColumnarBatch. " +
s"Defaults to $MEMORY_OFFHEAP_ENABLED.")
.version("2.3.0")
.booleanConf
.createWithDefault(false)
.fallbackConf(MEMORY_OFFHEAP_ENABLED)

val PREFER_SORTMERGEJOIN = buildConf("spark.sql.join.preferSortMergeJoin")
.internal()
Expand Down

0 comments on commit fb4509f

Please sign in to comment.