| spark.gluten.sql.columnar.backend.velox.IOThreads | ⚓ Static | <undefined> | The Size of the IO thread pool in the Connector. This thread pool is used for split preloading and DirectBufferedInput. By default, the value is the same as the maximum task slots per Spark executor. |
| spark.gluten.sql.columnar.backend.velox.SplitPreloadPerDriver | 🔄 Dynamic | 2 | The split preload per task |
| spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinPct | 🔄 Dynamic | 90 | If partial aggregation aggregationPct greater than this value, partial aggregation may be early abandoned. Note: this option only works when flushable partial aggregation is enabled. Ignored when spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation=false. |
| spark.gluten.sql.columnar.backend.velox.abandonPartialAggregationMinRows | 🔄 Dynamic | 100000 | If partial aggregation input rows number greater than this value, partial aggregation may be early abandoned. Note: this option only works when flushable partial aggregation is enabled. Ignored when spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation=false. |
| spark.gluten.sql.columnar.backend.velox.asyncTimeoutOnTaskStopping | ⚓ Static | 30000ms | Timeout in milliseconds when waiting for runtime-scoped async work to finish during teardown. |
| spark.gluten.sql.columnar.backend.velox.cacheEnabled | ⚓ Static | false | Enable Velox cache, default off. It's recommended to enablesoft-affinity as well when enable velox cache. |
| spark.gluten.sql.columnar.backend.velox.cachePrefetchMinPct | ⚓ Static | 0 | Set prefetch cache min pct for velox file scan |
| spark.gluten.sql.columnar.backend.velox.checkUsageLeak | ⚓ Static | true | Enable check memory usage leak. |
| spark.gluten.sql.columnar.backend.velox.cudf.allowCpuFallback | ⚓ Static | true | Allow cuDF to fall back to CPU execution for unsupported operators. |
| spark.gluten.sql.columnar.backend.velox.cudf.batchSize | 🔄 Dynamic | 2147483647 | Cudf input batch size after shuffle reader |
| spark.gluten.sql.columnar.backend.velox.cudf.concurrentGpuTasks | ⚓ Static | 1 | The number of concurrent GPU tasks to run. |
| spark.gluten.sql.columnar.backend.velox.cudf.enableTableScan | ⚓ Static | false | Enable cudf table scan |
| spark.gluten.sql.columnar.backend.velox.cudf.enableValidation | 🔄 Dynamic | true | Heuristics you can apply to validate a cuDF/GPU plan and only offload when the entire stage can be fully and profitably executed on GPU |
| spark.gluten.sql.columnar.backend.velox.cudf.memoryPercent | ⚓ Static | 50 | The initial percent of GPU memory to allocate for memory resource for one thread. |
| spark.gluten.sql.columnar.backend.velox.cudf.memoryResource | ⚓ Static | async | GPU RMM memory resource. |
| spark.gluten.sql.columnar.backend.velox.cudf.shuffleMaxPrefetchBytes | 🔄 Dynamic | 1028MB | Maximum bytes to prefetch in CPU memory during GPU shuffle read while waiting for GPU available. |
| spark.gluten.sql.columnar.backend.velox.directorySizeGuess | ⚓ Static | 32KB | Deprecated, rename to spark.gluten.sql.columnar.backend.velox.footerEstimatedSize |
| spark.gluten.sql.columnar.backend.velox.driverSideBroadcastHashTableBuild | 🔄 Dynamic | false | Enable driver-side broadcast hash table build. When enabled, the hash table is built and serialized on the driver, then broadcast to executors. When disabled, each executor builds its own hash table from the broadcast data. |
| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation | 🔄 Dynamic | false | Enable validation fallback for TimestampNTZ type. When true, any plan containing TimestampNTZ will fall back to Spark execution. When false, allows native execution for TimestampNTZ scan. |
| spark.gluten.sql.columnar.backend.velox.fileHandleCacheEnabled | ⚓ Static | true | Enables caching of open file handles to avoid repeated open/close overhead. Benefits both local filesystems (fewer open/close syscalls and file descriptor churn) and remote filesystems/object stores (reused connection state). Should be disabled if files are mutable, i.e. file content may change while the file path stays the same. |
| spark.gluten.sql.columnar.backend.velox.fileHandleExpirationDurationMs | ⚓ Static | 10m | Expiration time for cached file handles. Handles not accessed within this duration are evicted from the cache. This prevents stale handles from accumulating (e.g., expired HDFS leases, closed remote connections). Accepts a Spark duration string (e.g., "10m", "600s") or a plain number interpreted as milliseconds. A value of 0 disables TTL-based eviction. |
| spark.gluten.sql.columnar.backend.velox.filePreloadThreshold | ⚓ Static | 1MB | Set the file preload threshold for velox file scan, refer to Velox's file-preload-threshold |
| spark.gluten.sql.columnar.backend.velox.floatingPointMode | 🔄 Dynamic | loose | Config used to control the tolerance of floating point operations alignment with Spark. When the mode is set to strict, flushing is disabled for sum(float/double)and avg(float/double). When set to loose, flushing will be enabled. |
| spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation | 🔄 Dynamic | true | Enable flushable aggregation. If true, Gluten will try converting regular aggregation into Velox's flushable aggregation when applicable. A flushable aggregation could emit intermediate result at anytime when memory is full / data reduction ratio is low. |
| spark.gluten.sql.columnar.backend.velox.footerEstimatedSize | ⚓ Static | 32KB | Set the footer estimated size for velox file scan, refer to Velox's footer-estimated-size |
| spark.gluten.sql.columnar.backend.velox.gpuAsyncShuffleReader.enabled | 🔄 Dynamic | false | Experimental: Enable GPU async shuffle reader. When true, the gpu shuffle reader will use a thread pool to read and deserialize the input streams. When false, the shuffle reader will execute in the current thread. |
| spark.gluten.sql.columnar.backend.velox.gpuAsyncShuffleReader.maxPrefetchBytes | 🔄 Dynamic | 1GB | The maximum number of bytes to prefetch in CPU memory during GPU async shuffle read. |
| spark.gluten.sql.columnar.backend.velox.gpuAsyncShuffleReader.threadPoolSize | ⚓ Static | 1 | The number of threads used by GPU async shuffle reader for decompressing and deserializing input streams. |
| spark.gluten.sql.columnar.backend.velox.hashProbe.bloomFilter.bypassMinPct | 🔄 Dynamic | 85 | Bypass the build-side Bloom filter when its acceptance percentage reaches this value. |
| spark.gluten.sql.columnar.backend.velox.hashProbe.bloomFilter.bypassMinRows | 🔄 Dynamic | 0 | Number of probe rows used to decide whether to bypass the build-side Bloom filter for left outer, existence, and left anti joins. |
| spark.gluten.sql.columnar.backend.velox.hashProbe.bloomFilterPushdown.maxSize | 🔄 Dynamic | 0b | The maximum byte size of Bloom filter that can be generated from hash probe. When set to 0, no Bloom filter will be generated. To achieve optimal performance, this should not be too larger than the CPU cache size on the host. |
| spark.gluten.sql.columnar.backend.velox.hashProbe.dynamicFilterPushdown.enabled | 🔄 Dynamic | true | Whether hash probe can generate any dynamic filter (including Bloom filter) and push down to upstream operators. |
| spark.gluten.sql.columnar.backend.velox.hashShuffle.reader.streamMerge.enabled | 🔄 Dynamic | false | Enables a reader-side raw payload merge fast path for plain hash shuffle payloads within each shuffle input stream. This path merges payload buffers before Velox vectors are materialized, so it has lower per-batch overhead than generic VeloxResizeBatchesExec resizing, but it only covers plain payloads. Complex types and dictionary-encoded payloads are not merged by this path. VeloxResizeBatchesExec can still be enabled separately as a generic complement for types and encodings not covered by this fast path. If false, each hash shuffle payload is returned as its own columnar batch. |
| spark.gluten.sql.columnar.backend.velox.loadQuantum | ⚓ Static | 256MB | Set the load quantum for velox file scan, recommend to use the default value (256MB) for performance consideration. If Velox cache is enabled, it can be 8MB at most. |
| spark.gluten.sql.columnar.backend.velox.maxCoalescedBytes | ⚓ Static | 64MB | Set the max coalesced bytes for velox file scan |
| spark.gluten.sql.columnar.backend.velox.maxCoalescedDistance | ⚓ Static | 512KB | Set the max coalesced distance bytes for velox file scan |
| spark.gluten.sql.columnar.backend.velox.maxCompiledRegexes | 🔄 Dynamic | 100 | Controls maximum number of compiled regular expression patterns per function instance per thread of execution. |
| spark.gluten.sql.columnar.backend.velox.maxExtendedPartialAggregationMemory | 🔄 Dynamic | <undefined> | Set the max extended memory of partial aggregation in bytes. When this option is set to a value greater than 0, it will override spark.gluten.sql.columnar.backend.velox.maxExtendedPartialAggregationMemoryRatio. Note: this option only works when flushable partial aggregation is enabled. Ignored when spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation=false. |
| spark.gluten.sql.columnar.backend.velox.maxExtendedPartialAggregationMemoryRatio | 🔄 Dynamic | 0.15 | Set the max extended memory of partial aggregation as maxExtendedPartialAggregationMemoryRatio of offheap size. Note: this option only works when flushable partial aggregation is enabled. Ignored when spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation=false. |
| spark.gluten.sql.columnar.backend.velox.maxPartialAggregationMemory | 🔄 Dynamic | <undefined> | Set the max memory of partial aggregation in bytes. When this option is set to a value greater than 0, it will override spark.gluten.sql.columnar.backend.velox.maxPartialAggregationMemoryRatio. Note: this option only works when flushable partial aggregation is enabled. Ignored when spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation=false. |
| spark.gluten.sql.columnar.backend.velox.maxPartialAggregationMemoryRatio | 🔄 Dynamic | 0.1 | Set the max memory of partial aggregation as maxPartialAggregationMemoryRatio of offheap size. Note: this option only works when flushable partial aggregation is enabled. Ignored when spark.gluten.sql.columnar.backend.velox.flushablePartialAggregation=false. |
| spark.gluten.sql.columnar.backend.velox.maxPartitionsPerWritersSession | 🔄 Dynamic | 10000 | Maximum number of partitions per a single table writer instance. |
| spark.gluten.sql.columnar.backend.velox.maxSpillBytes | 🔄 Dynamic | 100G | The maximum file size of a query |
| spark.gluten.sql.columnar.backend.velox.maxSpillFileSize | 🔄 Dynamic | 1GB | The maximum size of a single spill file created |
| spark.gluten.sql.columnar.backend.velox.maxSpillLevel | 🔄 Dynamic | 4 | The max allowed spilling level with zero being the initial spilling level |
| spark.gluten.sql.columnar.backend.velox.maxSpillRunRows | 🔄 Dynamic | 3M | The maximum row size of a single spill run |
| spark.gluten.sql.columnar.backend.velox.memCacheSize | ⚓ Static | 1GB | The memory cache size |
| spark.gluten.sql.columnar.backend.velox.memInitCapacity | 🔄 Dynamic | 8MB | The initial memory capacity to reserve for a newly created Velox query memory pool. |
| spark.gluten.sql.columnar.backend.velox.memoryPoolCapacityTransferAcrossTasks | 🔄 Dynamic | true | Whether to allow memory capacity transfer between memory pools from different tasks. |
| spark.gluten.sql.columnar.backend.velox.memoryUseHugePages | 🔄 Dynamic | false | Use explicit huge pages for Velox memory allocation. |
| spark.gluten.sql.columnar.backend.velox.numCacheFileHandles | ⚓ Static | 10000 | Maximum number of entries in the file handle cache. Each entry holds an open file descriptor (local FS) or connection state (remote FS). Note that on local filesystems, high values may approach the OS file descriptor limit (ulimit -n). On remote object stores (S3, ABFS, GCS) entries represent network connections/sockets rather than per-file OS file descriptors, but they can still count toward OS resource limits (ulimit -n). |
| spark.gluten.sql.columnar.backend.velox.orc.scan.enabled | 🔄 Dynamic | true | Enable velox orc scan. If disabled, vanilla spark orc scan will be used. |
| spark.gluten.sql.columnar.backend.velox.parquet.dictionaryPageSizeBytes | 🔄 Dynamic | 2MB | The maximum size in bytes for a Parquet dictionary page |
| spark.gluten.sql.columnar.backend.velox.parquet.pageSizeBytes | 🔄 Dynamic | 1MB | The page size in bytes is for compression. |
| spark.gluten.sql.columnar.backend.velox.parquetMaxTargetFileSize | 🔄 Dynamic | 0b | The target file size for each output file when writing data. 0 means no limit on target file size, and the actual file size will be determined by other factors such as max partition number and shuffle batch size. |
| spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames | 🔄 Dynamic | true | Maps table field names to file field names using names, not indices for Parquet files. |
| spark.gluten.sql.columnar.backend.velox.prefetchRowGroups | ⚓ Static | 1 | Set the prefetch row groups for velox file scan |
| spark.gluten.sql.columnar.backend.velox.queryTraceEnabled | 🔄 Dynamic | false | Enable query tracing flag. |
| spark.gluten.sql.columnar.backend.velox.reclaimMaxWaitMs | 🔄 Dynamic | 3600000ms | The max time in ms to wait for memory reclaim. |
| spark.gluten.sql.columnar.backend.velox.resizeBatches.copyRanges.enabled | 🔄 Dynamic | true | Enables a VeloxResizeBatchesExec fast path that combines eligible batches using Velox vector copyRanges instead of generic RowVector append. When possible, it collects the small input batches for one VeloxResizeBatchesExec output, allocates the output RowVector once, and bulk-copies child vector ranges. This is most useful for shuffle-read outputs where plain hash shuffle payloads are materialized as dense flat vectors. Complex vectors can also use copyRanges, but ARRAY and MAP still rebuild nested offsets and sizes while bulk-copying child ranges. Unsupported encodings such as dictionary and constant vectors fall back to the generic copy path. This option is enabled by default and complements the reader-side raw payload merge fast path: that path avoids materializing small plain payload batches, while this option optimizes VeloxResizeBatchesExec when that operator is enabled. |
| spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInput | 🔄 Dynamic | true | If true, combine small columnar batches together before sending to shuffle. The default minimum output batch size is equal to 0.25 * spark.gluten.sql.columnar.maxBatchSize |
| spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInput.minSize | 🔄 Dynamic | <undefined> | The minimum batch size for shuffle. If size of an input batch is smaller than the value, it will be combined with other batches before sending to shuffle. Only functions when spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInput is set to true. Default value: 0.25 * |
| spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInputOutput.minSize | 🔄 Dynamic | <undefined> | The minimum batch size for shuffle input and output. If size of an input batch is smaller than the value, it will be combined with other batches before sending to shuffle. The same applies for batches output by shuffle read. Only functions when spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleInput or spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleOutput is set to true. Default value: 0.25 * |
| spark.gluten.sql.columnar.backend.velox.resizeBatches.shuffleOutput | 🔄 Dynamic | false | If true, combine small columnar batches together right after shuffle read. The default minimum output batch size is equal to 0.25 * spark.gluten.sql.columnar.maxBatchSize |
| spark.gluten.sql.columnar.backend.velox.scan.bloomFilterPushdown.enabled | ⚓ Static | false | Whether to push Bloom filters into Velox scans. |
| spark.gluten.sql.columnar.backend.velox.showTaskMetricsWhenFinished | 🔄 Dynamic | false | Show velox full task metrics when finished. |
| spark.gluten.sql.columnar.backend.velox.spillFileSystem | 🔄 Dynamic | local | The filesystem used to store spill data. local: The local file system. heap-over-local: Write file to JVM heap if having extra heap space. Otherwise write to local file system. |
| spark.gluten.sql.columnar.backend.velox.spillNumMaxMergeFiles | 🔄 Dynamic | 0 | The max number of files to merge at a time when merging sorted files into a single ordered stream. 0 means unlimited. |
| spark.gluten.sql.columnar.backend.velox.spillStrategy | 🔄 Dynamic | auto | none: Disable spill on Velox backend; auto: Let Spark memory manager manage Velox's spilling |
| spark.gluten.sql.columnar.backend.velox.ssdCacheIOThreads | ⚓ Static | 4 | The number of IO threads for SSD cache read/write operations |
| spark.gluten.sql.columnar.backend.velox.ssdCachePath | ⚓ Static | /tmp | The folder to store the cache files, better on SSD |
| spark.gluten.sql.columnar.backend.velox.ssdCacheShards | ⚓ Static | 1 | The cache shards |
| spark.gluten.sql.columnar.backend.velox.ssdCacheSize | ⚓ Static | 1GB | The SSD cache size, will do memory caching only if this value = 0 |
| spark.gluten.sql.columnar.backend.velox.ssdCheckpointIntervalBytes | ⚓ Static | 0 | Checkpoint after every 'checkpointIntervalBytes' for SSD cache. 0 means no checkpointing. |
| spark.gluten.sql.columnar.backend.velox.ssdChecksumEnabled | ⚓ Static | false | If true, checksum write to SSD is enabled. |
| spark.gluten.sql.columnar.backend.velox.ssdChecksumReadVerificationEnabled | ⚓ Static | false | If true, checksum read verification from SSD is enabled. |
| spark.gluten.sql.columnar.backend.velox.ssdDisableFileCow | ⚓ Static | false | True if copy on write should be disabled. |
| spark.gluten.sql.columnar.backend.velox.ssdODirect | ⚓ Static | false | The O_DIRECT flag for cache writing |
| spark.gluten.sql.columnar.backend.velox.valueStream.dynamicFilter.enabled | 🔄 Dynamic | false | Whether to apply dynamic filters pushed down from hash probe in the ValueStream (shuffle reader) operator to filter rows before they reach the hash join. |
| spark.gluten.sql.enable.enhancedFeatures | 🔄 Dynamic | true | Enable some features including iceberg native write and other features. |
| spark.gluten.sql.rewrite.castArrayToString | 🔄 Dynamic | true | When true, rewrite cast(array as String) to concat('[', array_join(array, ', ', null), ']') to allow offloading to Velox. |
| spark.gluten.velox.broadcast.build.targetBytesPerThread | ⚓ Static | 32MB | It is used to calculate the number of hash table build threads. Based on our testing across various thresholds (1MB to 128MB), we recommend a value of 32MB or 64MB, as these consistently provided the most significant performance gains. |
| spark.gluten.velox.broadcastBuild.mergeBatches | 🔄 Dynamic | false | If enabled, all columnar batches in a broadcast build relation will be serialized into a single buffer to reduce the number of addInput calls in HashBuild operator. This can significantly improve BHJ performance when the broadcast table has many small batches, but may increase driver-side peak memory and is not suitable for very large broadcasts. |
| spark.gluten.velox.castFromVarcharAddTrimNode | 🔄 Dynamic | false | If true, will add a trim node which has the same semantic as vanilla Spark to CAST-from-varchar.Otherwise, do nothing. |
| spark.gluten.velox.decimalToFloatHighPrecisionCastEnabled | 🔄 Dynamic | false | If true, enables high-precision casts from DECIMAL to REAL/DOUBLE in Velox, which match vanilla Spark for values that cannot be represented exactly by floating-point arithmetic. Disabled by default because it is slower than the default conversion; enable it if precision matters more than throughput. |
| spark.gluten.velox.s3MaxConcurrentUploadNum | ⚓ Static | 4 | The maximum number of in-flight S3 part uploads per file. |
| spark.gluten.velox.s3UploadPartAsync | ⚓ Static | false | If true, S3 multipart upload parts are uploaded asynchronously. |
| spark.gluten.velox.s3UploadThreads | ⚓ Static | 16 | The number of shared S3 part upload threads. |