December 05, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days have been exceptionally busy with 57 merged PRs and 21 open ones, showing strong momentum toward the next release. Spark 4.0 compatibility work dominates the queue, while infrastructure improvements (CI disk-space, vcpkg cleanup, Folly bump) keep the build healthy. A new “Bolt” backend prototype (#11261) appeared, and Velox continues to advance with daily version bumps and memory-management fixes.

Key Ongoing Projects

• Spark 4.0 compatibility push – led by @zhouyuan, @marin-ma, @jinchengchenghh, @zml1206 and many others; ~40 test suites left to green-light ([VL] Track on Spark-4.0 failed unit tests #11088)
• Delta Lake write enhancements – @zhztheplayer added overwrite-mode support ([VL] Delta: Support overwrite mode in write #11226) and fixed whole-stage ID generation ([GLUTEN-11251] Fix incorrect whole stage id in WholeStageTransformerExec #11252)
• Memory & stability – @rui-mo landed Velox memory-manager lifecycle fix ([GLUTEN-9456][VL] Release Velox memory manager when the reservedBytes become empty #11249); @zhztheplayer disabled Parquet metadata validation by default to avoid regression ([VL] Disable Parquet metadata validation by default due to performance regression #11233)
• Flink-on-Gluten expansion – @KevinyhZou wired Kafka connector ([GLUTEN-9553][FLINK] Support kafka connector for data source #9554) and fixed JSON parse-error handling ([GLUTEN-10798][CH] Fix json read can not ignore parse errors #10799)
• New backend experiment – @WangGuangxin opened a 1 000-file draft for “Bolt” backend (WIP: add bolt backend in gluten #11261)

Priority Items

1. [VL] Track on Spark-4.0 failed unit tests #11088 – Spark 4.0 test-suite tracker; needs volunteers for remaining suites (CSV, Parquet, Delta merge, streaming, etc.)
2. WIP: add bolt backend in gluten #11261 – Bolt backend PR is huge (264 k LOC); early design review requested
3. [GLUTEN-11169][VL] Trigger GC before throwing OOM to ensure unused off-heap broadcasted relations are correctly released #11236 – GC-before-OOM patch by @zhztheplayer to curb off-heap broadcast leaks; under active review
4. [DNM] TEST BHJ #11255 – “DNM TEST BHJ” draft by @zhouyuan; appears to hold critical broadcast-hash-join experiment
5. Gluten + Velox fallback when reading Delta Lake (Spark 3.5.3 + Delta 3.3.2) #11254 – Community discussion on Delta fallback; needs feedback from @zhztheplayer or @jinchengchenghh

Notable Discussions

• Gluten + Velox fallback when reading Delta Lake (Spark 3.5.3 + Delta 3.3.2) #11254 – Users seeing fallback to Spark when reading Delta 3.3.2; root cause is “UnknownFormat” for JSON manifest files and missing UDF registration
• Gluten Slack Channel #incubator-gluten #8429 – Slack channel #incubator-gluten invitation link refreshed; 15 comments show steady interest

Emerging Trends

• Spark 4.0 readiness is now the gating item for the next release; almost every merged PR carries the [VL] Track on Spark-4.0 failed unit tests #11088 tag
• CI hygiene is a parallel theme: disk-space fixes, Celeborn clean images, vcpkg pruning, clang-tidy introduction
• Small-file handling optimizations ([GLUTEN-11050] Regenerate input partitions for small files #11051, [GLUTEN-11231][CORE] Improve regeneratePartition for PartitionsUtil #11232) indicate growing production workloads on object stores
• Function-parity work accelerates (div, map_from_arrays, split_part, etc.) as users migrate complex SQL

Good First Issues

• [CH] support expression MakeYMInterval #6814 – Add ClickHouse expression MakeYMInterval; pure CH-side C++ implementation
• [CH] support function date_from_unix_date  #4730 – Implement date_from_unix_date for CH backend; good introduction to CH function registry
• [CH] support split_part function #6807 – Wire CH split_part string function; includes unit-test template
• [CH] support function SparkPartitionID  #6812 – Expose SparkPartitionID() in CH; requires reading Spark partition info
• [CH] support expression MapZipWith #6815 – Add MapZipWith higher-order function to CH; great for learning CH’s lambda infrastructure

All CH function tasks above are self-contained, come with existing examples, and need only basic C++ & function-registration knowledge—perfect for first-time contributors!