November 07, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days saw 35 merged PRs and 25 open PRs, with heavy focus on Velox backend stability, Spark 4.0 readiness, and daily upstream Velox syncs. Memory-management fixes, shuffle optimizations, and component-system hardening dominated the bug-fix queue, while new GCC-13 CI images and C++20 standardization signal infrastructure modernization.

Key Ongoing Projects

• Spark 4.0 support – @zhouyuan landed [GLUTEN-10207][VL] adding spark40 unit test framework #10725 (55 k lines, 350 files) adding the first comprehensive unit-test suite for Spark 4.0; [MINOR] Add Gluten-1.5 and spark-4.0.x option to bug issue template #11019 updates issue templates to track Spark-4.0.x bugs.
• Component-system hardening – @PHILO-HE and @zhztheplayer closed [CORE] Exclude dependent components of incompatible components #11009, [CORE] Add a test case where dependencies of a component are not registered #11006, [CORE] ComponentSuite: Add a test case against dependency cycles #11008 to exclude incompatible components at runtime and guard against cyclic dependencies.
• Daily Velox up-streaming – @GlutenPerfBot continues mechanical daily bumps ([GLUTEN-6887][VL] Daily Update Velox Version (2025_11_07) #11047, [GLUTEN-6887][VL] Daily Update Velox Version (2025_11_07) #11044, [GLUTEN-6887][VL] Daily Update Velox Version (2025_11_06) #11039, [GLUTEN-6887][VL] Daily Update Velox Version (2025_11_05) #11030, [GLUTEN-6887][VL] Daily Update Velox Version (2025_11_04) #11014, [GLUTEN-6887][VL] Daily Update Velox Version (2025_11_03) #11000, [GLUTEN-6887][VL] Daily Update Velox Version (2025_11_01) #10996) keeping Gluten in lock-step with facebookincubator/velox.
• GCC-13 / C++20 readiness – @zhouyuan added GCC-13 CI image ([GLUTEN-10926][VL] Add GCC-13 image for CI #10993) and @PHILO-HE standardized dependency builds on C++20 ([VL] Configure C++20 standard for dependency libraries #11005) ahead of Velox’s impending toolchain upgrade.
• GPU & cuDF acceleration – @jinchengchenghh opened [VL] Support GPU shuffle reader #10933 and [GLUTEN-9785][VL] Pass additional configuration options programmatically from Java to C++ #11032 to plumb GPU shuffle and enlarge GPU batches up to 1 GB.

Priority Items

• Spill-level regression – [VL] "Spill level 4 exceeds max spill level 3" error #11018 (open) reports “Spill level 4 exceeds max spill level 3” causing query failure; needs Velox-side fix.
• Uniffle shuffle metrics – [VL][UNIFFLE] NoSuchElementException on uniffle sort shuffle writer #11037 (open) NPE in sort-shuffle writer; @wForget has a patch under review ([GLUTEN-11037][VL] Corrected logic for updating uniffle shuffle metrics #11038).
• ARM CI instability – [CH] ARM CI is not stable #11052 (open) JDK-11 mismatch breaks Iceberg builds on ARM; blocking ClickHouse merges.
• Small-file coalesce skew – Regenerate input partitions for small files #11050 (open) can create straggler tasks; @marin-ma opened [GLUTEN-11050] Regenerate input partitions for small files #11051 to distribute small files evenly.
• ORC schema evolution – long-standing [VL] Read OrcFile error when schema in Orc file and the table file don't consist #5638 (open) breaks Hive ORC reads; @ccat3z’s [VL] Support read old ORC file without column names #8862 is merged but needs follow-up for footer compatibility.

Notable Discussions

• SSD cache + O_DIRECT problem #11054 – SSD cache + O_DIRECT alignment problem on 4 KB block devices; seeking config to guarantee alignment.
• Rethinking component management: backend as always-present exclusive base layer #11034 – @PHILO-HE proposes treating backend as an always-present exclusive base layer to simplify component management.
• Add a new backend: Bolt #10929 – @WangGuangxin introduces “Bolt”, a ByteDance Velox fork with JIT and OOM hardening, asking for guidance on upstreaming it as a new Gluten backend.
• How to dynamically switch between native Spark and Gluten backend based on the execution plan? #10954 – community brainstorm on dynamically switching between native Spark and Gluten backends per query plan.

Emerging Trends

• Infrastructure modernization – GCC-13 images, C++20 builds, clang-tidy enforcement ([GLUTEN-11027] Use clang-tidy to check cpp files #11028) show a push toward production-grade tooling.
• GPU-first features – daily cuDF patches, GPU shuffle reader, and 1-GB batch support signal a shift from CPU-only Velox.
• Component-aware classpath – duplicate-resource warnings ([CORE] Avoid getting duplicate resource files for component discovery #11043) and runtime-compatibility checks ([CORE] Follow-up: switch to direct class references guarded by component's runtime-compatibility validation #10999) make Gluten safer in fat-JAR deployments.
• Query-plan robustness – multiple fixes for ColumnarPartialProjectExec partitioning ([VL] Implement outputPartitioning for ColumnarPartialProjectExec #11048, [GLUTEN-11048][VL] Implement OrderPreservingNode/PartitioningPreservingNode for ColumnarPartialProjectExec #11049) harden AQE support.

Good First Issues

• [CH] support expression MakeYMInterval #6814 – Add ClickHouse expression MakeYMInterval; pure CH backend, no Velox/C++ needed.
• [CH] support function date_from_unix_date  #4730 – Implement date_from_unix_date for ClickHouse; follow the pattern of existing date functions.
• [CH] support split_part function #6807 – Add split_part string function for ClickHouse; well-scoped, CH-only.
• [CH] support function SparkPartitionID  #6812 – Implement SparkPartitionID for ClickHouse; single-row function, good intro to CH function registry.

All issues above are labeled “good first issue”, touch only the ClickHouse backend, and have clear function signatures—ideal entry points for new contributors comfortable with Java/Scala or C++ in the ClickHouse context.