Skip to content

misc: Sync upstream Velox main - #65

Draft
firestarman wants to merge 289 commits into
devfrom
sync/20260818-54fea71cce67
Draft

misc: Sync upstream Velox main#65
firestarman wants to merge 289 commits into
devfrom
sync/20260818-54fea71cce67

Conversation

@firestarman

@firestarman firestarman commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Merge summary

  • Upstream range: 796c6214c89a78ef86df29a64d4351129f62cb86..d13ef1e303af6a12cfb770f8144fad49594bd5ba
  • Original sync base: open PR head f95611c3ee9d3556e2a600aaf1faef20d487aba9
  • Recovered unpublished range: none
  • Upstream integration: merge commit 2cb8ace70220dc8a151816ab1412ea472b6a1202
  • Latest downstream dev: freshly fetched cf619d90eca1b83ad31f446b96482df096d24923 was already contained; no second merge commit was required.
  • Final Velox commit: 2cb8ace70220dc8a151816ab1412ea472b6a1202
  • .github: retained exactly from the recorded latest downstream/dev; upstream .github changes were ignored.

Notable changes

  • Torchwave adds per-column decomposition and chain-gather folding, a write/read barrier fix, fused sparse flip-and-truncate support, a standalone Triton masked_select_jagged, and .pt2 flat-input loading.
  • Nimble vectorizes ALP encoding and grid search, hardens FSST decode bounds and symbol isolation, synchronizes encoding-pool callbacks, skips index-projection stripes with no projected streams, structurally slices partial BlockBitPacking blocks, and exposes writer options through SerDe keys.
  • Spark SQL adds AverageAggregate::toIntermediate and btrim; vector decoding exposes whether buffers are borrowed or owned.
  • Connector downcasts and table-scan plan text were improved, and deprecated gMock Invoke wrappers were removed. This range has no net experimental/cudf changes.
  • No downstream Velox compatibility commit was needed after the merge. NVIDIA/spark-gluten#172 remains required to adapt Spark-Gluten's plan converter to the writer and Unnest APIs present at this Velox head.

Conflict resolutions

  • No manual Git conflicts.
  • .github was restored and verified against the recorded baseline after the upstream merge, per downstream-ownership policy.

Validation

  • Status: BLOCKED_ON_SPARK_GLUTEN_FIX — smoke and full Colt validation pass against the unmerged fix below; this Velox PR remains Draft until that fix lands in Spark-Gluten dev and validation is rerun from scratch against dev.
  • Spark-Gluten dev: 9f732196834cb3d1f67488dd1d2f93823802dc7a
  • Spark-Gluten fix: NVIDIA/spark-gluten#172, tested SHA df462d247c8f5193d578ee8f8242d1a8c909748e (Ready).
  • Colt: 73f99f12c9f0fe067f48a667a68e0c52e4dbd832
  • Verification config: ebb16dcd6fcb238b34d39cd00a2eb46a9525d5ec; generated manifest 9b45526e6ac686c3fb1a95fdb2f94f1ae48e90b7113de4dcfe80d58f5835e338; provenance at /home/liangcail/velox-upstream-sync-20260814/results/20260907T041509Z/verify-config/provenance.json (SHA-256 e28bcbb0f83e7c716050f5468e377e3ddc8878276d763ca0b78ba5f139a0732d).
  • Initial current-dev failure: build-libs, return code 2. InsertTableHandle required notNullColumns, and UnnestNode required optional output names. Evidence: /home/liangcail/velox-upstream-sync-20260814/results/20260907T033307Z/smoke/verify/workflow/logs/10-build-libs/spark-gluten-build-libs.log.
  • Repair: Spark-Gluten df462d247c8f5193d578ee8f8242d1a8c909748e supplies an empty notNullColumns set to preserve existing behavior and uses vector<optional<string>> for Unnest names. git diff --check, both independent clean consumer builds, and all downstream tests passed.
  • Smoke, TPC-H Query 1: PASS, 361 source/native cases recorded (235 passed, 126 suite-declared skips) and Query 1 passed; /home/liangcail/velox-upstream-sync-20260814/results/20260907T041509Z/smoke/verify.
  • Full, TPC-H Query 1–22: PASS, 361 source/native cases recorded (235 passed, 126 suite-declared skips) and all 22 queries passed; /home/liangcail/velox-upstream-sync-20260814/results/20260907T041509Z/full/verify.
  • Validation receipt: /home/liangcail/velox-upstream-sync-20260814/results/20260907T041509Z/sync-validation.json.
  • CUDA/GPU/images: CUDA 12.9; requested native, detected SM89 on GPU 0 (NVIDIA L4); driver mode auto resolved to host; cuDF 5beaa5954688fcb12236ffb434e192ea2c77db30; sealed dependency carrier sha256:b52df2b7521e0fb00e0ec14a3cf7428c7bbd8dba0caba037c5d308c9c19b3094; smoke/full build execution images sha256:b13917dd0d88e30d94aab058e18e754e419348c3f679c26cf05f8583f71a3782 and sha256:e2724244a0268243dfc11747b45c29d6c45ddfac5f5f30ea99769328e3c8828a; smoke/full runtime image sha256:1b42a229c8357223f5c3bdc06dceee2ddddf6c3d0e692db1b5a8b8b074ffaaa3.

jainxrohit and others added 30 commits August 11, 2026 20:31
Differential Revision: D115607344

Pull Request resolved: facebookincubator#18470
Differential Revision: D115654162

Pull Request resolved: facebookincubator#18469
Differential Revision: D112929824

Pull Request resolved: facebookincubator#18075
Differential Revision: D115767319

Pull Request resolved: facebookincubator#18285
Differential Revision: D115766716

Pull Request resolved: facebookincubator#18480
Differential Revision: D115604323

Pull Request resolved: facebookincubator#18479
Differential Revision: D115655249

Pull Request resolved: facebookincubator#18415
Differential Revision: D115255896

Pull Request resolved: facebookincubator#13907
Differential Revision: D115817317

Pull Request resolved: facebookincubator#18497
Differential Revision: D115766742

Pull Request resolved: facebookincubator#18458
Differential Revision: D115767508

Pull Request resolved: facebookincubator#18074
Differential Revision: D115807551

Pull Request resolved: facebookincubator#18438
Differential Revision: D115766774

Pull Request resolved: facebookincubator#18455
… check

Differential Revision: D115891195

Pull Request resolved: facebookincubator#18498
Differential Revision: D113550897

Pull Request resolved: facebookincubator#17391
Differential Revision: D115932280

Pull Request resolved: facebookincubator#18489
…spec

Differential Revision: D115933127

Pull Request resolved: facebookincubator#18477
Differential Revision: D115939177

Pull Request resolved: facebookincubator#18382
Differential Revision: D115933828

Pull Request resolved: facebookincubator#17219
Differential Revision: D115932182

Pull Request resolved: facebookincubator#18130
…eader

Differential Revision: D115933709

Pull Request resolved: facebookincubator#18383
PingLiuPing and others added 30 commits September 3, 2026 19:30
Differential Revision: D118496590

Pull Request resolved: facebookincubator#18779
Differential Revision: D118628694

Pull Request resolved: facebookincubator#18830
Differential Revision: D118563902

Pull Request resolved: facebookincubator#18823
Differential Revision: D118371081

Pull Request resolved: facebookincubator#11886
Differential Revision: D118338069

Pull Request resolved: facebookincubator#17931
Differential Revision: D118526897

Pull Request resolved: facebookincubator#18806
Differential Revision: D118212503

Pull Request resolved: facebookincubator#18561
Differential Revision: D118593412

Pull Request resolved: facebookincubator#18835
Differential Revision: D118750417

Pull Request resolved: facebookincubator#18843
…ad of repacking

Differential Revision: D117309777

Pull Request resolved: facebookincubator#18785
Differential Revision: D118543707

Pull Request resolved: facebookincubator#18567
Differential Revision: D117810561

Pull Request resolved: facebookincubator#18839
…ested

Differential Revision: D118061297

Pull Request resolved: facebookincubator#18857
Differential Revision: D118949931

Pull Request resolved: facebookincubator#18858
Differential Revision: D118983416

Pull Request resolved: facebookincubator#18870
Differential Revision: D118978729

Pull Request resolved: facebookincubator#18869
Differential Revision: D118984752

Pull Request resolved: facebookincubator#18808
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.