[FEA] Explore adding Vortex file format support to cuDF

[0ax1]

Context

Vortex is an extensible, state-of-the-art framework for columnar compression and the fastest FOSS columnar file format. Formerly at @spiraldb, Vortex is now an Incubation Stage project at LFAI & Data, part of the Linux Foundation.

Vortex provides GPU-native decoding and exports GPU-resident arrays through the Arrow C Device Interface. Existing interoperability tests already exercise this path with cuDF.

This issue is intended to work with the cuDF maintainers on a rough integration path. The APIs and build integration below are starting points for discussion rather than a finalized design.

Possible Python API

df = cudf.read_vortex(
    "dataset.vortex",
    columns=["id", "timestamp", "value"],
    filters=[("timestamp", ">=", cutoff)],
)

The reader would return a cudf.DataFrame directly, support projection and predicate pushdown, and avoid staging decoded data through host-backed Arrow, pandas, or NumPy.

Proposed integration direction

Vortex C++ layer

Vortex is primarily implemented in Rust and does not yet expose the full C++ reader API needed by libcudf. We expect to develop this API under lang/cpp, likely backed by a stable C ABI.

The existing CMake configuration in lang/cpp is currently bare-bones and would be expanded to:

• Build and link the required Vortex Rust and CUDA components.
• Expose installable CMake targets consumable by cuDF/RAPIDS.
• Support Linux x86_64 and aarch64.
• Provide file reading, projection, predicate pushdown, and CUDA stream handling.
• Export ArrowDeviceArray and ArrowSchema with explicit ownership semantics.

The exact packaging and dependency mechanism should be agreed on with the cuDF maintainers.

libcudf

A libcudf integration could add:

• cudf::io::vortex_reader_options
• cudf::io::read_vortex

The reader would pass projection and predicates to Vortex, then import the resulting Arrow device arrays directly into libcudf. It should preserve schema metadata, respect cuDF stream and RMM semantics, and avoid host materialization.

Python cuDF

The libcudf reader could be exposed through pylibcudf/Cython as cudf.read_vortex, following existing cuDF I/O conventions for columns=, filters=, and applicable storage options.

Questions for cuDF maintainers

• Is the Arrow C Device Interface an appropriate boundary between Vortex and libcudf?
• What libcudf reader API shape would fit existing I/O conventions?
• How should Vortex be discovered and built through RAPIDS/CMake?
• What are the expected CUDA stream, RMM, ownership, and fallback semantics?
• What would be a reasonable first integration milestone?

Existing GPU support

Vortex has CUDA execution for ALP, BitPacked/FastLanes, DateTimeParts, DecimalByteParts, Delta, Dictionary, Frame of Reference, FSST, OnPair, RunEnd, Sequence, ZigZag, and Zstandard via nvCOMP. It also supports constant, list, masked, and shared arrays, plus filter and slice operations.

Performance

CUDA kernel microbenchmarks on an NVIDIA GH200, using Vortex commit 1fe8dda:

• BitPacked: 1,298–2,711 GiB/s.
• RunEnd: up to 1,931 GiB/s.
• Dictionary: 1,442–2,265 GiB/s.
• ALP: 1,372 GiB/s for f32 and 1,629 GiB/s for f64.
• FSST: 248–281 GiB/s across decoding and Arrow C Device export.

These are kernel benchmarks, not end-to-end file-read results.

[0ax1]

cc: @mhaseeb123

[GregoryKimball]

Thank you @0ax1 for sharing this proposal. It sounds like you have good decompression and decoding pathways worked out. How are you approaching IO and HtoD copying? Presumably the integration of cuDF data sources into Vortex would be difficult. I assume Vortex readers support file-like sources, but how about host buffer, device buffer, remote objects?

One good starting point for potential cuDF integration would be to show a Vortex reader option in the libcudf-ndsh benchmarks. You could insert a write_vortex, read_vortex pair in the microbenchmark and directly compare with parquet. Nsight Systems profiles at SF100 would be my first target. Would you like to put a [POC] PR together?