v0.4.0

The headline themes for this release are a security-hardening sweep of the file-format
parser, a public-API cleanup of the Array hierarchy (the heap-allocated buffer(int) /
segment() accessors are gone from the interface), and cascading writer features that
close the compression gap with the Rust reference implementation on real-world workloads.

Security

Every malformed input now surfaces as VortexException rather than a JDK exception
(IndexOutOfBoundsException, ArrayIndexOutOfBoundsException, StackOverflowError, raw
FlatBuffer/Protobuf runtime exceptions). Regression suite lives under
reader/src/test/java/.../*SecurityTest.

• Zip-bomb protection — ConstantEncoding and dict-layout decode no longer pre-allocate
  O(rowCount) memory; a 150-byte crafted file claiming 10⁹ rows is now constant-cost.
  (10a7776)
• Trailer + postscript validation — VortexReader and VortexHttpReader reject unknown
  file version, postscriptLen == 0, and postscriptLen > fileSize - 8. Footer/layout/dtype
  blob offsets and Layout.encoding index are bounds-checked at parse time.
  (f8f89fe)
• Footer segmentSpecs bounds — every spec is validated against fileSize the moment
  the footer is materialised, eliminating later IndexOutOfBoundsException on
  MemorySegment.asSlice.
  (03845ac)
• PType ordinal bounds-check — PType.fromOrdinal(int) replaces all 22 PType.values()[idx]
  call sites across encodings; crafted Protobuf ptype fields are rejected up front.
  (b4988c3)
• Layout-tree depth cap — PostscriptParser.convertLayout is capped at depth 64,
  preventing both unbounded nesting and self-referential FlatBuffer cycles (a ~120-byte
  cycle attack previously triggered StackOverflowError).
  (29adbe0)
• Layout metadata size cap — per-layout metadataAsByteBuffer() is capped at 4 MiB
  (above any real encoding's footprint; FSST's symbol table is the largest at ~32 KiB).
  (ebbe644)
• Decimal field validation — DType.Decimal is rejected unless precision ∈ [1, 38]
  and scale ∈ [0, precision], matching IEEE 754-2008 decimal128.
  (ebbe644)
• readFlatStats bounds-check — zone-map stats reads now validate the trailing
  little-endian fbLen field against the segment size, returning empty stats on malformed
  input rather than throwing IndexOutOfBoundsException from MemorySegment.asSlice.
  (ebbe644)

Added

• vortex.sequence F16 encode/decode — half-precision floats now round-trip through the
  sequence encoding.
  (7b3d7a9)
• Writer: cascading with global dict layout — low-cardinality columns (≤ 256 distinct
  values in the chunk sample) are now emitted as a vortex.dict layout, with the dict
  candidate detection tightened to avoid false positives.
  (53b2a19,
  d383765)
• Writer: opt-in Zstd compression — WriteOptions.withZstd(boolean) exposes the
  size/throughput trade-off. Off by default; turn on for archival workloads.
  (ea10d37)
• Encoding.decodeSegment extension point — added as part of the typed-segment migration
  (see Changed below). Provides a typed alternative to Array.segment().
  (ed4a0ae)
• DecodeContext.decodeChild(int, DType, long) — typed child-decode helper that replaces
  the per-encoding decodeChildAs(...) private utilities.
  (d07faf0,
  a1512da)
• Typed accessors on concrete array types — LongArray.segment(), VarBinArray.offsetsSegment(),
  MaskedArray.inner(), and friends now live on the concrete types where they fit cleanly,
  rather than on the Array interface.
  (84a34f4)
• *SecurityTest test-naming convention — adversarial / robustness tests are now grouped
  under the *SecurityTest suffix, mirroring the existing *IntegrationTest convention.
  Run with ./mvnw test -Dtest='*SecurityTest'.
  (76ba67c)
• FlatSegmentDecoder — extracted from EncodingRegistry; the registry is now pure
  dispatch.
  (4a08356)

Changed

• Array interface slimmed down. buffer(int), child(int), and segment() are no
  longer part of the Array interface; consumers should use the typed accessors on the
  concrete subtype (e.g. LongArray.segment()) or ArraySegments.of(arr) for a generic
  fallback. buffer(int) is now package-private on the concrete array types.
  (bdb4e7d,
  1283168,
  ba5957c,
  df6ab3f,
  bb7b656)
• VarBinArray no longer keeps a redundant offsetsArr field; consumers read offsets
  via offsetsSegment().
  (96687f8)
• ArrayStats is no longer eagerly stored on decoded array types; statistics are now
  read on demand from the FlatBuffer node, matching the Rust reference implementation.
  (9237e28)

Fixed

• MaskedArray.segment() delegates correctly to its inner array (regression introduced
  during the typed-accessor migration).
  (8a16119)
• Constant-encoded array indexing broadcasts the index correctly when scanning multiple
  rows from a single stored value.
  (ed658b7)
• Performance benchmark (RustWritesJavaReadsBigFileBenchmark) migrated off the removed
  Array.buffer(int) accessor, unblocking ./mvnw verify and ./bench.
  (977a529)

Removed

• Array.buffer(int), Array.child(int), and Array.segment() from the public Array
  interface (see Changed). Callers should migrate to the concrete-type accessors or
  ArraySegments.of(arr).
  (bdb4e7d,
  1283168)
• Encoding.decodeSegment(...) is removed after the migration to DecodeContext.decodeChild.
  (977a529)
• ArrayStats field on decoded array types (statistics are now lazy).
  (9237e28)

Documentation

• Added CONTRIBUTING.md covering trunk-based workflow, commit conventions, and the
  three-touch-point rule for adding encodings.
  (d9825cf)
• Added an internal-architecture diagram set covering the file format, layout tree, and
  scan path.
  (75f4cea)
• Added a "Vortex vs Parquet" comparison section to the README.
  (886bb80)
• Expanded the ## Security section in TODO.md with the open hardening roadmap (resource
  caps, per-encoding adversarial tests, Jazzer fuzz harness, OSS-Fuzz submission).
  (1bb1465)

Build & Tooling

• Dependabot enabled for Maven and GitHub Actions.
  (dd118a7)
• Numerous dependency bumps: JUnit Jupiter (5.11.4 → 6.1.0, tests now require JUnit 6),
  Mockito, FastCSV (3 → 4), H2 (2.3 → 2.4), Checkstyle, Zstd-JNI,
  maven-compiler/surefire/failsafe/javadoc/source/shade/gpg/antrun/exec/build-helper plugins,
  actions/checkout (4 → 6), actions/setup-java (4 → 5), actions/cache (4 → 5),
  Sonatype central-publishing plugin.
  (81be668,
  2c8319f,
  f6dbcf1,
  dfabd8c,
  7b4a718,
  fb9b404,
  7b67000,
  fd1ea7e)
• pom.xml files now group dependencies under production / testing comment sections
  with a consistent project-internal-first ordering.
  (bcbbbfd)
• Checkstyle scope tightened to exclude generated fbs/proto packages.
  (f5ab433)