ORC: Apply top-level scalar initial-default on read (absent-only fill) by cbb330 · Pull Request #252 · linkedin/iceberg

cbb330 · 2026-06-30T00:02:46Z

Summary

Fills a scalar column initial-default on ORC read when the column is absent from the data file — at any nesting level (top-level fields and scalar subfields of structs/maps/lists) — for the generic (GenericOrcReader) and Spark 3.1 row (SparkOrcReader) read paths. Builds on the IC-1 API already on this branch (#250).

Mechanism (the #76 positional pattern, retargeted to `initialDefault`)

ORCSchemaUtil.buildOrcProjection(..., applyDefaults) omits an absent scalar field that declares an initial-default (at any level); the reader's record() step injects the converted default into idToConstant (ORCSchemaUtil.idToConstantWithDefaults), and the existing positional partition-constant path fills it for every row, consuming no column vector. The hot read loop is unchanged. idToConstant is keyed by global field id and consulted at every struct level, so nested fill needs no reader change.
Empty structs are fine: ORC accepts an empty nested read struct (as it already does the empty root struct used by select-only-default), so even a struct whose every subfield is absent + defaulted fills correctly — no special-casing.

Opt-in gate (off by default)

New ORC.read()...applyColumnDefaults(boolean) flag, off by default. OrcIterable omits defaulted columns only when a read opts in and the file carries embedded ids.
Only the Spark 3.1 row path (RowDataReader, inherited by EqualityDeleteRowReader and RowDataRewriter/compaction) and the generic path (GenericReader) opt in. Every other ORC reader (vectorized everywhere, Flink, Spark 2.4/3.2/3.3, MR) leaves the flag off, so columns are synthesized as null and read NULL — they stay positionally aligned and cannot misalign.
id-less / name-mapped files never apply defaults (read NULL), so an initial-default is never name-matched onto a legacy/migrated file.

No regression

Present-column binding is unchanged (ORC name-evolution + positional assembly). With the gate off by default and the fill guarded by initialDefault() != null, existing reads are byte-for-byte unaffected.

Deferred / out of scope

Where a default isn't applied, the column reads NULL (no wrong data silently returned), except the filter case noted below.

Vectorized ORC reads — deferred. ORC vectorization is off by default; when enabled, a defaulted column reads NULL via the vectorized path (row vs vectorized can disagree until the follow-up wires it).
Flink and Spark 2.4 / 3.2 / 3.3 — not wired (read NULL). Follow-up can opt them in via the same gate.
Filter / predicate pushdown on a defaulted column — not handled. SELECT of a defaulted column is correct, but a pushed predicate on it (e.g. WHERE country = 'US') can error or mis-prune, because the column is omitted from the ORC SearchArgument schema. Avoid filtering on defaulted columns until then.
Whole-container-valued defaults (a default value for a struct/list/map itself) — not expressible under the modern Literal-only initialDefault API, on any branch.
Parquet — this branch has no Parquet default-fill (initial-default is consumed only in api/core + orc), so this is ORC-only here.
Non-Java engines (Trino/Presto, Hive, etc.) — apply defaults independently of this code; unaffected by this PR.
Write side — writeDefault (value written when a new write omits the column) is not handled; initial-default is never persisted to metadata.json (by design).

Scope (in this PR)

Generic + Spark 3.1 row, scalar defaults at any nesting level, gated on embedded field ids.

Test plan

orc TestBuildOrcProjection — omit when applyDefaults=true (top-level + nested); synthesize/read-NULL when false; required-with-default omitted (not rejected); nested struct emptied by omit.
orc TestOrcDefaultValues — generic round-trip fills the default; select-only-default (empty projection) reads the default; all scalar types (boolean/int/long/float/double/string/decimal); required-field-with-default; nested struct / map-value / list-element scalar defaults (ported from upstream BaseFormatModelTests); empty nested struct (every subfield defaulted) still fills.
spark-3.1 TestSparkOrcReaderDefaults — row reader fills the default when opted in; reads NULL when not opted in (proves the gate).
Regression: full iceberg-orc suite + Spark 3.1 TestSparkOrcReader (complex nested random data) / record round-trip pass.
spotlessApply clean. (CI to run full suites + revapi.)

Fill a field's initial-default as a per-file constant when its column is absent from an ORC data file, for the generic, Spark (2.4/3.1/3.2/3.3, row + vectorized) and Flink (1.14/1.15/1.16) readers. buildOrcProjection omits an absent top-level scalar field that declares an initial-default; each reader's record() step then injects the converted default into idToConstant so the existing partition-constant path fills it for every row (consuming no column vector). The omit is gated on applyDefaults, which OrcIterable sets to hasIds(fileSchema): id-less (name-mapped/migrated) files synthesize a null column instead and read NULL, so a default is never name-matched onto a file lacking embedded field ids. Scope is top-level scalar defaults; nested defaults are synthesized as null (read NULL) and remain a follow-up. Present-column binding is unchanged (ORC name-evolution + positional assembly), so existing reads do not regress. Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330

Inline provenance annotations as requested, so the split is visible per-hunk.

Legend: 🟣 PRIOR ART (LI fork #76, same positional reader generation) · 🔵 BACKPORTED (decision/semantics carried from my apache/main ORC PR, re-implemented for the positional reader) · 🟢 NET NEW (did not exist on main or in #76).

Mental model: #76 = the how (omit + idToConstant inject on a positional reader); main = the what/why (top-level scalar scope, hasIds gate, NULL fallbacks); the shared idToConstantWithDefaults helper + all-engine wiring = the net-new glue.

cbb330

Stamping the remaining files so no file is left to interpretation. These are the per-version copies of the spots already annotated on the v3.3 / v1.16 representatives, plus the tests. Same legend: 🟣 PRIOR ART (#76) · 🔵 BACKPORTED (from apache/main) · 🟢 NET NEW.

Add an opt-in applyColumnDefaults flag on the ORC read builder (off by default). Only the row SparkOrcReader (Spark 3.1) and generic GenericOrcReader read paths enable it; OrcIterable then omits an absent top-level scalar default only when opted in (and the file has ids), so the reader can inject it via idToConstant. Every other ORC reader leaves the flag off, keeping the column synthesized as a null column (reads NULL) and therefore positionally aligned -- so it cannot misalign. Defers vectorized ORC and Flink defaults to a follow-up, and drops the Spark 2.4/3.2/3.3 wiring (those versions read NULL, unchanged). Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330

Re-stamped provenance for the now-scoped file set (generic + Spark 3.1 row + the opt-in gate). Supersedes my earlier two reviews, which were partly anchored to files that have since been reverted.

Legend: 🟣 PRIOR ART (LI fork #76, positional reader) · 🔵 BACKPORTED (decision/semantics from my apache/main ORC PR) · 🟢 NET NEW (did not exist on main or #76 — includes the opt-in gate introduced in this revision).

…th-default Add typed-default cases (boolean/int/long/float/double/string/decimal) and a required-field-with-default case (end-to-end + projection-level), confirming the fill reuses the partition-constant channel correctly across scalar types and that an absent required field with a default is omitted+filled rather than rejected by the required-missing check. Co-authored-by: Cursor <cursoragent@cursor.com>

Drop the top-level-only gate in buildOrcProjection so an absent scalar field declaring an initial-default is omitted (and filled via idToConstant) at any nesting level, not just the top level. The reader path already keys idToConstant by global field id and consults it at every struct level, so no reader change is needed. Add a safeguard so a non-root struct is never emitted empty (which ORC can reject): if every subfield of a nested struct is absent + defaulted, keep one synthesized null column (it reads NULL). The root struct may still be empty (select-only-default). Nested fill scenarios (struct/map/list) are ported from upstream BaseFormatModelTests. Co-authored-by: Cursor <cursoragent@cursor.com>

…ed structs) ORC accepts an empty nested read struct (as it already does for the empty root struct in select-only-default), so the "keep one synthesized null column" safeguard was unnecessary and introduced a surprising case where one subfield read NULL instead of its default. Remove it: an absent scalar default is now omitted and filled at any nesting level with no exceptions, including when every subfield of a nested struct is absent + defaulted (covered by a new end-to-end test). Also drops the now-unused topLevel parameter from buildOrcProjection. Co-authored-by: Cursor <cursoragent@cursor.com>

Add a row with a null nested struct to testNestedStructScalarDefault: a null parent struct stays null on read (the absent-only default is not fabricated), locking the positional reader's null-struct branch. Upstream's nested-default tests only cover this incidentally via random data, and via the id-binding reader rather than this branch. Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330

Provenance re-stamp for the current state (now includes nested-scalar defaults + the opt-in gate). Supersedes earlier reviews.

Legend: 🟣 PRIOR ART (LI fork #76, positional reader) · 🔵 BACKPORTED (decision/semantics from my apache/main ORC PR) · 🟢 NET NEW (did not exist on main or #76 — chiefly the opt-in gate and the single shared helper).

cbb330 · 2026-06-30T23:26:13Z

+     * so the reader can inject it; when disabled, the field is synthesized as a null column and
+     * reads NULL. The omit is additionally gated on the file carrying embedded field ids.
+     */
+    public ReadBuilder applyColumnDefaults(boolean newApplyColumnDefaults) {


🟢 NET NEW — the opt-in gate. Default-application is OFF unless a read calls applyColumnDefaults(true). Neither main nor #76 had this; it keeps the feature contained so every un-wired ORC reader (vectorized, Flink, Spark 2.4/3.2/3.3, MR) reads NULL and stays positionally aligned.

cbb330 · 2026-06-30T23:26:13Z

+      // read
+      // paths). Other readers leave it false, so an absent top-level scalar default is synthesized
+      // as a null column (reads NULL) rather than omitted, which keeps those readers aligned.
+      readOrcSchema = ORCSchemaUtil.buildOrcProjection(schema, fileSchema, applyColumnDefaults);


🟢 NET NEW + 🔵 BACKPORTED — applyDefaults = applyColumnDefaults (opt-in, 🟢) on the id path; id-less/name-mapped stays false. The hasIds gating is 🔵 from main; the opt-in flag is the net-new piece.

cbb330 · 2026-06-30T23:26:13Z

+   * @param convertConstant converts an internal default value to the engine's in-memory form
+   * @return {@code idToConstant} unchanged when no defaults apply, otherwise a new merged map
+   */
+  public static Map<Integer, ?> idToConstantWithDefaults(


🟢 NET NEW — shared, engine-agnostic default-injection helper. main fills inside the id-binding StructReader; #76 injected inline in a Spark-only visitor. One helper keyed on "expected field absent from the ORC record", reused by the generic + Spark-3.1-row record() steps at every nesting level. The BiFunction is the only per-engine seam.

cbb330 · 2026-06-30T23:26:13Z

+        Integer.MIN_VALUE, schema.asStruct(), true, applyDefaults, icebergToOrc);
+  }
+
+  private static boolean isOmittableDefault(


🔵 BACKPORTED (from main) — omit an absent scalar field that declares a default from the read projection, at ANY nesting level (matches main's scope). The reader then fills it via idToConstant. Empty nested read-structs are fine (verified against ORC source + an end-to-end test), so no special-casing.

cbb330 · 2026-06-30T23:26:13Z

+      return GenericOrcReaders.struct(
+          fields,
+          expected,
+          ORCSchemaUtil.idToConstantWithDefaults(


🟢 NET NEW (breadth) — generic reader wiring via the shared helper + IdentityPartitionConverters::convertConstant. #76 was Spark-only. Active only when the generic read opts in (see GenericReader).

cbb330 · 2026-06-30T23:26:13Z

+  }
+
+  @Test
+  public void testNestedStructScalarDefault() throws IOException {


🔵 BACKPORTED — end-to-end fill tests; the nested struct/map-value/list-element scenarios are ported from upstream BaseFormatModelTests (JUnit4). Also covers all scalar types, required-with-default, empty nested struct, and a null nested struct (default not fabricated for a null parent).

cbb330 · 2026-06-30T23:26:13Z

+      return SparkOrcValueReaders.struct(
+          fields,
+          expected,
+          ORCSchemaUtil.idToConstantWithDefaults(


🟣 PRIOR ART (#76) — the positional inject: detect the absent field, put its default into idToConstant; the unchanged StructReader fills it consuming no vector. Retargeted from #76's getDefaultValue()/BaseDataReader.convertConstant to initialDefault()/BaseDataReader.convertConstant.

cbb330 · 2026-06-30T23:26:13Z

  }

-  protected static Object convertConstant(Type type, Object value) {
+  public static Object convertConstant(Type type, Object value) {


🟣 PRIOR ART (#76) — convertConstant made public for cross-package access from the spark.data visitor, exactly as #76 did for this class.

cbb330 · 2026-06-30T23:26:13Z

            .filter(task.residual())
-            .caseSensitive(caseSensitive);
+            .caseSensitive(caseSensitive)
+            .applyColumnDefaults(true);


🟢 NET NEW — Spark 3.1 row read opts into the gate. The vectorized path deliberately does NOT opt in (reads NULL) — vectorized is the follow-up. Inherited by EqualityDeleteRowReader and RowDataRewriter (compaction).

cbb330 · 2026-06-30T23:26:13Z

+  }
+
+  @Test
+  public void testRowReadReadsNullWhenNotOptedIn() throws IOException {


🔵 BACKPORTED + gate proof — mirrors main's TestSparkOrcReaderDefaults (row fill), plus a net-new assertion that without opt-in the same read returns NULL, proving the gate is off by default.

github-actions Bot added SPARK ORC FLINK labels Jun 30, 2026

cbb330 commented Jun 30, 2026

View reviewed changes

github-actions Bot added DATA and removed FLINK labels Jun 30, 2026

cbb330 commented Jun 30, 2026

View reviewed changes

cbb330 and others added 4 commits June 30, 2026 15:33

cbb330 commented Jun 30, 2026

View reviewed changes

Uh oh!

Conversation

cbb330 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Mechanism (the #76 positional pattern, retargeted to initialDefault)

Opt-in gate (off by default)

No regression

Deferred / out of scope

Scope (in this PR)

Test plan

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

cbb330 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

cbb330 commented Jun 30, 2026 •

edited

Loading

Mechanism (the #76 positional pattern, retargeted to `initialDefault`)