fix: render a null struct row as an empty CSV cell, not {"field":null,...}

A nested Struct-dtype column whose entire row is null rendered as a JSON
object with every field null (e.g. {"id":null,"label":null}) instead of an
empty CSV cell, diverging from the parquet oracle and from every other
nullable column type in CsvExporter (discovered on the Raincloud oasst1
`labels` column while validating #268).

Root cause: struct-row nullability is represented in this codebase by
masking EACH field invalid (StructEncodingDecoder / StructLayoutDecoder wrap
every field array in the shared struct validity), never by wrapping the whole
StructArray in one MaskedArray. So a null struct row decodes as a StructArray
whose fields are each MaskedArray-invalid; the StructArray itself is never
masked. CsvExporter.cellValue routed a StructArray straight to jsonObject,
which called jsonValue per field — each field's own MaskedArray arm rendered
"null" — producing {"field":null,...} instead of an empty cell.

Fix at the exporter, not the decoder. The per-field masking is deliberate and
load-bearing: ScanIterator.expandStruct relies on the root StructArray being
un-wrapped (its `instanceof StructArray` checks) and on each field carrying
the struct-row validity so an expanded root column is null when the struct row
is null. Wrapping the whole StructArray once would break that core scan path
for every root struct and remove per-column validity propagation — a wide,
risky change. The actual invariant is intact; the bug is purely that the
exporter never recognized "every field invalid" as "the row is null". A new
isNullStructRow(StructArray, long) helper detects that, and cellValue's and
jsonValue's StructArray arms now render an empty cell / JSON null respectively.

Added CsvExporterTest#rendersNullStructRowAsEmptyCell: a nested struct column
with an all-fields-null row (the exact issue shape) asserts the cell is empty,
not {"id":null,"label":null}. Verified it fails on the pre-fix exporter with
the reported bad output.

Review round (#275): added
CsvExporterTest#rendersNullStructElementInListAsJsonNull to cover the nested
jsonValue StructArray arm — a null struct element inside a List[Struct] must
render as JSON null, not {"ref":null,...} — which was previously untested.
Also clarified isNullStructRow's javadoc: it detects "every field invalid at
this row", which is exact for struct-level-validity sources and an accepted
approximation (indistinguishable from "present with all-null fields") for
per-field-validity sources, matching the parquet oracle for the oasst1 case
and unambiguous in practice since the writer cannot emit struct-level validity.

Fixes #270

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: dfa1

csv/src/main/java/io/github/dfa1/vortex/csv/CsvExporter.java

@@ -148,12 +148,13 @@ private static void writeChunk(Chunk chunk, List<String> colNames, int colCount,
     /// `Double.toString`, `Boolean.toString`, the raw utf8 string); a null row (a [MaskedArray]
     /// invalid row or a [NullArray]) renders as an empty CSV field.
     ///
-    /// A nested struct column ([StructArray], possibly wrapped in a [MaskedArray] when nullable)
-    /// renders as a single JSON object cell `{"field":value,...}` with fields in the struct dtype's
-    /// declared order. Field values reuse the same leaf rendering rules — numbers and booleans
-    /// unquoted, strings JSON-escaped and double-quoted, nested structs recursed — and a null field
-    /// (or null nested row) becomes a JSON `null`. Only the JSON layer is escaped here; the CSV
-    /// writer independently quotes any cell containing the delimiter or a quote character.
+    /// A nested struct column ([StructArray]) renders as a single JSON object cell
+    /// `{"field":value,...}` with fields in the struct dtype's declared order, except a null struct
+    /// row — decoded as every field masked invalid — which exports as an empty field like any other
+    /// null. Field values reuse the same leaf rendering rules — numbers and booleans unquoted,
+    /// strings JSON-escaped and double-quoted, nested structs recursed — and a null field (or null
+    /// nested struct row) becomes a JSON `null`. Only the JSON layer is escaped here; the CSV writer
+    /// independently quotes any cell containing the delimiter or a quote character.
     ///
     /// A fixed-size list column ([FixedSizeListArray]) or variable-length list column ([ListArray]),
     /// either possibly wrapped in a [MaskedArray] when nullable, renders as a JSON array cell
@@ -192,8 +193,10 @@ private static String cellValue(Array arr, long rowIdx) {
             // All-null columns (DType.Null) hold only a row count: every cell is an empty
             // field, same rule as a MaskedArray null row.
             case NullArray ignored -> "";
-            // Nested struct column: render the whole row as a JSON object cell.
-            case StructArray sa -> jsonObject(sa, rowIdx);
+            // Nested struct column: render the whole row as a JSON object cell, unless the row is
+            // itself null (a null struct row is decoded as every field masked invalid, never as the
+            // StructArray being wrapped once), which exports as an empty field like any other null.
+            case StructArray sa -> isNullStructRow(sa, rowIdx) ? "" : jsonObject(sa, rowIdx);
             default -> throw new VortexException(
                     "unsupported array type for CSV export: " + arr.getClass().getSimpleName());
         };
@@ -218,14 +221,40 @@ private static String jsonObject(StructArray struct, long rowIdx) {
         return sb.toString();
     }
 
+    /// Whether the struct row `rowIdx` should be treated as a null struct row. This detects "every
+    /// field is a [MaskedArray] invalid at this row" and treats that as a null struct; a struct with
+    /// no fields, or any field that is not maskable/valid, is never a null row.
+    ///
+    /// The check is **exact** for struct-level-validity sources: `StructEncodingDecoder` represents a
+    /// null struct row by wrapping every field (including non-nullable ones) in a [MaskedArray] that
+    /// shares one struct-level validity bitmap, so all-fields-invalid means exactly "the row is null".
+    /// For sources that only carry per-field validity it is an accepted **approximation** — a row
+    /// where every field independently happens to be null is indistinguishable, on the wire, from a
+    /// genuinely null row, and we pick the empty-cell interpretation. The distinction is moot in
+    /// practice: `StructEncodingEncoder` cannot currently emit struct-level validity, so every null
+    /// struct row this codebase produces already has the all-fields-invalid shape. This choice matches
+    /// the parquet oracle's empty-cell rendering for the known conformance case (Raincloud `oasst1`).
+    private static boolean isNullStructRow(StructArray struct, long rowIdx) {
+        int fieldCount = struct.fieldCount();
+        if (fieldCount == 0) {
+            return false;
+        }
+        for (int i = 0; i < fieldCount; i++) {
+            if (!(struct.field(i) instanceof MaskedArray ma) || ma.isValid(rowIdx)) {
+                return false;
+            }
+        }
+        return true;
+    }
+
     /// Renders one struct field value as JSON text: a nested object for structs, a quoted escaped
     /// string for utf8, JSON `null` for a null row/field, and the bare leaf rendering (unquoted
     /// number/boolean) for everything else. Unlike [#cellValue(Array, long)], a null here is the
     /// JSON token `null` rather than an empty field, because inside a JSON object the empty-field
     /// convention of bare CSV would be ambiguous.
     private static String jsonValue(Array arr, long rowIdx) {
         return switch (arr) {
-            case StructArray sa -> jsonObject(sa, rowIdx);
+            case StructArray sa -> isNullStructRow(sa, rowIdx) ? "null" : jsonObject(sa, rowIdx);
             case MaskedArray ma -> ma.isValid(rowIdx) ? jsonValue(ma.inner(), rowIdx) : "null";
             case FixedSizeListArray fla -> jsonArray(fla.elements(), rowIdx * fla.fixedSize(), (rowIdx + 1L) * fla.fixedSize());
             case ListArray la -> {

csv/src/test/java/io/github/dfa1/vortex/csv/CsvExporterTest.java

@@ -8,6 +8,7 @@
 import io.github.dfa1.vortex.writer.encode.BoolEncodingEncoder;
 import io.github.dfa1.vortex.writer.encode.FixedSizeListData;
 import io.github.dfa1.vortex.writer.encode.ListData;
+import io.github.dfa1.vortex.writer.encode.ListEncodingEncoder;
 import io.github.dfa1.vortex.writer.encode.MaskedEncodingEncoder;
 import io.github.dfa1.vortex.writer.encode.NullEncodingEncoder;
 import io.github.dfa1.vortex.writer.encode.NullableData;
@@ -348,6 +349,49 @@ void rendersNullStructFieldAsJsonNull(@TempDir Path tmp) throws Exception {
                 "south,\"{\"\"id\"\":2,\"\"label\"\":42}\"");
     }
 
+    @Test
+    void rendersNullStructRowAsEmptyCell(@TempDir Path tmp) throws Exception {
+        // Given a nested struct column whose whole row is null (issue #270): the struct decoder
+        // represents a null struct row by masking every field invalid, never by wrapping the whole
+        // StructArray once. The first row has both fields null (a null struct row); the second has
+        // both valid. Regression: cellValue used to route straight to jsonObject, rendering the null
+        // row as {"id":null,"label":null} instead of an empty cell like every other nullable column.
+        // Two struct fields keep `data` from being unwrapped to its bare field on decode; the
+        // `region` sibling keeps it off the single-column expand path.
+        Path vortex = tmp.resolve("nullrow.vortex");
+        DType.Struct inner = new DType.Struct(
+                List.of(ColumnName.of("id"), ColumnName.of("label")),
+                List.of(new DType.Primitive(PType.I64, true), new DType.Primitive(PType.I64, true)),
+                false);
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("region"), ColumnName.of("data")),
+                List.of(DType.UTF8, inner),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.defaults(),
+                     List.of(new StructEncodingEncoder(), new PrimitiveEncodingEncoder(),
+                             new VarBinEncodingEncoder(), new MaskedEncodingEncoder(),
+                             new BoolEncodingEncoder()))) {
+            writer.writeChunk(Map.of(
+                    ColumnName.of("region"), new String[]{"north", "south"},
+                    ColumnName.of("data"), new StructData(List.of(
+                            new NullableData(new long[]{0L, 1L}, new boolean[]{false, true}),
+                            new NullableData(new long[]{0L, 42L}, new boolean[]{false, true})))));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+
+        // Then the all-fields-null row is an empty cell (not {"id":null,"label":null}); the valid
+        // row is the JSON object.
+        List<String> result = Files.readAllLines(csv);
+        assertThat(result).containsExactly(
+                "region,data",
+                "north,",
+                "south,\"{\"\"id\"\":1,\"\"label\"\":42}\"");
+    }
+
     @Test
     void escapesQuotesAndCommasInsideStructStringField(@TempDir Path tmp) throws Exception {
         // Given a struct string field value containing a JSON-significant quote and a comma: the
@@ -521,4 +565,51 @@ void rendersListOfStructsAsJsonArrayOfObjects(@TempDir Path tmp) throws Exceptio
                 "id,members",
                 "1,\"[{\"\"ref\"\":10,\"\"role\"\":\"\"\"\"},{\"\"ref\"\":20,\"\"role\"\":\"\"outer\"\"}]\"");
     }
+
+    @Test
+    void rendersNullStructElementInListAsJsonNull(@TempDir Path tmp) throws Exception {
+        // Given a List[Struct] whose first element is a null struct row (issue #270, nested case):
+        // the struct decoder masks every field invalid, so cellValue never reaches this element —
+        // jsonValue's StructArray arm does, and must emit the JSON token null (not
+        // {"ref":null,"role":null}) inside the array. This is the only test exercising jsonValue's
+        // isNullStructRow branch; the sibling rendersNullStructRowAsEmptyCell only covers the
+        // top-level cellValue path. Each field is an independent NullableData both invalid at row 0
+        // (the all-fields-invalid shape a genuinely-null struct row decodes to), matching the
+        // construction rendersNullStructRowAsEmptyCell uses. The struct needs two fields so it is
+        // not unwrapped on decode; the `id` sibling keeps the list column off the single-column path.
+        Path vortex = tmp.resolve("nullstructinlist.vortex");
+        DType.Struct memberDtype = new DType.Struct(
+                List.of(ColumnName.of("ref"), ColumnName.of("role")),
+                List.of(new DType.Primitive(PType.I64, true), new DType.Primitive(PType.I64, true)),
+                false);
+        DType.List listDtype = new DType.List(memberDtype, false);
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("id"), ColumnName.of("members")),
+                List.of(DType.I64, listDtype),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.defaults(),
+                     List.of(new ListEncodingEncoder(), new StructEncodingEncoder(),
+                             new PrimitiveEncodingEncoder(), new VarBinEncodingEncoder(),
+                             new MaskedEncodingEncoder(), new BoolEncodingEncoder()))) {
+            writer.writeChunk(Map.of(
+                    ColumnName.of("id"), new long[]{1L},
+                    ColumnName.of("members"), new ListData(
+                            new StructData(List.of(
+                                    new NullableData(new long[]{0L, 20L}, new boolean[]{false, true}),
+                                    new NullableData(new long[]{0L, 42L}, new boolean[]{false, true}))),
+                            new long[]{0, 2},
+                            1)));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+        List<String> result = Files.readAllLines(csv);
+
+        // Then the null struct element is the JSON token null; the valid one is a JSON object.
+        assertThat(result).containsExactly(
+                "id,members",
+                "1,\"[null,{\"\"ref\"\":20,\"\"role\"\":42}]\"");
+    }
 }