Skip to content

Commit dade351

Browse files
dfa1claude
andcommitted
docs: ADR 0021 — the frequency remap is primitive-path-only
Checked writeGlobalDictUtf8Column before implementation started: it assigns codes in first-seen order with no frequency sort, unlike writeGlobalDictColumn (primitives), which sorts by occurrence count so the dominant value gets code 0. Split the Decision's step 3 accordingly — Utf8 needs zero remap work, primitives need the O(rows) code-remap pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent e3874d1 commit dade351

1 file changed

Lines changed: 16 additions & 9 deletions

File tree

adr/0021-cardinality-bounded-global-dict-buffering.md

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -79,15 +79,22 @@ Replace raw-value buffering with **cardinality-bounded** buffering:
7979
`close()` via the existing `codePTypeForSize` (`U8` when ≤ 256 distinct
8080
values), exactly as today — buffering width and wire width are
8181
independent, so no upfront width decision is needed.
82-
3. At `close()`, rank the map's values by occurrence count descending —
83-
preserving the existing invariant that the dominant value gets code 0,
84-
which is what lets `SparseEncodingEncoder` (fill = 0) compress the codes
85-
child (deliberately matching Rust's `FloatDictScheme`) — then remap the
86-
buffered code arrays through a first-seen → frequency-ranked code table
87-
(one O(rows) table-lookup pass) and write them as the column's chunked
88-
codes segment. No re-scan of raw values is needed, but this remap pass
89-
is mandatory: skipping it would silently lose the dict+sparse win on
90-
dominant-value columns.
82+
3. This step is **path-specific**, since the two existing close()-time
83+
builders already disagree on code order:
84+
- **Primitive columns** (`writeGlobalDictColumn`): today ranks distinct
85+
values by occurrence count descending so the dominant value gets code 0
86+
— the invariant that lets `SparseEncodingEncoder` (fill = 0) compress
87+
the codes child, deliberately matching Rust's `FloatDictScheme`. The
88+
incremental map is first-seen-ordered, so this path needs a `close()`-
89+
time remap: rank by the incrementally-tracked occurrence count, build a
90+
first-seen → frequency-rank code table, and pass every buffered code
91+
array through it (one O(rows) table-lookup pass, no re-scan of raw
92+
values). Skipping this remap would silently lose the dict+sparse win on
93+
dominant-value columns.
94+
- **Utf8 columns** (`writeGlobalDictUtf8Column`): today assigns codes in
95+
first-seen order with no frequency sort at all. The incremental map's
96+
order already matches this exactly — no remap pass needed, no
97+
behavior change to port.
9198

9299
Memory for a surviving candidate is then bounded by
93100
`cardinality × avg_value_size + row_count × code_width` — proportional to

0 commit comments

Comments
 (0)