You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit dc16a73
Browse filesBrowse the repository at this point in the historyBrowse files
Docs: T1.11c as built, E40 to E42, register P1B-3 proposed
The T1.11c row's as-built notes, each claim with the mechanism that
holds it, and the tracker; contract version 7 (E40), materialize's thin
scope and its departures (E41), the harness and the test host's key-set
binding (E42), all for the owner to confirm; P1B-3, the thin route's
refusals of inputs the reference accepts, proposed and waiting for the
owner, with the coverage warning's choice (E41(c)). dependencies.md
records the ported streams and labels and duckdb-rs's vtab-arrow.
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
Copy file name to clipboardExpand all lines: docs/design/rust-core/phase-1/decisions.md
+5-1Lines changed: 5 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ Decisions made for Phase 1, newest last.
4
4
Rows marked "orchestrator, owner to confirm" were made by the orchestrator; implementation proceeds on them, and the owner confirms or changes them at the check-in that lists them (design, "Owner check-ins"; decision H12); only tasks depending on a listed row wait, and defect fixes never wait.
5
5
Rows marked "approved by owner 2026-10-08" record the owner's written answers to re-review rounds 1 and 2, the T1.7b check-in and checkpoint 1, and "approved by owner 2026-10-09" the owner's answers of that day (E27 to E31); no other row is owner-approved unless its status says so.
6
6
Only these wait for the owner: the 1b go or no-go, any new refusal of an input Python accepts (P1B-1), relaxing a comparison rule, adding an exclusion, downgrading a field to unavailable or adding one as unavailable, a new acceptance capability, an addition to `emulation-allowlist.txt`, the Q2 narrowing fallback, a same-way counter reaching 2 (a check failing twice the same way), and two fix rounds on the same product divergence (migration-plan.md section 9). A review's fix-round counter is reported to the owner but not waited on (E29).
7
-
Rows marked "revised R1" or "revised R2" were changed after that review round; rows C1 to C9 come from round 1, D1 from round 2, E1 to E15 from implementation, E16 and E17 from the T1.7b check-in, E18 and E19 from the owner at checkpoint 1, E20 to E25 from the checkpoint 1 review (T1.8b), E26 from T1.9's code review, E27 from the owner on 2026-10-09, E28 to E31 from the owner at checkpoint 2 (2026-10-09), E32 from the checkpoint 2 corrections (T1.10d), E33 to E35 from T1.11a, E36 to E39 from T1.11b, H1 to H26 and F1 to F7 from re-review round 1, H27 from the batch 2 code review. Re-review round 2 and its closure checks added no row: their corrections revised rows in place ("re-review R2") and the design text, and superseded rows are marked, not deleted.
7
+
Rows marked "revised R1" or "revised R2" were changed after that review round; rows C1 to C9 come from round 1, D1 from round 2, E1 to E15 from implementation, E16 and E17 from the T1.7b check-in, E18 and E19 from the owner at checkpoint 1, E20 to E25 from the checkpoint 1 review (T1.8b), E26 from T1.9's code review, E27 from the owner on 2026-10-09, E28 to E31 from the owner at checkpoint 2 (2026-10-09), E32 from the checkpoint 2 corrections (T1.10d), E33 to E35 from T1.11a, E36 to E39 from T1.11b, E40 to E42 from T1.11c, H1 to H26 and F1 to F7 from re-review round 1, H27 from the batch 2 code review. Re-review round 2 and its closure checks added no row: their corrections revised rows in place ("re-review R2") and the design text, and superseded rows are marked, not deleted.
8
8
The design is updated in the same change as a row that changes it.
9
9
Standing decisions are in [migration-plan.md](../migration-plan.md) section 2 and [phase-0/decisions.md](../phase-0/decisions.md).
10
10
@@ -77,6 +77,9 @@ Standing decisions are in [migration-plan.md](../migration-plan.md) section 2 an
77
77
| E37 | The test host's ICU is loaded explicitly, in place of `duckdb`'s `icu` feature, which `dependencies.md` planned: at `=1.10506.0` the feature needs `bundled-cmake`, which needs DuckDB's sources beside the crate (`duckdb-sources/CMakeLists.txt`), and the published crate carries neither them nor ICU's, so the build script panics. `sdm-testkit`'s `duck::open` turns autoloading and autoinstalling off, then runs `LOAD icu`, and, the first time on a machine, `INSTALL icu` from DuckDB's repository, the extension DuckDB signs for this exact version (signature checking stays on); it checks the loaded ICU's version is v1.5.6 and the linked DuckDB's too. The Python wheel of DuckDB links ICU statically, so both hosts then read `TIMESTAMPTZ` alike. The cost: `rust-test` and a developer's first run download one signed file (about 15 MB). This is a signature-verified network bootstrap, not a hermetic setup: an offline first run fails. Since T1.11b's first review round, the installed file's SHA-256 is checked against the digest recorded for its platform (`linux_amd64`, `linux_arm64`, `osx_amd64`, `osx_arm64`, as `extensions.duckdb.org` serves it) before it is loaded, `rust-test` caches the extension directory for v1.5.6, and the test asserts autoinstall is off as well as autoload | The design's mechanism cannot be built at the pinned version; a git-sourced `duckdb-rs` with its submodules would change the source rules `cargo-deny` holds and the DuckDB build, and a prebuilt `libduckdb` would change the linking; an explicit, version-checked load keeps the intent (no extension loads implicitly) | implementer (T1.11b), owner to confirm |
78
78
| E38 | `describe_source` is thin at T1.11b, by the reference's rules where it ports them: (a) each column's data type comes from the Arrow type its preview returned (bool, integers and decimals of scale 0 as `int`, floats and other decimals as `float`, `utf8` as `string`, dates and timestamps as `date`, `time64` as `time`), not from the catalog's spelling of its type, which is the dialect's and so `sdm-data`'s; for those types the two agree, and any other Arrow type is refused; (b) every input whose description takes a rule not ported is refused with `UNSUPPORTED_REQUEST` naming the rule (register entry P1B-2, proposed for the owner), never guessed: text that may hold dates (T1.25, DT-7) or several categories, a key tie or a key whose score depends on the reference's set order, keys that repeat or miss values, a keyless table the declared foreign keys would key, a time column other than the one plain timestamp, a time of day (`TIME`) as a time column candidate (the reference reads none of its values as a time and refuses it, with a warning; rule `time_of_day`, from T1.11b's first review round, where the thin choice had taken it), a preview of 1,000 rows (the reference then reads the whole table), an empty table, and the overrides (tables named, keys, time columns); `sdm-data` reads the table list and the indexes only whole, as the reference does, refusing tables named (T1.24's); (c) the schema bridge builds today's `Schema` with the client's own assembly after the description (relationships inferred by name over the declared links, edit stamps settled, key columns marked, the same warnings), as `Schema.from_source` does after the engine's reading; relationships passed in, and a time column cleared, are refused there, naming the rules `relationships` and `time_column_cleared`; the old materializer reads the source itself afterwards, from an empty `EngineTables`; (d) one data request per reference statement, in its order; `key_counts` counts an enforced unique key with `COUNT(DISTINCT …)` where the reference writes `COUNT(*) - nulls`, the same value, and numbers its keys among those it counts where the reference numbers them among every keyed table, which differ only when a key the source enforces comes before one it does not (H9 gates on results: T1.23 settles it with the comparator's rules) | The design gives the inference to T1.23 and the bridge's full form to T1.24; T1.11b's exit is that `base.regression`'s schema completes, and the owner's rule is that refusing a reference-accepted input waits for the owner, so the thin route says what it cannot do rather than answer differently | implementer (T1.11b), owner to confirm; P1B-2 for the owner |
79
79
| E39 | The DuckDB provider reads each request on its own cursor on both hosts (Python `connection.cursor()`, Rust `Connection::try_clone`), as the reference's reads do: a cursor takes the database's settings and none of the source connection's session. So the `TIMESTAMPTZ` parity check of T1.11b's exit is held under `SET GLOBAL TimeZone='America/New_York'`, where both hosts read New York's wall clock (`contract/provider-vectors.json`, the same rows in each host's test), and a session `SET TimeZone='America/New_York'` on the source's own connection is shown to be seen by no read, on either host, as in the reference. Harness changes with it: the observation's `core` records every fetch with its plan (the step trace) and `describe_source`'s description, which the comparator holds equal between `python-core` and `rust-host`; the case format names `describe_source`'s inputs; the runner's schema projection reads the schema's `relationships`, where it read a `links` attribute the schema never had, so every projection before recorded none | The exit check's session `SET` reaches no read through a cursor, in the reference or either host; read literally it would compare each host's local zone, not New York's | implementer (T1.11b), owner to confirm |
80
+
| E40 | Contract version 7 (T1.11c): a described column carries `nullable` and `unique`, the catalog's word the sampler reads (the reference's `is_nullable`, which decides the reservoir sample's not-null filters, and `is_unique_key`, which decides its missing-index warning), and `materialize` takes its meaning: the entities to predict for one Arrow column of the entity key's type, the anchor time a zone-less `ts`, the seed any non-negative integer; its result a `RequestState` of version 1, the payload a bytes attachment of its JSON and the query the text as the reference writes it back (`to_string`). A `SourceDescription` travels as an attachment, so the generated schema changes only in its `$id`; the change is one of shape and meaning (H25), so the working entry is rewritten under H4: `sdm-contract-6.json` becomes `sdm-contract-7.json` and the lock holds that one entry. A test-only `keyed` fetch (not in the schema) sends a key set the core attaches to the step as its own | The sampler reads what the catalog declares of each column, which T1.11b's description did not carry, and a second read of the catalog would add statements H9 counts | implementer (T1.11c), owner to confirm at the Journey check-in |
81
+
| E41 | `materialize` is thin at T1.11c, by the reference's rules where it ports them, and departs from them here: (a) the bridge keeps the client's materializer around the core's gathering, as E38(c) keeps the client's assembly around the description: its checks and notices before, and the limits, the statistic, the context that answers the question taken out and `Request.materialized` after; the core gives the engine's payload (`materialize_task`'s), from the schema as it stands with the catalog's word on each column from the description `Schema.from_source` kept on the core route (a schema not described on the core has no core route); (b) every input whose request takes a rule not ported is refused with `UNSUPPORTED_REQUEST` naming the rule (register entry P1B-3, proposed for the owner), never guessed; (c) among them the coverage warning ("Failed to collect 1,000 context examples"), which the reference gives once per process through a module global and a core holding no state across runs cannot promise (`context_coverage`), for the owner to choose: refuse it, warn on every run, or hold it in the host; (d) key identity is exact (H22): a sampled row is told apart by its key and example as values, where the reference's `Mapper` multiplies an integer key by the example count in int64, which wraps past about 9.2e18 divided by that count and could take two keys for one; no case reaches it; (e) a seed DuckDB's parser refuses, from 2**31, is refused by `sdm-data`'s plan of the sample (`INVALID_REQUEST` naming the request and the field), after the index and time-bound reads, where the reference's statement fails in DuckDB's parser as `INTERNAL_ERROR`: register RS-7's approved behavior, ahead of T1.22, which moves the range to a capability the provider declares; the sample carries the seed as the narrowest exact scalar (`u64`, else a 38-digit decimal), and one past 38 digits is refused (`seed_size`); (f) `sample_entities`' `key` is honored as columns the sample projects, and the core keeps one row per key, the first sampled, as `drop_duplicates` does; (g) the payload's numbers are written as the shortest text that reads back as the same double (`serde_json`'s), not Python's `repr` (`1e-5` for `1e-05`); on `python-core` the client's transport reads and writes the payload again, so the endpoint receives the reference's text; the 30 MB payload limit is counted on the core's own bytes (PD-2's count is T1.28's); (h) the query is resolved on the core against the schema the description is, with the client's PQL type mapping (a timestamp is PQL's `time`, as `to_schema_definition` writes it), and written back by the interim parser's port of `to_string`; (i) the sampler's warnings carry codes (`MISSING_INDICES`, `CONTEXT_BEFORE_DATA`, `ANCHOR_AFTER_DATA`) the Python host shows as `UserWarning`, the reference's class; their text is the core's | T1.11c's exit is that `base.regression` reaches its payload on both core runners; the design gives sampling, labels and the payload to T1.26 to T1.28, and the owner's rule is that refusing a reference-accepted input waits for the owner, so the thin route says what it cannot do rather than answer differently | implementer (T1.11c), owner to confirm; P1B-3 and (c) for the owner |
82
+
| E42 | The harness at T1.11c: the observation's `core` records `materialize`'s inputs and its request state (`materialized`: version, query and the payload, each number as the text the core wrote), which the comparator compares between the core runners and holds to the outcome; a case names `materialize`'s core inputs (`core.materialize`, its schema the description of the schema the call holds, with the schema's relationships as links, as `validate_query`'s case names its `QuerySchema`); the pin of the two hosts' equality covers the payload; `contract/tests` holds `python-core`'s public projection to the reference observation's with `capture_check.projection` (E35's value-only exclusion, not widened) and the core's statements to the reference statements' results, each reference statement re-run on a key table of the core's key set's rows, which the reference capture's binding digest shows to be its own; `base.regression`'s reference observation is recorded again for its changed case file. The test host binds a key set through `duckdb-rs`'s Arrow table function (`vtab-arrow`: `CREATE TEMPORARY TABLE … AS SELECT * FROM arrow(?, ?)` on the step's own cursor), which adds the `num` crates to the test host only and keeps each batch it is handed in a store for the life of the process; that store is the test host's, never a shipped crate's | H9's full comparator is T1.11e's, so the result comparison is written for `base.regression`'s six statements; the Python provider registers a relation, which the Rust driver has only as a table function | implementer (T1.11c), owner to confirm |
80
83
| H1 | Runtime cells: CI cells are named by runtime and pinned from `phase-0/support-matrix.md`, as local semantic cells, not Databricks runtime certification: `ref-pandas3` (the recorded reference environment) compared with the reference on all fields; `dbr17-pandas2` (the observed DBR 17.3 LTS set) where `python-core` equals `python-old`; `floor-pandas2.2` (Python 3.10, pandas 2.2.0, pyarrow 14.0.1, NumPy 1.26.4, the E16 floor), compared like `dbr17-pandas2`; revised by E16 from A8's S0-only pandas 1.5 floor. A runtime-only difference is a register entry naming the cell, never an exclusion; observation identity records platform and cell. Re-review R2 (owner answer 5): the reference stays recorded on macOS and reference comparisons run locally through `gate.sh`; CI gates platform-independent fields; a Linux reference is a later CI build | Locked check 4 never said which pandas it compares on; the first attempt ended with 165 pandas-2 exemptions | approved by owner 2026-10-08 (round 1 answer 1, round 2 answer 5); the cell versions are the orchestrator's reading of support-matrix.md |
81
84
| H2 | Gate integrity: the checkpoint is the last reviewed landing, with no tag, serialized promotion or separate accept commit (re-review R2): `CHECKPOINT_SHA = git merge-base CANDIDATE_SHA BASE_SHA`, with `BASE_SHA` the phase-branch tip resolved at package start and both SHAs recorded and passed to every gate run, never recomputed from a ref that contains the candidate (R2 closure check); each gate run checks it out into its own worktree, overlays only its observation projection and capture (`run_python.py`, the test kit's case loader and observation writer; the pure-Rust host, providers and scripted endpoint are protected candidate code, and `KNOWN_DIFFERING` is a candidate data file accepted only as a subset of the checkpoint's), and runs its harness, comparator, ratchet, L0 scripts and expectations against the candidate's observations; `contract/protected-paths.txt` is read at the checkpoint, and a protected change passes only when the reviewer's `Reviewed-by:` trailer names it, an outcome change only with a reviewer-approved register entry, a relaxation only with an owner-approved row; before T1.14 lands the candidate's harness runs under review, with no acceptance claim; `contract/availability.yaml` fixes availability per runner and field, and "unavailable" cannot pass a locked check; the owner's local merge is the only hard approval; no signing machinery | Re-review found the gate judged by code and expectation files the candidate can edit (learnings lesson 9), and round 2 found the tag procedure could not bootstrap and let the executor authorize its own protected changes | approved by owner 2026-10-08 (round 1 answer 2, round 2 answer 1); the bootstrap, the authorization rule and the narrowed overlay approved at the T1.7b check-in |
82
85
| H3 | A change is one reviewed work package: its proposed baselines are reviewed with it, one final gate run judges it against the checkpoint, it lands only with evidence for its exact SHA (one rerun after review fixes), and landing is the promotion; `rust-lint` clippy skips `sdm-testkit`, so only `rust-test` builds DuckDB | A CI round trip per commit and per baseline-only commit repeats the first attempt's validation overhead; the cold Rust jobs are dominated by the bundled DuckDB build | approved by owner 2026-10-08 (T1.7b check-in); revised re-review R2 |
@@ -146,6 +149,7 @@ Rows that change [migration-plan.md](../migration-plan.md) are applied there whe
146
149
- Q2's narrowing fallback, only if T1.11a finds the driver capture infeasible; T1.11a then stops for the owner.
147
150
- At T1.11e: H9's rule against the measured counts.
148
151
- P1B-1: any Phase 1b case whose new outcome refuses an input the reference accepts comes to the owner case by case, as its case lands (H19).
152
+
- P1B-2 (T1.11b) and P1B-3 (T1.11c): the thin core routes' named refusals of inputs the reference accepts, proposed as temporary; with P1B-3, E41(c): the reference's once-per-process coverage warning refused on the core route, warned on every run, or held by the host.
149
153
- The held-out no-go threshold (H11), at T1.20.
150
154
- E6: the instants at or past 9999-12-30T22:00Z that T1.25's time module meets, and whether any path cannot avoid a refusal there (then a case and a register entry, P1B-1).
151
155
- E25: whether a small fix round may use one reviewer.
0 commit comments