Skip to content

Commit 7b4d0fc

Browse files
adi-kumoadi-kumo
committed
Phase 0: support matrix, scope manifests and the Databricks job map
Signed-off-by: Adi Wadaskar <awadaskar@nvidia.com>
1 parent 0d21144 commit 7b4d0fc

7 files changed

Lines changed: 2654 additions & 2 deletions

File tree

‎docs/design/rust-core/phase-0/databricks-jobs.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Databricks job matrix to public paths
2+
3+
Task T0.5 of [Phase 0](design.md), part 3.
4+
This maps each cell of the first attempt's 20-cell Databricks job manifest, through today's notebooks in `examples/databricks/`, to the public paths in [scope-python.yaml](scope-python.yaml) it exercises.
5+
The cell list is the historical one from the first attempt's `investigations/P95-ACCEPTANCE-PREPLAN.md` (section 4), which its decision DV55 adopted as the minimum; the old `matrix*.sh` scripts point at notebooks that no longer exist and are not used.
6+
The notebooks are `adventureworks_relational.ipynb` (Unity Catalog, over a SQL warehouse or over the notebook's Spark) and `lakebase_relational.ipynb` (Lakebase through `PostgresSource`).
7+
8+
## The 20 cells
9+
10+
Python versions are as observed in the first attempt's job outputs (`~/.local/share/sdm-dbx/result-review-fix.json` and `result-typed.json`); see [support-matrix.md](support-matrix.md) for the rest of each environment.
11+
12+
| Cell | Route | Runtime | Architecture | Python | Notebook | Data backend |
13+
|---|---|---|---|---|---|---|
14+
| J01 | Unity Catalog via SQL warehouse | DBR 14.3 LTS | x86_64 | 3.10.12 | adventureworks_relational, `warehouse_id` set | `databricks-sql` |
15+
| J02 | Unity Catalog via SQL warehouse | DBR 14.3 LTS | aarch64 | 3.10.12 | same | `databricks-sql` |
16+
| J03 | Unity Catalog via SQL warehouse | DBR 15.4 LTS | x86_64 | 3.11.11 | same | `databricks-sql` |
17+
| J04 | Unity Catalog via SQL warehouse | DBR 15.4 LTS | aarch64 | 3.11.11 | same | `databricks-sql` |
18+
| J05 | Unity Catalog via SQL warehouse | DBR 16.4 LTS | x86_64 | 3.12.3 | same | `databricks-sql` |
19+
| J06 | Unity Catalog via SQL warehouse | DBR 16.4 LTS | aarch64 | 3.12.3 | same | `databricks-sql` |
20+
| J07 | Unity Catalog via SQL warehouse | Serverless env 1 | not proven (x86_64 observed) | 3.10.12 | same | `databricks-sql` |
21+
| J08 | Unity Catalog via SQL warehouse | Serverless env 2 | not proven | 3.11.10 | same | `databricks-sql` |
22+
| J09 | Unity Catalog via SQL warehouse | Serverless env 3 | not proven | 3.12.3 | same | `databricks-sql` |
23+
| J10 | Unity Catalog via Spark | DBR 17.3 LTS | x86_64 | 3.12.3 | adventureworks_relational, `warehouse_id` empty | `databricks-spark` |
24+
| J11 | Unity Catalog via Spark | DBR 17.3 LTS | aarch64 | 3.12.3 | same | `databricks-spark` |
25+
| J12 | Unity Catalog via Spark | Serverless env 4 | not proven | 3.12.3 | same | `databricks-spark` |
26+
| J13 | Unity Catalog via Spark | Serverless env 5 | not proven | 3.12.3 | same | `databricks-spark` |
27+
| J14 | Unity Catalog via Spark | Serverless env 6 | not proven | 3.12.3 | same | `databricks-spark` |
28+
| J15 | Lakebase | Serverless env 1 | not proven | 3.10.12 | lakebase_relational | `lakebase` |
29+
| J16 | Lakebase | Serverless env 2 | not proven | 3.11.10 | same | `lakebase` |
30+
| J17 | Lakebase | Serverless env 3 | not proven | 3.12.3 | same | `lakebase` |
31+
| J18 | Lakebase | Serverless env 4 | not proven | 3.12.3 | same | `lakebase` |
32+
| J19 | Lakebase | Serverless env 5 | not proven | 3.12.3 | same | `lakebase` |
33+
| J20 | Lakebase | Serverless env 6 | not proven | 3.12.3 | same | `lakebase` |
34+
35+
Serverless architecture is "not proven" because a serverless job does not choose it; the preplan requires each cell to record the observed architecture.
36+
37+
## Paths every cell exercises
38+
39+
`B` is the cell's data backend from the table above.
40+
41+
| Notebook step | Path | Slice |
42+
|---|---|---|
43+
| `%pip install --only-binary=:all:` with the runtime's NumPy pinned | Packaging, not a path: a wheel must exist for the cell's Python and architecture (L8, [support-matrix.md](support-matrix.md)) | Not a slice |
44+
| `warehouse(warehouse_id, workspace=...)` (J01 to J09) | `source.connect-databricks/warehouse/databricks-sql` | S2 |
45+
| `lakebase(endpoint, database=..., workspace=...)` (J15 to J20) | `source.connect-databricks/lakebase/lakebase` | S2 |
46+
| `DatabricksSource(connection or spark, catalog=, schema=)`, or `PostgresSource(connection, schema=)` | `source.open/explicit/B` | S2 |
47+
| Unity Catalog: `Schema.from_source` with `primary_keys`, `time_columns` and `relationships` declared | `schema.from-source/tables/databricks-sql` or `.../databricks-spark` | S2 |
48+
| Lakebase: `Schema.from_source` with keys and links read from the catalog | `schema.from-source/catalog-keys/lakebase` | S2 |
49+
| `schema['products'].time_column = None` | `schema.edit/set-time-column/any` | S1 |
50+
| `del table['rowguid']` on each `SourceTable` | `schema.edit/drop-column/any` | S1 |
51+
| `'rowguid' in table`, `print(table)`, `print(schema.relationships)` | `schema.inspect/attributes/any` | S1 |
52+
| `SDMClient.for_databricks_serving(endpoint)` | `client.connect/databricks-serving/any` | S2 |
53+
| `PredictiveQuery(text, schema)` for every question | `query.validate/accept/any` | S0 |
54+
| `answer.to_pandas()`, then `display` and `to_dict('records')` | `result.read/<kind>/any` (per question below) | per question |
55+
56+
A refused question is caught as `StructuredDataError` and reported; that is a case outcome under the question's paths, not a separate path.
57+
58+
## Paths per question
59+
60+
Every Unity Catalog cell asks the six base questions and, when its `metric_view` widget names one, the two view questions; every Lakebase cell asks the six base questions.
61+
That is 8 questions in each of 14 cells and 6 in each of 6, 148 question runs, as the preplan counts.
62+
63+
| Question | PQL kind and options | Paths (data backend `B`) | Slice |
64+
|---|---|---|---|
65+
| `demand_30d` | Regression; `indices`, `inference_config` | `query.materialize/indices-regression/B`, `client.predict/regression/B`, `result.read/regression/any` | S2 |
66+
| `demand_30d_historical` | Regression; adds `anchor_time` | Same as `demand_30d` | S2 |
67+
| `demand_3_timeframes` | `FORECAST 3 TIMEFRAMES`; `anchor_time`, and `inference_config` on Unity Catalog only | `query.materialize/indices-forecast/B`, `client.predict/forecast/B`, `result.read/forecast/any` | **S3** |
68+
| `inactivity_90d` | Binary (`COUNT(...)=0`); 25 `indices` | `query.materialize/indices-binary/B`, `client.predict/binary/B`, `result.read/binary/any` | S2 |
69+
| `recommend_top5` | `RANK TOP 5` over `LIST_DISTINCT`; `candidates`, `anchor_time`, `num_context_samples` | `query.materialize/indices-ranking/B`, `client.predict/ranking/B`, `result.read/ranking/any` | **S3** |
70+
| `product_color` | Multiclass (a static column of the entity) | `query.materialize/indices-multiclass/B`, `client.predict/multiclass/B`, `result.read/multiclass/any` | S2 |
71+
| `view_demand_30d` (J01 to J14) | Regression through a metric view | `schema.from-source/semantic-model/databricks-metric-view`, `query.materialize/indices-regression/databricks-metric-view`, `client.predict/regression/databricks-metric-view`, `result.read/regression/any` | S2 |
72+
| `view_inactivity_90d` (J01 to J14) | Binary through a metric view | `schema.from-source/semantic-model/databricks-metric-view`, `query.materialize/indices-binary/databricks-metric-view`, `client.predict/binary/databricks-metric-view`, `result.read/binary/any` | S2 |
73+
74+
The metric view questions run over the warehouse connection in J01 to J09 and over Spark in J10 to J14; the manifest has one `databricks-metric-view` backend for both, so cases must cover both connections.
75+
76+
## S3 paths the S2 job matrix needs
77+
78+
Two of the six base questions fall in S3, on every one of the 20 cells:
79+
80+
- `demand_3_timeframes` needs `query.materialize/indices-forecast/B`, `client.predict/forecast/B` and `result.read/forecast/any`.
81+
- `recommend_top5` needs `query.materialize/indices-ranking/B`, `client.predict/ranking/B` and `result.read/ranking/any`.
82+
83+
Here `B` is `databricks-sql`, `databricks-spark` and `lakebase`.
84+
These paths carry a note in [scope-python.yaml](scope-python.yaml).
85+
The plan requires the job matrix at the S2 gate and "the Databricks job matrix equal to the baseline" at the Python switch for S0, S1 and S2 paths, so the conflict has to be settled before Phase 3.
86+
87+
## Decisions for the owner
88+
89+
**J1. Where ranking and forecasting on the Databricks and Lakebase backends belong.**
90+
Option A: move the `indices-ranking` and `indices-forecast` modes of `query.materialize`, `client.predict` and `result.read` on the SQL backends into S1 (offline, DuckDB, SQLite, Postgres) and S2 (live), which pulls the decomposition and result merging in `sdm-request` forward into Phase 2 and Phase 3.
91+
Option B: keep them in S3 and, at the S2 switch, run those two questions on the old path; a `Schema` built by the switched `from_source` then feeds an unswitched `materialize` in the same session, which is the cross-call mix the plan's switch-together rule avoids, and it needs its own compatibility check.
92+
Option C: drop the two questions from the S2 job matrix and run them in an S3 job matrix, which narrows the S2 gate to four of six base questions.
93+
Recommendation: Option A, since the job matrix is the main live proof and both questions are what Databricks users run; frames, sessions, link prediction and tabular requests stay in S3.
94+
95+
**J2. What "Python and Java agree on the same Databricks data" covers in S2.**
96+
Java's manifest has only the warehouse route (`databricks-sql`) and no ranking, forecasting, Spark or Lakebase paths, so Java can match Python on `demand_30d`, `demand_30d_historical`, `inactivity_90d`, `product_color` and, if confirmed, the two metric view questions.
97+
Recommendation: state that set as Java's S2 agreement, and keep ranking and forecasting out of Java until the owner adds scope.
98+
99+
**J3. Routes the historical cells never cross.**
100+
The warehouse route runs only on DBR 14.3 to 16.4 and serverless env 1 to 3, and the Spark route only on DBR 17.3 and serverless env 4 to 6, although the warehouse route also works on the newer runtimes.
101+
Recommendation: keep the 20 cells as the minimum (DV55), and have the acceptance author add one warehouse-route cell on DBR 17.3 or a newer serverless environment, since that is a supported user route with no cell.
102+
103+
## Notes for the acceptance author
104+
105+
- The Unity Catalog notebook asks the view questions only when `metric_view` is set; the job runner sets it on every Unity Catalog cell, so a required view question is never a silent skip.
106+
- The two notebooks differ in one option: `demand_3_timeframes` passes `inference_config` on Unity Catalog and not on Lakebase; a runner that reproduces the notebooks keeps that difference or records the change.
107+
- The Unity Catalog notebook leaves `sales_order_headers` without a primary key on purpose; the cases under `schema.from-source/tables/...` should include a table without one.
108+
- J01, J02, J07 and J15 run Python 3.10 and start with pyarrow 8 before the install upgrades it, which is where the pyarrow floor decision in [support-matrix.md](support-matrix.md) bites first.

‎docs/design/rust-core/phase-0/scope-format.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# Scope manifest and case inventory format
2+
3+
Task T0.5 of [Phase 0](design.md).
4+
Each language has one scope manifest: [scope-python.yaml](scope-python.yaml) and [scope-java.yaml](scope-java.yaml).
5+
A manifest lists the public paths and the type x role cells in scope for that language, and the contract case IDs that cover each one.
6+
It is the input to the L2 coverage matrix (plan section 6): for the current slice, every path and every cell needs at least one case, and an empty one fails.
7+
The JSON Schema is [scope.schema.json](scope.schema.json).
8+
9+
## Shape
10+
11+
A manifest is one YAML document with seven keys, all required.
12+
13+
| Key | Holds |
14+
|---|---|
15+
| `format` | The literal `sdm-scope/1`; a change to this format bumps it |
16+
| `language` | `python` or `java` |
17+
| `slices` | `S0` to `S3`, each with a one-line description from plan section 4 |
18+
| `backends` | The backends this language supports, each an `id` and a `what` |
19+
| `operations` | The public operations, each an `id`, its `api` in this language, its `layer` and its `modes` |
20+
| `paths` | One entry per operation x mode x backend in scope, with its `slice` and `cases` |
21+
| `cells` | One entry per Arrow type x column role, with its `expect` and `cases` |
22+
23+
Any entry may carry a `note`.
24+
25+
### Backends
26+
27+
A backend is where a path's data comes from, named by the provider the host runs, not by the vendor alone.
28+
`databricks-sql` and `databricks-spark` are separate backends because they reach the core through different drivers.
29+
A semantic model (`databricks-metric-view`, `snowflake-semantic-view`) is a backend of its own, because it is a source of its own in the public API.
30+
`frames` is pandas DataFrames in memory, whatever they were read from.
31+
`local-file` and `s3` are only the backends of `connectors.read`; the frames it returns are `frames`.
32+
`any` marks behavior that does not depend on the backend: a case on any backend covers it.
33+
The allowed IDs are the union across languages, fixed in the schema.
34+
35+
### Operations and modes
36+
37+
An operation ID is language-neutral (`query.materialize`, `client.predict`), so the two manifests can be compared row by row.
38+
`api` names the public symbols in this language, or is `null` until that language's API exists.
39+
`layer` is `core` when the Rust core implements the behavior, and `host` when the language binding keeps it (presentation, HTTP, opening connections, reading files).
40+
Host paths are in the manifest because they must not regress either, and because the Databricks jobs exercise them; the routing boundary only switches `core` paths.
41+
A mode is the input shape that changes which code runs: for `query.materialize`, the kind of question and whether rows are named by `indices` or passed as `predict`.
42+
Options that only change values inside one route (`anchor_time`, `num_context_samples`, `inference_config`, `outputs`, the model handle) are case variants, not modes.
43+
44+
### Paths
45+
46+
A path ID is `<operation>/<mode>/<backend>`, the plan's "one public operation with one input mode on one backend".
47+
`slice` is the slice whose gate the path must pass before it switches; S0 and S1 and S2 paths switch together after S2, and S3 paths after S3 (plan section 4).
48+
`cases` lists the contract case IDs covering the path.
49+
50+
### Cells
51+
52+
A cell ID is `<type>/<role>`.
53+
Types are the closed Arrow type set from plan section 3, with timestamps split by unit and zone: `int8` to `uint64`, `float32`, `float64`, `decimal128`, `decimal256`, `utf8`, `binary`, `date32`, `timestamp-ns`, `timestamp-us`, `timestamp-ns-tz`, `timestamp-us-tz`, `duration-ns`, `duration-us`, `time64`, `list`, `struct`, `map`, `dictionary-utf8`, `bool`, `null`.
54+
Roles are the parts a column plays in a request: `primary-key`, `foreign-key`, `time-column`, `feature`, `target`.
55+
`expect` is `accept`, `refuse`, or `unknown`.
56+
`accept` and `refuse` say what today's Python client does; `unknown` means nobody has checked yet.
57+
Host-side conversions (`large_utf8`, `timestamp[s]`, `large_list` and the like) are checked by L5 against the conversion tables, not here: a cell names the type the core receives.
58+
Every cell applies from S1, since S0 reads no data.
59+
60+
### Case IDs
61+
62+
Case IDs are the IDs of the shared L2 contract corpus, so one ID means the same case in every language.
63+
They match `^[a-z0-9][a-z0-9._-]*$`.
64+
Every `cases` list is empty until Phase 1a creates the corpus.
65+
A case may cover several paths and several cells, and is listed under each.
66+
67+
## Checks
68+
69+
The schema checks shape: keys, enums, the ID patterns, and unique case IDs within a list.
70+
These referential rules are checked by the coverage generator Phase 1a writes (they were checked by a one-off script for the first manifests):
71+
72+
1. Backend, operation, path and cell IDs are unique within a manifest.
73+
2. Each path's operation is declared, its mode is one of that operation's modes, and its backend is declared.
74+
3. Every declared operation, mode and backend appears in at least one path.
75+
4. From Phase 1a on, every case ID resolves to a case in the L2 corpus.
76+
5. At a slice gate, every path of that slice or an earlier one has at least one case, and so does every cell whose `expect` is not `unknown`.
77+
6. No cell is `unknown` at the S1 gate: Phase 1a and Phase 1b record the reference's outcome for each one, and a cell the core refuses where the reference accepted needs an owner-approved register entry.
78+
7. A path or cell is removed only with an owner decision, since dropping one is the kind of regression decision 3 names.
79+
80+
Validate the shape with:
81+
82+
```text
83+
uvx --from jsonschema --with pyyaml python -c "import json,sys,yaml,jsonschema; s=json.load(open('scope.schema.json')); [jsonschema.validate(yaml.safe_load(open(f)), s) for f in sys.argv[1:]]" scope-python.yaml scope-java.yaml
84+
```
85+
86+
## What the first manifests contain
87+
88+
Python has 281 paths (S0 2, S1 58, S2 61, S3 160) and Java 66 (S0 2, S1 46, S2 18); each has 140 cells.
89+
The operations come from `sdm_client.__all__`, `sdm_connectors.__all__` and `sdm_connectors.databricks.__all__`, as recorded in `tests/golden/public_api.json`.
90+
Python covers every backend of plan decision 2: DuckDB, SQLite, Postgres, Lakebase, Databricks over a warehouse and over Spark with metric views, Snowflake with semantic views, in-memory frames, and files read from Amazon S3.
91+
Java covers DuckDB, SQLite, Postgres and Databricks over a SQL warehouse, with no frames, sessions, tabular requests or S3 paths, since those wait until the owner adds Java scope.
92+
93+
Choices worth a second look:
94+
95+
- `query.validate` has no public method behind it: today PQL is validated inside `materialize()`, so S0 observes validation as the refusal raised before data is read, or as the resolved query.
96+
- Classification is split into `binary` and `multiclass`, because the jobs ask both (`inactivity_90d` and `product_color`) and their results have different columns.
97+
- `predict` and `fit` paths are per backend, following the plan's example path ("`predict` of a regression query on Snowflake").
98+
- Ranking and forecasting on Databricks and Lakebase are S3, as the plan says, but the S2 job matrix needs them; those paths carry a note pointing at [databricks-jobs.md](databricks-jobs.md), and the owner decides (J1 there).
99+
- Java's metric-view paths and its `databricks-serving` connection are assumptions, noted on each path, until the owner and the Java consumer confirm them.
100+
- A custom (non-SQL) data provider is not a path in either manifest: the S1 test provider and the swapped data layer are Rust-level checks, and a public provider API in Java is a question for when Java's API is designed.
101+
- Most cells outside the obvious ones are `unknown`; Phase 1b's case inventory (T0.7) and Phase 1a's runners resolve them against the reference.

0 commit comments

Comments
 (0)