From 8b2a87b4db3a1607968956c2fef93449c9be9ff8 Mon Sep 17 00:00:00 2001 From: Andre Date: Sat, 25 Jul 2026 06:47:13 -0300 Subject: [PATCH 1/9] feat: commit the data-divergence skill, the workflow hooks, and their wiring MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a repo-owned `data-divergence` skill for investigating why two datasets that should agree don't, and commits the three workflow hooks plus the settings file that registers them. The hooks enforce rules CLAUDE.md and specs/workflow.md already state as project rules (branch protection, PR description as merge body, CHANGELOG gate), but they lived only in one developer's ignored config — so the rules were documentation of one machine rather than a property of the repo. settings.json now refers to them via $CLAUDE_PROJECT_DIR; machine-specific entries move to the still-ignored settings.local.json. .gitignore keeps `.claude/*` blanket-ignored and un-ignores only these paths by name, so the AI Dev Kit's user-level skills in .claude/skills/ stay out. `git add -An .claude/` is the check that this holds. Co-Authored-By: Claude Opus 5 --- .claude/hooks/pr-merge-description.sh | 73 +++++++++ .claude/hooks/protect-main-branch.sh | 22 +++ .claude/hooks/require-changelog-entry.sh | 48 ++++++ .claude/settings.json | 30 ++++ .claude/skills/data-divergence/SKILL.md | 180 +++++++++++++++++++++++ .gitignore | 15 ++ CLAUDE.md | 1 + specs/tooling.md | 47 +++++- 8 files changed, 412 insertions(+), 4 deletions(-) create mode 100755 .claude/hooks/pr-merge-description.sh create mode 100755 .claude/hooks/protect-main-branch.sh create mode 100755 .claude/hooks/require-changelog-entry.sh create mode 100644 .claude/settings.json create mode 100644 .claude/skills/data-divergence/SKILL.md diff --git a/.claude/hooks/pr-merge-description.sh b/.claude/hooks/pr-merge-description.sh new file mode 100755 index 0000000..05fdc10 --- /dev/null +++ b/.claude/hooks/pr-merge-description.sh @@ -0,0 +1,73 @@ +#!/bin/bash +# PreToolUse hook: update PR description and inject it as the merge commit message +set -euo pipefail + +INPUT=$(cat) + +# Use Python for all JSON parsing and output (jq has snap confinement issues in this env) +python3 - "$INPUT" <<'PYEOF' +import json, sys, os, subprocess, tempfile, shlex + +raw = sys.argv[1] +try: + data = json.loads(raw) +except json.JSONDecodeError: + sys.exit(0) + +command = data.get("tool_input", {}).get("command", "") + +# Already has explicit body/subject — don't override user's intent +if any(f in command for f in ("--body", "--subject", "--body-file")): + sys.exit(0) + +# Rebase: individual commits keep their own messages +if "--rebase" in command: + sys.exit(0) + +# Extract PR ref (number, URL, or branch) — first non-flag token after 'gh pr merge' +import re +m = re.search(r'gh pr merge\s+([^\s-]\S*)', command) +pr_arg = m.group(1) if m else None + +# Fetch PR metadata +try: + cmd = ["gh", "pr", "view", "--json", "number,title,body"] + if pr_arg: + cmd = ["gh", "pr", "view", pr_arg, "--json", "number,title,body"] + result = subprocess.run(cmd, capture_output=True, text=True, timeout=15) + if result.returncode != 0: + sys.exit(0) + pr = json.loads(result.stdout) +except Exception: + sys.exit(0) + +number = str(pr["number"]) +title = pr["title"] +body = pr.get("body") or "" + +# Write body to temp file — avoids quoting/newline issues +with tempfile.NamedTemporaryFile(mode="w", suffix=".md", prefix="pr-body-", + dir="/tmp", delete=False) as f: + f.write(body) + tmpfile = f.name + +# Update the PR description on GitHub +try: + subprocess.run( + ["gh", "pr", "edit", number, "--title", title, "--body-file", tmpfile], + capture_output=True, timeout=15 + ) +except Exception: + pass # non-fatal + +# Build updated command with --subject and --body-file +new_cmd = command + " --subject " + shlex.quote(title) + " --body-file " + tmpfile + +# Return updated input to the harness +print(json.dumps({ + "hookSpecificOutput": { + "hookEventName": "PreToolUse", + "updatedInput": {"command": new_cmd} + } +})) +PYEOF diff --git a/.claude/hooks/protect-main-branch.sh b/.claude/hooks/protect-main-branch.sh new file mode 100755 index 0000000..009dcdf --- /dev/null +++ b/.claude/hooks/protect-main-branch.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# PreToolUse hook: block direct commits or pushes to main +set -euo pipefail + +INPUT=$(cat) +COMMAND=$(echo "$INPUT" | jq -r '.tool_input.command // ""') + +# Block any git push that targets main (including HEAD:main, refs/heads/main) +if echo "$COMMAND" | grep -qE '\bgit\b.*\bpush\b' && \ + echo "$COMMAND" | grep -qE '(\s|:)main(\s|$)'; then + printf '{"continue": false, "stopReason": "Direct push to main is blocked. Create a branch and open a PR instead."}' + exit 0 +fi + +# Block git commit when currently on main +if echo "$COMMAND" | grep -qE '\bgit\b.*\bcommit\b'; then + CURRENT_BRANCH=$(git branch --show-current 2>/dev/null || echo "") + if [ "$CURRENT_BRANCH" = "main" ]; then + printf '{"continue": false, "stopReason": "Direct commit to main is blocked. Create a branch first: git checkout -b "}' + exit 0 + fi +fi diff --git a/.claude/hooks/require-changelog-entry.sh b/.claude/hooks/require-changelog-entry.sh new file mode 100755 index 0000000..aeddc91 --- /dev/null +++ b/.claude/hooks/require-changelog-entry.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# PreToolUse hook: block `gh pr merge` unless this branch adds a specs/CHANGELOG.md entry. +# +# A hook can't *write* the entry (hooks are deterministic shell commands, not authors) — +# it enforces that one exists and tells the agent to write it. The signal is the branch +# diff: if CHANGELOG.md isn't touched relative to main, no entry was added. +# +# Self-gates on the command text rather than relying on the settings.json `if:` filter, +# so it stays correct even if that filter doesn't apply. +set -euo pipefail + +INPUT=$(cat) +COMMAND=$(echo "$INPUT" | jq -r '.tool_input.command // ""') + +# Not a merge → nothing to enforce. +echo "$COMMAND" | grep -qE '\bgh\b\s+pr\s+merge\b' || exit 0 + +CHANGELOG="specs/CHANGELOG.md" + +# Compare against the upstream default branch; fall back to a local main. +BASE="" +for ref in origin/main main; do + if git rev-parse --verify --quiet "$ref" >/dev/null 2>&1; then BASE="$ref"; break; fi +done + +# No main to compare against — don't block on an unanswerable question. +[ -z "$BASE" ] && exit 0 + +# `...` diffs against the merge-base, so this is "what this branch changed", not +# "how this branch differs from a main that moved on". +if git diff --name-only "$BASE...HEAD" -- "$CHANGELOG" | grep -q .; then + exit 0 +fi + +BRANCH=$(git branch --show-current 2>/dev/null || echo "?") +python3 - "$BRANCH" <<'PYEOF' +import json, sys +branch = sys.argv[1] +print(json.dumps({ + "continue": False, + "stopReason": ( + f"Merge blocked: this branch ({branch}) adds no entry to specs/CHANGELOG.md.\n" + "Add one at the top before merging — append-only, never edit an existing entry.\n" + "Header: ## [#] · " + branch + " · YYYY-MM-DD · \n" + "Body: at most 3 sentences. Replace the branch name with the PR URL after merge." + ), +})) +PYEOF diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..8e00d85 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,30 @@ +{ + "hooks": { + "PreToolUse": [ + { + "matcher": "Bash", + "hooks": [ + { + "type": "command", + "if": "Bash(gh pr merge*)", + "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/pr-merge-description.sh\"", + "timeout": 15, + "statusMessage": "Updating PR description and injecting into merge commit..." + }, + { + "type": "command", + "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/protect-main-branch.sh\"", + "timeout": 10, + "statusMessage": "Checking branch protection..." + }, + { + "type": "command", + "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/require-changelog-entry.sh\"", + "timeout": 10, + "statusMessage": "Checking for a CHANGELOG entry..." + } + ] + } + ] + } +} diff --git a/.claude/skills/data-divergence/SKILL.md b/.claude/skills/data-divergence/SKILL.md new file mode 100644 index 0000000..2379e8f --- /dev/null +++ b/.claude/skills/data-divergence/SKILL.md @@ -0,0 +1,180 @@ +--- +name: data-divergence +description: Investigate why two datasets that should agree don't — two pipelines writing the same logical table, a rollup vs the detail it aggregates, a dashboard vs its source, one environment vs another. Use when row counts, totals, or date ranges disagree and the question is what happened rather than just what differs. Covers localizing the first layer that diverges, diffing by grain and by key, reconciling across an aggregation boundary, reading Delta history and row_commit_version, why append-only tables diverge permanently, and what a fix actually costs. +--- + +# Investigating a data divergence + +Report what happened, not just what differs. + +**Verify the premise first.** "There's a divergence between X and Y" is a hypothesis, not a finding. +Reconcile before theorizing — a good share of reports turn out to be the wrong two columns compared, +and saying so plainly with numbers is a complete and useful answer. + +## Steps + +1. **Establish the two sides and the grain.** Name the exact objects and the key that identifies one + row on each side. If the two sides have *different* grain — one is a rollup of the other — jump to + [Across an aggregation boundary](#across-an-aggregation-boundary); a row-count comparison is + meaningless there. If the complaint came from a chart, read its query first — see + [Dashboards](#dashboards-lie-before-tables-do). +2. **Count, at every layer.** One `UNION ALL` down both paths — source → each intermediate → output — + with `COUNT(*)`, `MIN`/`MAX` of the partitioning/date column, and a distinct count of the key. + Find the **first layer where the two sides stop agreeing**; that one query eliminates everything + upstream of it. +3. **Diff by grain**, then by key (below). Stop and read the *pattern* before forming a theory. +4. **Read the Delta log** for the layer that first diverged (below). +5. **Check run state** — a missing tail is a failed job, not a logic bug. Look at job/pipeline run + history and events, and compare the *deployed* revision against the one in source control; a fix + that exists in the repo but was never shipped presents exactly like a code bug that isn't there. +6. **Report** what is proven, what is inferred, and what the retained logs can no longer answer. + +On Databricks, drive all of this with `execute_sql` rather than shelling out. Confirm which identity +the connection authenticates as before trusting environment isolation — an MCP server or shared +service principal may reach further than your own account does. + +## Diff by grain, then by key + +Group both sides by the dimension the report slices on and show **only** the buckets that differ: + +```sql +WITH a AS (SELECT <grain_col>, COUNT(*) n FROM <left> GROUP BY <grain_col>), + b AS (SELECT <grain_col>, COUNT(*) n FROM <right> GROUP BY <grain_col>) +SELECT COALESCE(a.<grain_col>, b.<grain_col>) AS g, a.n, b.n, COALESCE(b.n,0)-COALESCE(a.n,0) AS diff +FROM a FULL OUTER JOIN b USING (<grain_col>) +WHERE COALESCE(a.n,0) <> COALESCE(b.n,0) ORDER BY g +``` + +The shape of that output *is* the diagnosis: + +- **A missing block at the tail** → a failed or not-yet-run job. Go to step 5. +- **Both edges shifted by a constant, middle identical** → the same rows under a shifted window. + Nothing is missing; a derived column moved. +- **Scattered small deltas at one boundary** → usually the shift crossing a step in the data's own + distribution, not a second bug. Don't chase it separately. +- **Totals equal but distribution different** → the decisive reframe. This is not "rows missing", + it is "same rows, different values", and it points at a *column*, not at a join or a filter. + +Then join on the key and histogram the delta: + +```sql +SELECT <a.suspect_col> - <b.suspect_col> AS delta, COUNT(*) n -- or datediff() for dates +FROM <left> a JOIN <right> b USING (<key>) GROUP BY 1 ORDER BY n DESC +``` + +One bucket holding nearly all rows means a **formula**, not corruption. Select the other columns in +the same query: if everything else is byte-identical, the join keys and the transform are exonerated +and only that one column's provenance is still in question. Confirm by tracing a single key +end-to-end through every layer of both paths. + +## Across an aggregation boundary + +When one side is a rollup of the other, the two don't share a grain and `COUNT(*)` comparisons say +nothing. Reconcile instead: + +1. **The rollup's `GROUP BY` tuple is the grain.** Read it out of the transform, not out of the + table's column list. +2. **Additive measures must survive.** `SUM` of the detail equals `SUM` of the rollup — check + globally *and* per bucket, because a global total hides offsetting errors in both directions. +3. **Non-additive measures must not be summed at all.** `COUNT(DISTINCT)`, `MIN`/`MAX`, ratios and + averages don't compose across groups. Summing a `COUNT(DISTINCT)` column over-counts whenever one + entity spans more than one group. If it happens to reconcile anyway, that is a property of the + current data — often because the grouping columns are functionally dependent on the entity — not a + guarantee. Say so rather than banking it. +4. **Then full-outer join the regrouped detail to the rollup on the whole tuple** and count + mismatches by category: + +```sql +WITH s AS (SELECT <group_cols>, SUM(<measure>) m, COUNT(DISTINCT <entity>) e + FROM <detail> GROUP BY <group_cols>) +SELECT COUNT(*) AS groups, + COUNT(*) FILTER (WHERE g.<any_group_col> IS NULL) AS only_in_detail, + COUNT(*) FILTER (WHERE s.<any_group_col> IS NULL) AS only_in_rollup, + COUNT(*) FILTER (WHERE ABS(s.m - g.<measure>) > <tolerance>) AS measure_mismatch, + COUNT(*) FILTER (WHERE s.e <> g.<entity_count>) AS entity_mismatch +FROM s FULL OUTER JOIN <rollup> g USING (<group_cols>) +``` + +**Check you are comparing the right column.** Most "the aggregate doesn't match" reports are a +header-level amount being compared against a sum of line-level amounts. A parent total repeated onto +each child row is multiplied by the fan-out when summed — and frequently isn't the same quantity as +the sum of its children in the first place. Establish what each column *means* before treating a gap +as a defect. Use a float tolerance on money, and beware `FloatType` vs `DoubleType` on the two sides. + +## Reading the Delta log + +`DESCRIBE HISTORY` answers *when and by what*, but three traps cost real time: + +- **Filtering on `operation` alone hides overwrites.** An overwrite is often logged as `WRITE` with + `operationParameters.mode = 'Overwrite'`, so `operation NOT IN ('WRITE', …)` silently drops the + exact event you are hunting. Filter on the mode too, and page through *all* versions — the + interesting one is rarely in the most recent page. +- **`numOutputRows` per version reconstructs the growth curve** and distinguishes a first-run full + rebuild (one huge write) from steady incrementals (one small write per period). +- **v0's timestamp is when the table was created, not when the data began.** A `DROP` + recreate + resets versions to 0 and erases the prior incarnation. Absence of history is not absence of + events — say so rather than concluding nothing happened. + +**`_metadata.row_commit_version` is the sharpest tool in the box**, and the one to reach for first: + +```sql +SELECT _metadata.row_commit_version AS v, COUNT(*) n, MIN(<col>), MAX(<col>) +FROM <table> GROUP BY 1 ORDER BY 1 +``` + +It attributes **live rows** to the commit that wrote them, so it pins the blast radius to specific +commits — and unlike time travel it is not bounded by retention. Seeing all the bad rows in one +commit, with the very next commit already correct, converts a theory into a fact. + +Time travel does not survive: `VERSION AS OF` fails past `delta.deletedFileRetentionDuration` +(168 hours by default), and pipeline event logs age out too. Never build an investigation plan +around time-travelling a month-old event. + +## Append-only semantics: why divergence becomes permanent + +The load-bearing intuition. A **materialized view** is defined as a query over current inputs and +recomputes from scratch, so it self-heals when an input is re-derived. A **streaming table** — and an +insert-only `MERGE` on a batch path — appends each row once and never revisits it. + +So an append-only table freezes **every column it appends**, not just the one the freeze was designed +for. A pipeline that deliberately freezes a slowly-changing attribute at append time (a name, a +country, a price) also freezes whatever else it read from the static side of that join — including +columns nobody thought of as frozen. When the upstream is later re-derived, the recomputing side +moves and the appending side cannot, and the two drift apart permanently. + +Two corollaries worth stating in any report: + +- **"One side fixed itself" is a clue, not a reassurance.** It usually means that side took a + full-rebuild branch (a first-run overwrite, a full refresh), so it is *newer* — not necessarily + *righter*. +- **Ask whether the disputed column is a stable fact at all.** Anything derived from "now" at load + time — `current_date()`, a run-date anchor, an offset from a job parameter — silently re-derives + itself on every reload, so a reseed rewrites history that looked immutable. That is a source bug + wearing a pipeline divergence as a disguise, and no downstream reset fixes it permanently. + +## Dashboards lie before tables do + +When the report is "the dashboard doesn't match", suspect presentation before data. Check the tile's +query for a different grain, a filter the table query lacks, a `LEFT` vs `INNER` join, an implicit +`LIMIT`, and any latest-value binding — a chart that deliberately consolidates a renamed entity under +its *current* label will legitimately read differently from a table that keeps the frozen historical +label on every row. Confirm the two sides disagree at the same grain before opening the Delta log. + +## When two facts contradict + +If the evidence says two things that cannot both be true, an **identity assumption** is wrong, not +the evidence. In order of likelihood: the object was dropped and recreated; it is a view or +materialized view rather than a table (`DESCRIBE HISTORY` refuses views — that refusal is +information); a stale materialization was read; or the job is pointed at a different catalog or +environment than you assume. Check those before inventing a mechanism. + +Say plainly which parts of the reconstruction are proven by data and which are inference from a log +that no longer reaches back far enough. A confident wrong timeline is worse than an honest gap. + +## Before proposing a fix + +Divergences rarely have a free repair. A full refresh of an append-only pipeline re-derives the +broken column **and** discards every frozen value it was carrying — you may be trading a divergence +in one column for a divergence in another. Say which columns move, in both directions, before +recommending it. Check the run state of the other path first too: resetting a table whose writer is +currently failing leaves it empty. diff --git a/.gitignore b/.gitignore index 965d557..b4dc8d9 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,21 @@ notes/ .claude/* !.claude/commands/ !.claude/commands/** +# The workflow hooks referenced by specs/workflow.md and CLAUDE.md (branch protection, PR-description +# merge body, CHANGELOG gate). Committed so those documented rules hold for a fresh clone; note the +# scripts are inert until wired up in a settings file — see specs/tooling.md#hooks. +!.claude/hooks/ +!.claude/hooks/** +# ...and the settings file that registers them, kept portable via $CLAUDE_PROJECT_DIR. +# settings.local.json stays ignored — it is personal (permissions, machine-specific paths). +!.claude/settings.json +# The AI Dev Kit's skills are user-level tooling and stay ignored (specs/tooling.md#install-layout); +# only this repo's own skills are committed. Git won't descend into an excluded directory, so +# .claude/skills/ has to be re-included before a leaf under it can be. +!.claude/skills/ +.claude/skills/* +!.claude/skills/data-divergence/ +!.claude/skills/data-divergence/** .pytest_cache/ dist/ build/ diff --git a/CLAUDE.md b/CLAUDE.md index 6ec0838..0e78a1d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,6 +14,7 @@ Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutio - **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, and route job edits through `scripts/sdk_generate_template_job.py` + `make deploy` ([skills](specs/tooling.md#skills)). - **Library/SDK docs** (PySpark, Databricks SDK, uv, ruff) → `context7` MCP, not memory or web search. - **Cloud spend / cost analysis** → `aws-billing-cost` MCP (`AWS_PROFILE=costs`) + `/project-costs`. **AWS docs** → `aws-documentation` MCP. +- **Records disagree between two tables** (batch vs SDP, dashboard vs its table, prod vs staging) → the `data-divergence` skill ([skills](specs/tooling.md#skills)) before writing ad-hoc diff SQL. - Use the `dev` profile unless told otherwise (`prod` for prod ops). If MCP tools are unavailable, fall back to CLI/SDK and flag it. - **MCP calls run as the prod SP, not as you** — `dev` is your user account, but the `databricks` MCP server is pinned to `DEFAULT`, which resolves to the same `template-sp` that `prod` uses. It can read/write `prod` tables; the catalog is the guardrail ([why](specs/tooling.md#mcp-runs-as-the-production-service-principal)). diff --git a/specs/tooling.md b/specs/tooling.md index 7a70296..6547dfd 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -32,8 +32,16 @@ undetectable from a session. This repo carried exactly that: a project-scoped in `.github/skills/` and `.ai-dev-kit/` remain (gitignored, stale, and only relevant to Copilot). `.gitignore` keeps all of it out of the repo: `.mcp.json`, `.ai-dev-kit/`, `.github/skills/`, and -`.claude/*`. The one exception is `.claude/commands/`, which is un-ignored so project slash commands -(e.g. `/project-costs`) are committed and every developer gets them. +`.claude/*`. Four things are un-ignored so every developer gets them: `.claude/commands/` (project +slash commands, e.g. `/project-costs`), `.claude/hooks/` and `.claude/settings.json` (see +[Hooks](#hooks)), and `.claude/skills/data-divergence/` — this repo's own skill, named explicitly +rather than by a wildcard so the kit's skills in the same directory stay ignored. Adding another repo +skill means adding another negation pair; `git add -An .claude/` should still stage only our files, +and is the check to re-run after touching those lines. + +**`.claude/settings.local.json` stays ignored** — it is personal (permissions, machine-specific +paths, per-developer MCP toggles). Anything absolute or specific to one machine belongs there, not in +the committed `settings.json`; Claude Code merges the two. ## MCP servers @@ -77,6 +85,31 @@ uses `prod`), which is what `make whoami` reports on. If MCP tools are unavailable in a session, fall back to the `databricks` CLI or `databricks-sdk` directly (or `aws` CLI / web search for the AWS and context7 cases) — but flag the fallback. +## Hooks + +`.claude/hooks/` holds the three shell hooks that enforce the git workflow in +[workflow.md](workflow.md). They are **committed** — the rules they enforce are stated as project +rules in `CLAUDE.md`, so shipping the scripts is what makes those statements true for a fresh clone +rather than a description of one machine's setup. + +| Hook | Fires on | Effect | +|---|---|---| +| `protect-main-branch.sh` | any Bash `git commit` / `git push` | blocks a commit made while on `main`, and any push targeting `main`. | +| `require-changelog-entry.sh` | Bash `gh pr merge` | blocks the merge unless the branch diff touches `specs/CHANGELOG.md`, compared against `origin/main`/`main` via a merge-base (`...`) diff. | +| `pr-merge-description.sh` | Bash `gh pr merge` | pushes the PR title/body to GitHub, then rewrites the command with `--subject`/`--body-file` so the description becomes the merge commit message. Skips if `--body`/`--subject`/`--body-file` or `--rebase` is already present. | + +The scripts are portable — no absolute paths, no secrets — and are committed mode `755`. + +**The wiring is committed too.** A hook only runs if a settings file registers it, so +`.claude/settings.json` is un-ignored and registers all three as `PreToolUse` command hooks matching +`Bash`. It refers to them as `$CLAUDE_PROJECT_DIR/.claude/hooks/<name>.sh`, never as an absolute +path — that variable is what keeps the file valid in any clone, and a hardcoded path is the one edit +that would quietly break it for everyone else. Keep machine-specific hooks (an update check pointing +into `~/.ai-dev-kit/`, say) in `.claude/settings.local.json` instead; the two files are merged. + +`require-changelog-entry.sh` and `protect-main-branch.sh` self-gate on the command text instead of +trusting a settings-level `if:` filter, so they stay correct however they are registered. + ## Databricks CLI Used for bundle work and as the MCP fallback. The day-to-day surface is wrapped in the `Makefile` @@ -96,11 +129,17 @@ is the kit's entry point for CLI, auth, and bundle work — load it first, then - **databricks-python-sdk** — SDK code under `src/template/` and in `scripts/`. - **databricks-unity-catalog**, **databricks-aibi-dashboards**, **databricks-spark-declarative-pipelines**, etc. — invoke when the task is squarely in that area. +- **data-divergence** — *this repo's own skill*, not the kit's: investigating why two datasets that + should agree don't. Written generically (no table or column names from this project), so it covers + batch vs SDP, a gold rollup vs the silver it aggregates, a dashboard tile vs its source, and prod + vs staging alike. Lives in `.claude/skills/data-divergence/` and is committed; see the un-ignore + note above. Two gotchas. Some skills' frontmatter `name:` differs from their directory (`databricks` declares `databricks-core`; `analyze-mlflow-trace` declares `analyzing-mlflow-trace`) — **invoke by directory -name**, which is what the session's skill list shows; the frontmatter name is not the handle. And -`/project-costs` is **not** a kit skill; it's this repo's own committed slash command +name**, which is what the session's skill list shows; the frontmatter name is not the handle. +(`data-divergence` declares a matching name, so it has no such split.) And `/project-costs` is +**not** a kit skill either; it's this repo's own committed slash command (`.claude/commands/project-costs.md`) wrapping `scripts/project_costs.py`. `databricks-core` also cross-references skills by their *post-migration* names — it points at From 379df0d6bf9a5178c873ec55a4380a4b7cc8a617 Mon Sep 17 00:00:00 2001 From: Andre <andre.f.salvati@gmail.com> Date: Sat, 25 Jul 2026 06:55:27 -0300 Subject: [PATCH 2/9] refactor: convert the project slash commands to skills, with worked examples MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /project-costs and /sql-diagram were slash commands, so their instructions loaded only when the user typed them. As skills their descriptions sit in context and match on relevance — "why is our Databricks spend up?" now reaches project-costs without anyone remembering the command exists. Typing the slash name still works. Each ships an example.md alongside the SKILL.md, walking a committed artifact end to end: reports/sql-diagram/job_spend_plan.* for the plan-mode reading (CTE sub-pipelines, LEFT joins that must stay LEFT, the range predicates that stop a slowly-changing dimension fanning out), and reports/cost/2026-07-22.md for the cost analysis (per-active-day normalisation, reconciling the attributed total, treating SQL-warehouse silence as a finding). The examples reference those artifacts rather than duplicating them, so there is one copy to keep current. .claude/commands/ is gone; the .gitignore block is restructured to un-ignore the three repo skills by name, with the no-wildcard rule stated inline — a wildcard there would commit the AI Dev Kit's user-level skills. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- .../project-costs/SKILL.md} | 10 +++ .claude/skills/project-costs/example.md | 59 ++++++++++++++ .../sql-diagram/SKILL.md} | 10 +++ .claude/skills/sql-diagram/example.md | 80 +++++++++++++++++++ .gitignore | 23 +++--- specs/tooling.md | 49 +++++++----- 6 files changed, 203 insertions(+), 28 deletions(-) rename .claude/{commands/project-costs.md => skills/project-costs/SKILL.md} (90%) create mode 100644 .claude/skills/project-costs/example.md rename .claude/{commands/sql-diagram.md => skills/sql-diagram/SKILL.md} (90%) create mode 100644 .claude/skills/sql-diagram/example.md diff --git a/.claude/commands/project-costs.md b/.claude/skills/project-costs/SKILL.md similarity index 90% rename from .claude/commands/project-costs.md rename to .claude/skills/project-costs/SKILL.md index 94568da..a9d24ed 100644 --- a/.claude/commands/project-costs.md +++ b/.claude/skills/project-costs/SKILL.md @@ -1,3 +1,10 @@ +--- +name: project-costs +description: Run the project cost report and write the analysis into it. Use when asked about this project's cloud spend, cost anomalies, spikes or trends, DBU/DSU consumption, per-job or per-pipeline cost, or the AWS vs Databricks split. Runs `make project-costs` (AWS Cost Explorer + Databricks system.billing), then analyses the generated report and replaces its Analysis placeholder. See example.md for a committed report read end to end. +--- + +# Project cost analysis + Run the project cost script, analyze the output for anomalies, spikes and trends, and write the analysis into the generated markdown report. @@ -77,6 +84,9 @@ to dollars before calling them big or small. charging the AWS account for EC2. AWS spend is therefore a *proxy for job activity*, never a measure of pipeline cost. +`example.md` walks a committed report showing what each of these sections looks like when written +against real numbers. + ## Caveats to respect - **Databricks USD is list price.** `list_prices` excludes account discounts and commit contracts, diff --git a/.claude/skills/project-costs/example.md b/.claude/skills/project-costs/example.md new file mode 100644 index 0000000..74aead0 --- /dev/null +++ b/.claude/skills/project-costs/example.md @@ -0,0 +1,59 @@ +# Worked example — `reports/cost/2026-07-22.md` + +A committed 30-day report, force-added past the `reports/cost/` gitignore so one finished example +survives. Read the file itself; this page is about *why* its Analysis section is written the way it +is. Do not copy its numbers into a new report — they are a snapshot of one window. + +## What the numbers were + +$39.81 total: Databricks $39.43 (99.0%) at list, AWS $0.38 (1.0%). Jobs Serverless 68.09 DBU +($23.83), SQL Serverless 21.57 DBU ($15.10), storage 20.66 DSU ($0.48). Attributed to jobs: $21.49 +of $39.43. + +## What the analysis did with them, and why + +**It led with the split, not with AWS.** $0.38 of AWS is noise; opening with it would bury the fact +that the entire cost conversation is about DBUs. Section order in the report follows the file; +narrative order should follow the money. + +**It converted a spike to dollars before judging it.** The AWS week of 2026-07-13 is ~5× the +surrounding baseline, which sounds alarming until you say it is $0.1925. Ratios without dollars +mislead on a project this small. + +**It used the daily `<details>` block to attribute the spike to a date.** The weekly pivot only says +"that week"; the daily block pinned it to 2026-07-16 and identified Cost Explorer API calls as the +driver — i.e. the report measuring itself, since Cost Explorer bills per request. That conclusion is +unreachable from stdout alone, which is why the skill insists on reading the report file. + +**It normalized the edge weeks before claiming a trend.** Raw weekly totals suggested a decline; +per-day Jobs Serverless ($1.22/day → $0.42/day → ~$0.72/day) showed a stable baseline with one +late-June spike instead. Opposite conclusion, same data. + +**It treated SQL Serverless silence as a finding.** Two and a half weeks at exactly $0.00, then +$2.90 and $3.58 on two days. The absence is the signal — nothing scheduled touches the warehouse and +nobody opens the dashboard on an ordinary day — and at $15.10 it is the second most expensive thing +in the project despite running about five days out of thirty. + +**It reconciled before trusting the per-job table.** $21.49 attributed against $39.43 total, with +the ~$17.94 gap explained by the SQL warehouse carrying no `job_id`. Because that reconciles, the +breakdown is trustworthy *as a picture of scheduled work only* — stated explicitly rather than +letting the reader assume it covers everything. + +**It compared per active day, never raw totals.** prod ran 31 days at $0.46/day; staging ran 5 days +at $1.35/day — 2.9× prod's daily burn, invisible in the raw column where prod looks far more +expensive. The `Days` column exists for exactly this. + +**It found the batch-vs-SDP gap and argued it was real.** `job1_prod` $6.58 vs `job1_sdp_prod` $4.03 +for the same medallion tables, repeated in staging ($2.48 vs $1.62). A 35–39% gap holding over 31 +days *and* across two environments is what separates a finding from noise — the durability is the +argument, not the single number. + +**It checked the integration tests.** `job1_prod_integration` at $3.78 is 57% of the pipeline it +validates, and in staging the integration test cost *more* than the job under test. Easy to skim +past; the skill calls them out because of this. + +## The shape to reproduce + +One paragraph per section, every claim carrying its number, comparisons normalized before they are +made, and absences reported as findings. No filler, no restating tables that are already in the file +directly above. diff --git a/.claude/commands/sql-diagram.md b/.claude/skills/sql-diagram/SKILL.md similarity index 90% rename from .claude/commands/sql-diagram.md rename to .claude/skills/sql-diagram/SKILL.md index 5c8ed70..bbaba63 100644 --- a/.claude/commands/sql-diagram.md +++ b/.claude/skills/sql-diagram/SKILL.md @@ -1,3 +1,10 @@ +--- +name: sql-diagram +description: Diagram a SQL query and explain what it shows — either its execution steps (mode=plan) or its column lineage (mode=lineage). Use when asked to visualize, diagram, explain or review what a query does, how it joins its tables, or where an output column comes from. Wraps `make sql-diagram`, which emits .mmd and .svg into reports/sql-diagram/. See example.md for a worked reading of a committed diagram. +--- + +# Diagramming a SQL query + Diagram a SQL query and explain what it shows — either its execution steps or its column lineage. ## Steps @@ -74,6 +81,9 @@ has no comment the space is blank, and that absence is itself worth reporting. time-range predicate fans rows out and silently multiplies aggregates — this repo has been bitten by exactly that (see the `#47` entry in `specs/CHANGELOG.md`). +See `example.md` for a committed diagram read end to end, including what each of these points looks +like when it actually fires. + ## Limits worth stating rather than hiding - `SELECT *` errors out in lineage mode by design — tracing it needs the table schemas, which the diff --git a/.claude/skills/sql-diagram/example.md b/.claude/skills/sql-diagram/example.md new file mode 100644 index 0000000..8891cff --- /dev/null +++ b/.claude/skills/sql-diagram/example.md @@ -0,0 +1,80 @@ +# Worked example — `job_spend_plan` + +A committed `mode=plan` diagram of the per-job spend query from `scripts/project_costs.py`. The +three artifacts live in `reports/sql-diagram/` and are force-added past the gitignore precisely so +this example survives: + +- `job_spend_plan.sql` — the query as analysed, f-string placeholders already resolved +- `job_spend_plan.mmd` — the graph +- `job_spend_plan.svg` — the same graph, for prose that can't render Mermaid + +Regenerate it with: + +```bash +make sql-diagram sql=reports/sql-diagram/job_spend_plan.sql name=job_spend_plan comments=1 +``` + +Because the committed `.sql` is the post-substitution query, that command reproduces the diagram +exactly — which is the whole reason the `.sql` is committed alongside the picture. + +## The graph + +```mermaid +flowchart LR + n0[("<b>SCAN system.lakeflow.pipelines</b>")] + n1["<b>AGGREGATE</b><br/>GROUP BY pipeline_id<br/>MAX_BY(name, change_time) AS name"] + n2[("<b>SCAN pipe_names</b>")] + n3[("<b>SCAN system.billing.list_prices</b>")] + n4[("<b>SCAN system.billing.usage</b>")] + n5{{"<b>JOIN 1 · LEFT · p</b><br/>u.sku_name = p.sku_name<br/>and u.usage_end_time >= p.price_start_time<br/>and (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)"}} + n6{{"<b>JOIN 2 · LEFT · n</b><br/>u.usage_metadata.dlt_pipeline_id = n.pipeline_id"}} + n7["<b>WHERE</b><br/>u.usage_date >= CURRENT_DATE - INTERVAL '30' DAYS<br/>…"] + n8["<b>AGGREGATE</b><br/>GROUP BY entity, kind, u.usage_unit<br/>SUM(u.usage_quantity) AS quantity<br/>SUM(`_a_0`) AS usd<br/>COUNT(`_a_1`) AS active_days"] + n9["<b>SORT</b><br/>usd DESC"] + n10(["<b>OUTPUT</b><br/>entity<br/>kind<br/>quantity<br/>usage_unit<br/>usd<br/>active_days"]) + n0 --> n1 + n1 --> n2 + n4 --> n5 + n3 --> n5 + n5 --> n6 + n2 --> n6 + n6 --> n7 + n7 --> n8 + n8 --> n9 + n9 --> n10 +``` + +## How to read it + +**Shape first.** Three source tables, two joins, one CTE, grouped to one row per entity × kind × +unit and sorted by dollars. `system.billing.usage` is the fact; the other two are lookups. + +**The CTE is its own sub-pipeline.** `n0 → n1` is `pipe_names` being built (dedupe +`system.lakeflow.pipelines` to one current name per `pipeline_id` via `MAX_BY`), and `n2` is the +`SCAN` that reads the finished CTE back. That two-node shape is what a CTE always looks like here — +it isn't a duplicate scan of the same table. + +**Both joins are `LEFT`, and that is load-bearing.** Usage rows survive even when no price row +matches or the pipeline has no name. An `INNER JOIN` here would silently drop unpriced SKUs and +under-report spend — exactly the kind of thing to say out loud, because it changes what a missing +row in the output means. + +**JOIN 1 carries three predicates, not one.** The equality on `sku_name` plus two range predicates +on `price_start_time` / `price_end_time`. `list_prices` is a slowly-changing dimension with one row +per price period, so those ranges are what pick a single price rather than fanning every usage row +out across every historical price. This is the under-constrained-join failure mode the skill warns +about — here it is correctly constrained, and worth naming as such. + +**`usage_metadata` is the hub.** Three separate expressions read it (`job_name`, `dlt_pipeline_id`, +`job_id`), so it is where a schema change would hurt most. Note that lineage mode would collapse all +three to `usage_metadata`, the struct root — this is the case the skill's "struct columns collapse" +limit describes, and the reason `plan` is the better mode for this query. + +**`_a_0` and `_a_1` are synthetic.** `sqlglot` lifted the `usage_quantity * pricing…` product and +the `DISTINCT usage_date` out of their aggregates. Read the intent from the `.sql` — `usd` is a +priced sum, `active_days` a distinct-day count — rather than repeating the placeholder names at the +user. + +**The grey comment lines** under the two `system.billing` scans came from `comments=1` reading Unity +Catalog. They are fetched, never authored. `pipe_names` has none because a CTE isn't a catalog +object. diff --git a/.gitignore b/.gitignore index b4dc8d9..a2cb45e 100644 --- a/.gitignore +++ b/.gitignore @@ -2,24 +2,27 @@ notes/ .databricks/ .vscode/ .venv/ +# .claude/ is ignored wholesale; this repo's own agent tooling is un-ignored BY NAME below. +# Never use a wildcard here: the AI Dev Kit installs its user-level skills into .claude/skills/ +# (specs/tooling.md#install-layout) and they must stay out of the repo. Git will not descend into an +# excluded directory, so .claude/skills/ is re-included before its children are re-excluded. +# After editing this block run `git add -An .claude/` — it must list only the paths named here. .claude/* -!.claude/commands/ -!.claude/commands/** -# The workflow hooks referenced by specs/workflow.md and CLAUDE.md (branch protection, PR-description -# merge body, CHANGELOG gate). Committed so those documented rules hold for a fresh clone; note the -# scripts are inert until wired up in a settings file — see specs/tooling.md#hooks. +# Workflow hooks + the settings file that registers them (specs/tooling.md#hooks). Committed so the +# rules CLAUDE.md and specs/workflow.md state hold for a fresh clone, not just on one machine. +# settings.local.json stays ignored — it is personal (permissions, machine-specific paths). !.claude/hooks/ !.claude/hooks/** -# ...and the settings file that registers them, kept portable via $CLAUDE_PROJECT_DIR. -# settings.local.json stays ignored — it is personal (permissions, machine-specific paths). !.claude/settings.json -# The AI Dev Kit's skills are user-level tooling and stay ignored (specs/tooling.md#install-layout); -# only this repo's own skills are committed. Git won't descend into an excluded directory, so -# .claude/skills/ has to be re-included before a leaf under it can be. +# Project skills — one negation pair each; add another pair when adding a skill. !.claude/skills/ .claude/skills/* !.claude/skills/data-divergence/ !.claude/skills/data-divergence/** +!.claude/skills/project-costs/ +!.claude/skills/project-costs/** +!.claude/skills/sql-diagram/ +!.claude/skills/sql-diagram/** .pytest_cache/ dist/ build/ diff --git a/specs/tooling.md b/specs/tooling.md index 6547dfd..639d58c 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -32,12 +32,15 @@ undetectable from a session. This repo carried exactly that: a project-scoped in `.github/skills/` and `.ai-dev-kit/` remain (gitignored, stale, and only relevant to Copilot). `.gitignore` keeps all of it out of the repo: `.mcp.json`, `.ai-dev-kit/`, `.github/skills/`, and -`.claude/*`. Four things are un-ignored so every developer gets them: `.claude/commands/` (project -slash commands, e.g. `/project-costs`), `.claude/hooks/` and `.claude/settings.json` (see -[Hooks](#hooks)), and `.claude/skills/data-divergence/` — this repo's own skill, named explicitly -rather than by a wildcard so the kit's skills in the same directory stay ignored. Adding another repo -skill means adding another negation pair; `git add -An .claude/` should still stage only our files, -and is the check to re-run after touching those lines. +`.claude/*`. Un-ignored so every developer gets them: `.claude/hooks/` and `.claude/settings.json` +(see [Hooks](#hooks)), and this repo's own three skills — `data-divergence`, `project-costs` and +`sql-diagram` — each named explicitly rather than by a wildcard, so the kit's skills in the same +directory stay ignored. Adding a repo skill means adding another negation pair; `git add -An +.claude/` should still stage only our files, and is the check to re-run after touching those lines. + +There is no `.claude/commands/` any more: `/project-costs` and `/sql-diagram` began as slash commands +and were converted to skills, so their instructions load on relevance rather than only when typed. +Typing `/project-costs` still works — it resolves to the skill. **`.claude/settings.local.json` stays ignored** — it is personal (permissions, machine-specific paths, per-developer MCP toggles). Anything absolute or specific to one machine belongs there, not in @@ -129,18 +132,28 @@ is the kit's entry point for CLI, auth, and bundle work — load it first, then - **databricks-python-sdk** — SDK code under `src/template/` and in `scripts/`. - **databricks-unity-catalog**, **databricks-aibi-dashboards**, **databricks-spark-declarative-pipelines**, etc. — invoke when the task is squarely in that area. -- **data-divergence** — *this repo's own skill*, not the kit's: investigating why two datasets that - should agree don't. Written generically (no table or column names from this project), so it covers - batch vs SDP, a gold rollup vs the silver it aggregates, a dashboard tile vs its source, and prod - vs staging alike. Lives in `.claude/skills/data-divergence/` and is committed; see the un-ignore - note above. - -Two gotchas. Some skills' frontmatter `name:` differs from their directory (`databricks` declares -`databricks-core`; `analyze-mlflow-trace` declares `analyzing-mlflow-trace`) — **invoke by directory -name**, which is what the session's skill list shows; the frontmatter name is not the handle. -(`data-divergence` declares a matching name, so it has no such split.) And `/project-costs` is -**not** a kit skill either; it's this repo's own committed slash command -(`.claude/commands/project-costs.md`) wrapping `scripts/project_costs.py`. +### This repo's own skills + +Committed under `.claude/skills/`, and **not** part of the kit — don't expect `install.sh` to +update them, and do keep them in sync with the code they wrap. + +- **data-divergence** — investigating why two datasets that should agree don't. Written generically + (no table or column names from this project), so it covers batch vs SDP, a gold rollup vs the + silver it aggregates, a dashboard tile vs its source, and prod vs staging alike. +- **project-costs** — wraps `scripts/project_costs.py` via `make project-costs`: runs the report, + then writes the analysis into its `## Analysis` placeholder. +- **sql-diagram** — wraps `scripts/sql_diagram.py` via `make sql-diagram`: query plan or column + lineage as `.mmd` + `.svg`, plus how to read each mode. + +The latter two each ship an `example.md` beside the `SKILL.md`, walking a committed artifact +(`reports/cost/2026-07-22.md`, `reports/sql-diagram/job_spend_plan.*`) to show what good output +looks like. Those artifacts are force-added past the `reports/` gitignore for exactly that reason — +if you regenerate them, re-add with `git add -f` or the example silently goes stale. + +One gotcha on the kit's skills. Some have a frontmatter `name:` that differs from their directory +(`databricks` declares `databricks-core`; `analyze-mlflow-trace` declares `analyzing-mlflow-trace`) +— **invoke by directory name**, which is what the session's skill list shows; the frontmatter name +is not the handle. All three repo skills above declare a matching name, so they have no such split. `databricks-core` also cross-references skills by their *post-migration* names — it points at `/databricks-dabs` and `databricks-data-discovery`, neither of which is installed yet. Read those as From 06c07f1df9fbc89924d1a25284bcd05f5abec02d Mon Sep 17 00:00:00 2001 From: Andre <andre.f.salvati@gmail.com> Date: Sat, 25 Jul 2026 07:33:27 -0300 Subject: [PATCH 3/9] chore: gitignore all of reports/, and move the examples out of it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit reports/ is generated output — coverage, spend reports carrying account cost figures, query diagrams — and none of it should be in git. The three directory-level rules are replaced by a single `reports/`. Four artifacts were tracked there, force-added as worked examples for the project-costs and sql-diagram skills. .gitignore only governs untracked files, so ignoring the directory would have left them in the index and made the rule a lie; they are moved to .claude/skills/<skill>/examples/ instead, which is a committed path. The examples and the two README links keep working, and reports/ becomes purely the place the tools write. The "git add -f one to keep it as an example" advice is removed from both the gitignore comment and the sql-diagram skill, and replaced with the rule that a new example is a copy into the skill's examples/ directory. Also drops the now redundant reports/coverage/ entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- .claude/skills/project-costs/example.md | 9 ++-- .../project-costs/examples}/2026-07-22.md | 0 .claude/skills/sql-diagram/SKILL.md | 44 +++++++++++++++++-- .claude/skills/sql-diagram/example.md | 16 ++++--- .../sql-diagram/examples}/job_spend_plan.mmd | 0 .../sql-diagram/examples}/job_spend_plan.sql | 0 .../sql-diagram/examples}/job_spend_plan.svg | 0 .gitignore | 15 +++---- README.md | 4 +- specs/tooling.md | 9 ++-- 10 files changed, 68 insertions(+), 29 deletions(-) rename {reports/cost => .claude/skills/project-costs/examples}/2026-07-22.md (100%) rename {reports/sql-diagram => .claude/skills/sql-diagram/examples}/job_spend_plan.mmd (100%) rename {reports/sql-diagram => .claude/skills/sql-diagram/examples}/job_spend_plan.sql (100%) rename {reports/sql-diagram => .claude/skills/sql-diagram/examples}/job_spend_plan.svg (100%) diff --git a/.claude/skills/project-costs/example.md b/.claude/skills/project-costs/example.md index 74aead0..ec75441 100644 --- a/.claude/skills/project-costs/example.md +++ b/.claude/skills/project-costs/example.md @@ -1,8 +1,9 @@ -# Worked example — `reports/cost/2026-07-22.md` +# Worked example — `examples/2026-07-22.md` -A committed 30-day report, force-added past the `reports/cost/` gitignore so one finished example -survives. Read the file itself; this page is about *why* its Analysis section is written the way it -is. Do not copy its numbers into a new report — they are a snapshot of one window. +A finished 30-day report, kept here rather than under `reports/` (which is entirely gitignored +generated output). Read the file itself; this page is about *why* its Analysis section is written the +way it is. Do not copy its numbers into a new report — they are a snapshot of one window, and a live +run writes to `reports/cost/YYYY-MM-DD.md`. ## What the numbers were diff --git a/reports/cost/2026-07-22.md b/.claude/skills/project-costs/examples/2026-07-22.md similarity index 100% rename from reports/cost/2026-07-22.md rename to .claude/skills/project-costs/examples/2026-07-22.md diff --git a/.claude/skills/sql-diagram/SKILL.md b/.claude/skills/sql-diagram/SKILL.md index bbaba63..2a91637 100644 --- a/.claude/skills/sql-diagram/SKILL.md +++ b/.claude/skills/sql-diagram/SKILL.md @@ -16,12 +16,19 @@ Diagram a SQL query and explain what it shows — either its execution steps or `mode=lineage` answers "where does this output column come from". When the user asks about joins, stages, filters or ordering, they want `plan`. 3. Run `make sql-diagram sql=<path> name=<basename> comments=1` via Bash. It writes three files to - `reports/sql-diagram/`: `<basename>.sql` (the query as analysed), `.mmd` and `.svg` — all - gitignored, so `git add -f` them only if they are meant to be a committed example. Pass - `--stdout` to `scripts/sql_diagram.py` for a throwaway look with no files written. + `reports/sql-diagram/`: `<basename>.sql` (the query as analysed), `.mmd` and `.svg`. All of + `reports/` is gitignored generated output — never `git add -f` out of it. To keep a diagram as a + committed example, copy the trio into `.claude/skills/sql-diagram/examples/`. Pass `--stdout` to + `scripts/sql_diagram.py` for a throwaway look with no files written. 4. Read the `.mmd`, show it in a ```mermaid fence, and explain it (see below). The `.svg` is the same graph for linking from prose where no Mermaid renderer is available. +**Explaining a query and reviewing one are different jobs.** For an explanation the diagram is +enough. For a *review*, read the `.sql` alongside it and treat the graph as an index into the text: +it is authoritative on scans, joins and join predicates, and silent on filters, windows and +projections (see the blind spots below). Defects severe enough to produce wrong numbers usually live +in exactly the parts the picture omits. + The emitted `.sql` is what makes the diagram auditable: it is the query *after* any f-string placeholders were filled in, so `make sql-diagram sql=reports/sql-diagram/<basename>.sql` reproduces the diagram exactly. When you commit a diagram as an example, commit its `.sql` with it. @@ -43,11 +50,26 @@ that reads it. - **Extra `ON` predicates beyond the equality keys** are listed under the keys as `and …`. On a slowly-changing dimension those range predicates are what stop the join fanning out; call them out rather than treating them as noise. +- **Window functions are not drawn.** A CTE whose job is a `ROW_NUMBER()` / `RANK()` / `LAG()` + collapses to a bare `SCAN <cte>` with no node for the window, so its `PARTITION BY` / `ORDER BY` + are invisible — and a downstream `WHERE rank <= n` then filters on a column with no visible origin + anywhere in the graph. Whenever a rank, top-N or dedupe is involved, read the `PARTITION BY` off + the `.sql` and state it; the picture cannot show whether the ranking is right. +- **`WHERE` is only drawn when it sits above a join.** Filters inside a CTE, and the `WHERE` of a + join-free query, do not appear; neither does `LIMIT`. `HAVING` appears disguised as a synthetic + `… AS _h` line inside the `AGGREGATE` node, and `QUALIFY` not at all. Never conclude "this scans + the whole table" or "there is no date filter" from the graph — count the filters in the `.sql` and + say how many the diagram omitted. +- **`OUTPUT` lists aliases, never expressions.** A `COALESCE(rate, 1.0)` default, a cast or a + division in the select list shows only as its output name. Read the projection list from the + `.sql` and call out any silent default — that is where a broken join stops being a visibly empty + column and starts being a confidently wrong number. - **This is the logical plan, not the physical one.** Databricks reorders joins, chooses broadcast versus shuffle, and prunes columns. Say "as written" — and if the real execution matters, point at the query profile in the UI or `EXPLAIN FORMATTED`, which is the only authority on what ran. - `AGGREGATE` may show synthetic operand names (`_a_0`) for `DISTINCT`/expression arguments that - `sqlglot` lifted out. Read the intent off the original SQL rather than repeating the placeholder. + `sqlglot` lifted out. Read the intent off the original SQL rather than repeating the placeholder — + `COUNT(\`_a_0\`)` and `COUNT(DISTINCT …)` are indistinguishable in the picture. ## Reading `mode=lineage` @@ -80,6 +102,15 @@ has no comment the space is blank, and that absence is itself worth reporting. - Join predicates that look under-constrained. A join on a slowly-changing dimension without a time-range predicate fans rows out and silently multiplies aggregates — this repo has been bitten by exactly that (see the `#47` entry in `specs/CHANGELOG.md`). +- **What each join key *means*, not just that it exists.** The plan normalizes predicates, so a + join written `c.id = r.product_id` prints as `r.product_id = c.id`, and an equality between two + integer ids looks correct no matter which entities they identify. Pull both schemas + (`mcp__databricks__get_table_stats_and_schema`) and compare domains and value ranges. A + customer-id-to-product-id join can match **100% of rows** — no NULLs, no fan-out, no error — and + hand back a plausible, entirely fabricated dimension column. +- **The grain, against the dimensions hung off it.** If the pipeline aggregates to `A × B` and then + joins a dimension that varies *within* `A × B`, either that dimension is fabricated or the join + fans out and inflates every measure. The `AGGREGATE` node's `GROUP BY` line is where to check. See `example.md` for a committed diagram read end to end, including what each of these points looks like when it actually fires. @@ -96,5 +127,10 @@ like when it actually fires. - Dialect defaults to `databricks`; pass `--dialect` to `scripts/sql_diagram.py` directly for others. - CTEs resolve through to their base tables, but a query reading a **view** stops at the view name; the view's own definition is not expanded. +- **CTE scans use the same cylinder as base tables** — only the name tells them apart, and a CTE + that merely adds a window has a producing pipeline contributing nothing visible, so it reads as a + table. A `VALUES` CTE renders as two chained identical scans. +- SQL comments leak into node labels as truncated `/* …` fragments; strip them before diagramming + if the labels get noisy. - `--comments` is the only part that touches the network, and it uses the `dev` profile: the MCP service principal lacks `USE SCHEMA` on `system.billing`. diff --git a/.claude/skills/sql-diagram/example.md b/.claude/skills/sql-diagram/example.md index 8891cff..2e5157f 100644 --- a/.claude/skills/sql-diagram/example.md +++ b/.claude/skills/sql-diagram/example.md @@ -1,17 +1,19 @@ # Worked example — `job_spend_plan` -A committed `mode=plan` diagram of the per-job spend query from `scripts/project_costs.py`. The -three artifacts live in `reports/sql-diagram/` and are force-added past the gitignore precisely so -this example survives: +A committed `mode=plan` diagram of the per-job spend query from `scripts/project_costs.py`. All of +`reports/` is gitignored generated output, so the three artifacts live here in `examples/` instead — +a committed path — and that is where any future example belongs too: -- `job_spend_plan.sql` — the query as analysed, f-string placeholders already resolved -- `job_spend_plan.mmd` — the graph -- `job_spend_plan.svg` — the same graph, for prose that can't render Mermaid +- `examples/job_spend_plan.sql` — the query as analysed, f-string placeholders already resolved +- `examples/job_spend_plan.mmd` — the graph +- `examples/job_spend_plan.svg` — the same graph, for prose that can't render Mermaid Regenerate it with: ```bash -make sql-diagram sql=reports/sql-diagram/job_spend_plan.sql name=job_spend_plan comments=1 +make sql-diagram sql=.claude/skills/sql-diagram/examples/job_spend_plan.sql \ + name=job_spend_plan comments=1 +# then copy reports/sql-diagram/job_spend_plan.* back over examples/ to refresh this example ``` Because the committed `.sql` is the post-substitution query, that command reproduces the diagram diff --git a/reports/sql-diagram/job_spend_plan.mmd b/.claude/skills/sql-diagram/examples/job_spend_plan.mmd similarity index 100% rename from reports/sql-diagram/job_spend_plan.mmd rename to .claude/skills/sql-diagram/examples/job_spend_plan.mmd diff --git a/reports/sql-diagram/job_spend_plan.sql b/.claude/skills/sql-diagram/examples/job_spend_plan.sql similarity index 100% rename from reports/sql-diagram/job_spend_plan.sql rename to .claude/skills/sql-diagram/examples/job_spend_plan.sql diff --git a/reports/sql-diagram/job_spend_plan.svg b/.claude/skills/sql-diagram/examples/job_spend_plan.svg similarity index 100% rename from reports/sql-diagram/job_spend_plan.svg rename to .claude/skills/sql-diagram/examples/job_spend_plan.svg diff --git a/.gitignore b/.gitignore index a2cb45e..754b3f7 100644 --- a/.gitignore +++ b/.gitignore @@ -26,7 +26,6 @@ notes/ .pytest_cache/ dist/ build/ -reports/coverage/ src/template.egg-info/ *.pyc # uv.lock IS tracked — it pins transitive deps (notably databricks-sdk, which arrives @@ -44,10 +43,10 @@ resources/orders_dashboard_deploy.lvdash.json # Dev Kit 0.1.14 also writes Codex/Copilot agent configs into the project dir .agents/ .codex/ -# Generated spend reports — local artifacts containing account cost figures. -# A report that has been committed stays tracked (.gitignore only governs untracked files), so -# new dated reports are ignored by default; `git add -f` a specific one to keep it. -reports/cost/ -# Generated query diagrams — same rule: ignored by default, `git add -f` one to keep it as an -# example. They are derived artifacts, regenerable from the query at any time. -reports/sql-diagram/ +# reports/ is generated output, all of it: coverage, spend reports (which carry account cost +# figures), query diagrams. Nothing in here is committed — it is all regenerable from the code or +# the query that produced it. Don't `git add -f` an artifact to keep it as an example: put the +# example in the skill that explains it (.claude/skills/<skill>/examples/), which is a committed +# path. Note .gitignore only governs UNTRACKED files, so anything already tracked under here must +# be `git rm --cached`-ed before this rule takes effect on it. +reports/ diff --git a/README.md b/README.md index ff2050c..39b382a 100644 --- a/README.md +++ b/README.md @@ -58,8 +58,8 @@ This project template demonstrates how to: - utilize the [Databricks SDK for Python](https://docs.databricks.com/en/dev-tools/sdk-python.html) to manage catalogs, schemas, workspaces, and accounts. Refer to the `scripts` folder for examples. - utilize [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) to manage permissions and get data lineage. - enforce production guardrails out of the box — identity-locked CI deploys, a health-check task, wheel version pinning, per-task timeouts, schema-drift guards, queued runs, and on-call alerting. -- track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](reports/cost/2026-07-22.md). -- diagram any SQL query with [sqlglot](https://github.com/tobymao/sqlglot) — see an [example](reports/sql-diagram/job_spend_plan.svg). +- track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](.claude/skills/project-costs/examples/2026-07-22.md). +- diagram any SQL query with [sqlglot](https://github.com/tobymao/sqlglot) — see an [example](.claude/skills/sql-diagram/examples/job_spend_plan.svg). - utilize serverless job clusters on [Databricks Free Edition](https://docs.databricks.com/aws/en/getting-started/free-edition) to deploy your pipelines. diff --git a/specs/tooling.md b/specs/tooling.md index 639d58c..be82384 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -145,10 +145,11 @@ update them, and do keep them in sync with the code they wrap. - **sql-diagram** — wraps `scripts/sql_diagram.py` via `make sql-diagram`: query plan or column lineage as `.mmd` + `.svg`, plus how to read each mode. -The latter two each ship an `example.md` beside the `SKILL.md`, walking a committed artifact -(`reports/cost/2026-07-22.md`, `reports/sql-diagram/job_spend_plan.*`) to show what good output -looks like. Those artifacts are force-added past the `reports/` gitignore for exactly that reason — -if you regenerate them, re-add with `git add -f` or the example silently goes stale. +The latter two each ship an `example.md` beside the `SKILL.md`, walking a real artifact in the +skill's own `examples/` directory to show what good output looks like. Those artifacts live under +`.claude/skills/<skill>/examples/` rather than in `reports/`, because **all of `reports/` is +gitignored generated output** — that is where the tools write, and nothing there is committed. A new +example is a copy into `examples/`, never a `git add -f` out of `reports/`. One gotcha on the kit's skills. Some have a frontmatter `name:` that differs from their directory (`databricks` declares `databricks-core`; `analyze-mlflow-trace` declares `analyzing-mlflow-trace`) From 1875627f4565c6b4dfee5cd9b5a9e3dc35379643 Mon Sep 17 00:00:00 2001 From: Andre <andre.f.salvati@gmail.com> Date: Sat, 25 Jul 2026 15:46:33 -0300 Subject: [PATCH 4/9] docs: correct two stale statements in workflow.md and the changelog hook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit workflow.md said the schema-drift guard had a single exception (ops._health). There are two: raw.order_quarantine also writes with overwriteSchema=true, added in #52 and already documented in CLAUDE.md — the spec was not updated in the same commit, which is what its own "keep docs in sync" section exists to prevent. Also corrects the Co-Authored-By trailer, which named Opus 4.8. require-changelog-entry.sh printed guidance the spec had superseded: "at most 3 sentences" (dropped at #49 because it constrained nothing) and a header template built from the branch name. Both now match specs/workflow.md — a ~1000-character single paragraph and the PR-URL header — and the message links to the section so the two cannot drift silently again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> EOF --subject 'Commit the agent tooling: workflow hooks, three project skills, and a fully ignored reports/' --body-file /tmp/pr-body-c1_otrlp.md --- .claude/hooks/require-changelog-entry.sh | 9 ++++++--- specs/workflow.md | 7 ++++--- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/.claude/hooks/require-changelog-entry.sh b/.claude/hooks/require-changelog-entry.sh index aeddc91..ff72412 100755 --- a/.claude/hooks/require-changelog-entry.sh +++ b/.claude/hooks/require-changelog-entry.sh @@ -40,9 +40,12 @@ print(json.dumps({ "continue": False, "stopReason": ( f"Merge blocked: this branch ({branch}) adds no entry to specs/CHANGELOG.md.\n" - "Add one at the top before merging — append-only, never edit an existing entry.\n" - "Header: ## [#<PR>] · " + branch + " · YYYY-MM-DD · <title>\n" - "Body: at most 3 sentences. Replace the branch name with the PR URL after merge." + "Add one at the top before merging — append-only, never edit or reformat an existing entry.\n" + "Header (use the PR URL directly; the number is known once the PR is open):\n" + " ## [#NN](https://github.com/andre-salvati/databricks-template/pull/NN) · YYYY-MM-DD · <title>\n" + "Body: one unwrapped paragraph, around 1000 characters. Character count is the only limit —\n" + "do not hard-wrap, and do not pad a short entry to reach it.\n" + "Full rule: specs/workflow.md#changelog-discipline" ), })) PYEOF diff --git a/specs/workflow.md b/specs/workflow.md index 99e6f5e..3fb84b7 100644 --- a/specs/workflow.md +++ b/specs/workflow.md @@ -23,7 +23,7 @@ before starting any change. For what the code does, see [architecture.md](archit (`git branch --show-current`): if you're on `main` or on a stale/already-merged branch, run `git checkout main && git pull && git checkout -b <branch>` before editing. Starting from a diverged base causes conflicts and can silently regress work from a merged PR. -- Commit messages end with the `Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>` trailer. +- Commit messages end with the `Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>` trailer. - **Never commit generated / local-state files** (all gitignored): `resources/jobs.yml`, `resources/orders_dashboard_deploy.lvdash.json`, `.databricks-resources.json`. @@ -55,8 +55,9 @@ Operational rules: The **final check before merge** must anticipate whether the change can break tables in production and **raise an explicit alert in the PR**, classifying the change and declaring the remediation. This is not merely documentation: every medallion write uses `.option("overwriteSchema", "false")` -(the schema-drift guard — the only exception is `ops._health`), so **any schema drift hard-fails the -job at runtime by design**. Schema drift is a failure signal, not something to absorb silently. The +(the schema-drift guard — the two exceptions are `ops._health` and `raw.order_quarantine`, whose +`_errors`/`_warnings` structs are shaped by the DQX version rather than the data contract), so **any +schema drift hard-fails the job at runtime by design**. Schema drift is a failure signal, not something to absorb silently. The alert in the PR is the human anticipation of that failure. Changes that touch table schemas typically live in `src/template/commonSchemas.py` (the canonical From a90c7d130b00abf8829de89decc3180f28985434 Mon Sep 17 00:00:00 2001 From: Andre <andre.f.salvati@gmail.com> Date: Sat, 25 Jul 2026 15:46:33 -0300 Subject: [PATCH 5/9] feat: sql-diagram delivers a data simulation, and loses its example.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The plan diagram is authoritative on scans, joins and join predicates and silent on windows, CTE-level filters and projection expressions — which is where the defects that produce wrong numbers live. The skill now traces the query through a handful of hand-built rows after the chart and ships both as one self-contained HTML page. Two rules the trace depends on: never hand-trace (rewrite each source table as a literal VALUES CTE, run it, paste the real result — NULL propagation, COUNT(DISTINCT) and window ordering are all easy to get wrong on paper), and embed the rendered .svg as a data URI rather than re-rendering the .mmd, which also keeps the SVG's own generic class names out of the page cascade. examples/job_spend_plan.html is the worked instance: five usage rows through the repo's own per-job spend query, showing a date-scoped price join picking one price per row — with the verified counterfactual that dropping the range predicates doubles quantity and inflates spend from $10.00 to $19.50 — a LEFT join keeping an unpriced SKU alive, and a filter that discards unattributable usage, which is why the attributed total never equals the Databricks total. example.md is deleted. It asserted in prose what the page now demonstrates with data; its unique content (the regenerate step, the CTE two-node shape) moved into SKILL.md. project-costs keeps its example.md, because a page of cost tables cannot explain why its analysis is written the way it is — the rule is now stated in specs/tooling.md: add a commentary file only when the artifact cannot speak for itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --- .claude/skills/sql-diagram/SKILL.md | 55 ++- .claude/skills/sql-diagram/example.md | 82 ----- .../sql-diagram/examples/job_spend_plan.html | 327 ++++++++++++++++++ specs/tooling.md | 10 +- 4 files changed, 385 insertions(+), 89 deletions(-) delete mode 100644 .claude/skills/sql-diagram/example.md create mode 100644 .claude/skills/sql-diagram/examples/job_spend_plan.html diff --git a/.claude/skills/sql-diagram/SKILL.md b/.claude/skills/sql-diagram/SKILL.md index 2a91637..babbabd 100644 --- a/.claude/skills/sql-diagram/SKILL.md +++ b/.claude/skills/sql-diagram/SKILL.md @@ -1,6 +1,6 @@ --- name: sql-diagram -description: Diagram a SQL query and explain what it shows — either its execution steps (mode=plan) or its column lineage (mode=lineage). Use when asked to visualize, diagram, explain or review what a query does, how it joins its tables, or where an output column comes from. Wraps `make sql-diagram`, which emits .mmd and .svg into reports/sql-diagram/. See example.md for a worked reading of a committed diagram. +description: Diagram a SQL query and explain what it shows — either its execution steps (mode=plan) or its column lineage (mode=lineage) — then trace it through small data so the defects the picture cannot show become visible. Use when asked to visualize, diagram, explain or review what a query does, how it joins its tables, or where an output column comes from. Wraps `make sql-diagram`, which emits .mmd and .svg into reports/sql-diagram/, and delivers a self-contained HTML page. See examples/job_spend_plan.html for a worked instance. --- # Diagramming a SQL query @@ -18,10 +18,14 @@ Diagram a SQL query and explain what it shows — either its execution steps or 3. Run `make sql-diagram sql=<path> name=<basename> comments=1` via Bash. It writes three files to `reports/sql-diagram/`: `<basename>.sql` (the query as analysed), `.mmd` and `.svg`. All of `reports/` is gitignored generated output — never `git add -f` out of it. To keep a diagram as a - committed example, copy the trio into `.claude/skills/sql-diagram/examples/`. Pass `--stdout` to + committed example, copy the trio into `.claude/skills/sql-diagram/examples/` and re-run against + the copied `.sql`; because that file is the *post-substitution* query, the run reproduces the + diagram exactly, which is why the `.sql` is kept beside the picture. Pass `--stdout` to `scripts/sql_diagram.py` for a throwaway look with no files written. 4. Read the `.mmd`, show it in a ```mermaid fence, and explain it (see below). The `.svg` is the same graph for linking from prose where no Mermaid renderer is available. +5. **Simulate the query on small data** (see [Simulation](#simulation)) and deliver the diagram and + the trace together as one HTML page in `reports/sql-diagram/<basename>.html`. **Explaining a query and reviewing one are different jobs.** For an explanation the diagram is enough. For a *review*, read the `.sql` alongside it and treat the graph as an index into the text: @@ -41,7 +45,8 @@ to question. Nodes are the query's steps, bottom-up: `SCAN` per table, one `JOIN n` per individual join, then `WHERE`, `AGGREGATE`, `SORT`, `OUTPUT`. A CTE appears as its own sub-pipeline feeding the `SCAN` -that reads it. +that reads it — a two-node `SCAN <base table> → AGGREGATE → SCAN <cte>` chain is one CTE being built +and then read back, not the same table scanned twice. - **Each join is numbered in the order the query writes it** and carries its side and keys. `sqlglot` models a multi-table join as one n-ary step; the script splits it back apart. An @@ -71,6 +76,46 @@ that reads it. `sqlglot` lifted out. Read the intent off the original SQL rather than repeating the placeholder — `COUNT(\`_a_0\`)` and `COUNT(DISTINCT …)` are indistinguishable in the picture. +## Simulation + +The diagram shows structure; it cannot show what the query *does to rows*. After the chart, trace the +query on a handful of hand-built rows — that trace is where defects become visible, because the graph +is silent on exactly the stages that produce wrong numbers. + +1. **Build the smallest input that can expose something.** A few rows per source table, not a + realistic extract. Choose the values deliberately: two dates, a renamed dimension row, a fact whose + key is missing from a dimension it inner-joins to, ranges that overlap between two id columns you + suspect are being confused. Data that can only produce a clean result proves nothing. +2. **Show the initial state first** — every source table, in full, before anything runs. +3. **Then one block per stage**, in execution order, each with the operation and the *whole* output + at that point. Row counts should be small enough to print entirely; never elide with "…". +4. **Mark what changed at each stage** — a row dropped, a column newly populated, a rank assigned. + Strike dropped rows rather than deleting them silently; that disappearance is usually the finding. +5. **Never hand-trace.** Rewrite the query with each source table replaced by a literal `VALUES` CTE + and run it through `mcp__databricks__execute_sql`, then paste the real result. A hand-trace that + quietly disagrees with the engine is worse than no trace, and `NULL` propagation, `COUNT(DISTINCT)` + and window ordering are all easy to get wrong on paper. Run intermediate CTEs the same way. +6. **State that the trace was executed**, and that the sample data is illustrative while the defects + are real. + +The output is one self-contained HTML page: the plan diagram, then the initial tables, then the +per-stage trace, then the final result set. + +**Embed the `.svg`, not the `.mmd`.** The script already rendered the graph; re-rendering it from +Mermaid in the page only adds a way for it to break. Inline the SVG as a base64 `data:` URI in an +`<img>` — the artifact CSP blocks external refs, and an `<img>` also walls the SVG's own `<style>` +block off from the page cascade (it ships generic class names like `.output`, `.filter` and `.step` +that will otherwise collide with yours). Give it a white plate in both themes; it is a printed +figure, not a UI surface. + +`.claude/skills/sql-diagram/examples/job_spend_plan.html` is the worked instance: five usage rows +through the repo's own per-job spend query. It shows a date-scoped price join picking exactly one +price per row — with the verified counterfactual that removing the range predicates doubles quantity +and inflates spend from $10.00 to $19.50 — a `LEFT` join keeping an unpriced SKU alive, and a filter +that deliberately discards unattributable usage, which is why the attributed total never equals the +Databricks total. Only two of the five are visible in the diagram; the one that silently doubles the +bill needs the data. + ## Reading `mode=lineage` - **Subgraphs are source tables**, one node per source column actually read. A column the query @@ -112,8 +157,8 @@ has no comment the space is blank, and that absence is itself worth reporting. joins a dimension that varies *within* `A × B`, either that dimension is fabricated or the join fans out and inflates every measure. The `AGGREGATE` node's `GROUP BY` line is where to check. -See `example.md` for a committed diagram read end to end, including what each of these points looks -like when it actually fires. +`examples/job_spend_plan.html` is a committed instance of all of this — the diagram read end to end, +then the same query traced through data so each point is demonstrated rather than asserted. ## Limits worth stating rather than hiding diff --git a/.claude/skills/sql-diagram/example.md b/.claude/skills/sql-diagram/example.md deleted file mode 100644 index 2e5157f..0000000 --- a/.claude/skills/sql-diagram/example.md +++ /dev/null @@ -1,82 +0,0 @@ -# Worked example — `job_spend_plan` - -A committed `mode=plan` diagram of the per-job spend query from `scripts/project_costs.py`. All of -`reports/` is gitignored generated output, so the three artifacts live here in `examples/` instead — -a committed path — and that is where any future example belongs too: - -- `examples/job_spend_plan.sql` — the query as analysed, f-string placeholders already resolved -- `examples/job_spend_plan.mmd` — the graph -- `examples/job_spend_plan.svg` — the same graph, for prose that can't render Mermaid - -Regenerate it with: - -```bash -make sql-diagram sql=.claude/skills/sql-diagram/examples/job_spend_plan.sql \ - name=job_spend_plan comments=1 -# then copy reports/sql-diagram/job_spend_plan.* back over examples/ to refresh this example -``` - -Because the committed `.sql` is the post-substitution query, that command reproduces the diagram -exactly — which is the whole reason the `.sql` is committed alongside the picture. - -## The graph - -```mermaid -flowchart LR - n0[("<b>SCAN system.lakeflow.pipelines</b>")] - n1["<b>AGGREGATE</b><br/>GROUP BY pipeline_id<br/>MAX_BY(name, change_time) AS name"] - n2[("<b>SCAN pipe_names</b>")] - n3[("<b>SCAN system.billing.list_prices</b>")] - n4[("<b>SCAN system.billing.usage</b>")] - n5{{"<b>JOIN 1 · LEFT · p</b><br/>u.sku_name = p.sku_name<br/>and u.usage_end_time >= p.price_start_time<br/>and (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)"}} - n6{{"<b>JOIN 2 · LEFT · n</b><br/>u.usage_metadata.dlt_pipeline_id = n.pipeline_id"}} - n7["<b>WHERE</b><br/>u.usage_date >= CURRENT_DATE - INTERVAL '30' DAYS<br/>…"] - n8["<b>AGGREGATE</b><br/>GROUP BY entity, kind, u.usage_unit<br/>SUM(u.usage_quantity) AS quantity<br/>SUM(`_a_0`) AS usd<br/>COUNT(`_a_1`) AS active_days"] - n9["<b>SORT</b><br/>usd DESC"] - n10(["<b>OUTPUT</b><br/>entity<br/>kind<br/>quantity<br/>usage_unit<br/>usd<br/>active_days"]) - n0 --> n1 - n1 --> n2 - n4 --> n5 - n3 --> n5 - n5 --> n6 - n2 --> n6 - n6 --> n7 - n7 --> n8 - n8 --> n9 - n9 --> n10 -``` - -## How to read it - -**Shape first.** Three source tables, two joins, one CTE, grouped to one row per entity × kind × -unit and sorted by dollars. `system.billing.usage` is the fact; the other two are lookups. - -**The CTE is its own sub-pipeline.** `n0 → n1` is `pipe_names` being built (dedupe -`system.lakeflow.pipelines` to one current name per `pipeline_id` via `MAX_BY`), and `n2` is the -`SCAN` that reads the finished CTE back. That two-node shape is what a CTE always looks like here — -it isn't a duplicate scan of the same table. - -**Both joins are `LEFT`, and that is load-bearing.** Usage rows survive even when no price row -matches or the pipeline has no name. An `INNER JOIN` here would silently drop unpriced SKUs and -under-report spend — exactly the kind of thing to say out loud, because it changes what a missing -row in the output means. - -**JOIN 1 carries three predicates, not one.** The equality on `sku_name` plus two range predicates -on `price_start_time` / `price_end_time`. `list_prices` is a slowly-changing dimension with one row -per price period, so those ranges are what pick a single price rather than fanning every usage row -out across every historical price. This is the under-constrained-join failure mode the skill warns -about — here it is correctly constrained, and worth naming as such. - -**`usage_metadata` is the hub.** Three separate expressions read it (`job_name`, `dlt_pipeline_id`, -`job_id`), so it is where a schema change would hurt most. Note that lineage mode would collapse all -three to `usage_metadata`, the struct root — this is the case the skill's "struct columns collapse" -limit describes, and the reason `plan` is the better mode for this query. - -**`_a_0` and `_a_1` are synthetic.** `sqlglot` lifted the `usage_quantity * pricing…` product and -the `DISTINCT usage_date` out of their aggregates. Read the intent from the `.sql` — `usd` is a -priced sum, `active_days` a distinct-day count — rather than repeating the placeholder names at the -user. - -**The grey comment lines** under the two `system.billing` scans came from `comments=1` reading Unity -Catalog. They are fetched, never authored. `pipe_names` has none because a CTE isn't a catalog -object. diff --git a/.claude/skills/sql-diagram/examples/job_spend_plan.html b/.claude/skills/sql-diagram/examples/job_spend_plan.html new file mode 100644 index 0000000..c3b36a6 --- /dev/null +++ b/.claude/skills/sql-diagram/examples/job_spend_plan.html @@ -0,0 +1,327 @@ +<title>job_spend_plan.sql — plan and data walkthrough + + + +
+ +

Query plan · mode=plan

+

job_spend_plan.sql

+

+ Thirty days of Databricks spend, attributed to the job or pipeline that incurred it — the query + behind the by Job / Pipeline table in scripts/project_costs.py. Below: the + execution plan, then the same query traced through five usage rows, one stage at a time. The trace + was executed on Databricks SQL with each source table replaced by a literal VALUES CTE; + every result on this page is that run's real output. +

+ +

1 · Execution plan

+
+ Execution plan: three source tables, two LEFT joins, a filter, an aggregate and a sort. +
+

Generated by scripts/sql_diagram.py --mode plan. Note what the graph does not contain: no node for either CTE-level filter, and COUNT(DISTINCT …) renders as COUNT(_a_1).

+ +

2 · Initial state

+
+ +
+
system.lakeflow.pipelines · 2 rows
+
+ + + + + +
pipeline_idnamechange_time
pl1job1_sdp2026-06-01
pl1job1_sdp_prod2026-07-01
+
+ +
+
system.billing.list_prices · 3 rows
+
+ + + + + + +
sku_name…effective_list.defaultprice_start_timeprice_end_time
JOBS_SERVERLESS0.302026-06-012026-07-10
JOBS_SERVERLESS0.352026-07-10NULL
SQL_SERVERLESS0.702026-06-01NULL
+
+ +
+
system.billing.usage · 5 rows  ·  job_id / job_name / dlt_pipeline_id are fields of the usage_metadata struct
+
+ + + + + + + + +
usage_datesku_nameqtyunit…job_id…job_name…dlt_pipeline_id
2026-07-08JOBS_SERVERLESS10.0DBUj1job1_prodNULL
2026-07-12JOBS_SERVERLESS20.0DBUj1job1_prodNULL
2026-07-12JOBS_SERVERLESS8.0DBUNULLNULLpl1
2026-07-14STORAGE5.0DSUj1job1_prodNULL
2026-07-15SQL_SERVERLESS4.0DBUNULLNULLNULL
+
+ +
+

Three deliberate shapes: a price that changes mid-window, a SKU (STORAGE) with no price row at all, and a usage row carrying neither a job nor a pipeline.

+ +

3 · Stage by stage

+ +
+
01
Aggregate
1 row
+
+
pipe_names — GROUP BY pipeline_id, MAX_BY(name, change_time)
+
+ + +
pipeline_idname
pl1job1_sdp_prod
+

system.lakeflow.pipelines is an SCD2 table — one row per change. MAX_BY + collapses the two rows to the name at the latest change_time, so the pipeline resolves to its + current name and the old job1_sdp never reaches the report.

+
+
+ +
+
02
Join 1 · Left
5 rows
+
+
LEFT JOIN list_prices p ON u.sku_name = p.sku_name + AND u.usage_end_time >= p.price_start_time + AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
+
+ + + + + + + + +
usage_datesku_nameqtyprice pickedrow usd
2026-07-08JOBS_SERVERLESS10.00.303.00
2026-07-12JOBS_SERVERLESS20.00.357.00
2026-07-12JOBS_SERVERLESS8.00.352.80
2026-07-14STORAGE5.0no price rowNULL
2026-07-15SQL_SERVERLESS4.00.702.80
+
+ Still 5 rows — the join did not fan out + JOBS_SERVERLESS has two price rows, but the two range predicates select exactly one per + usage row: the 07-08 row prices at the old 0.30, the 07-12 rows at the new 0.35. And because the + join is LEFT, the unpriced STORAGE row survives with a NULL price instead of + vanishing — an INNER join here would silently drop 5 DSU of storage from the report. +
+
+ Counterfactual · drop the two range predicates + Joining on sku_name alone, each JOBS_SERVERLESS row matches both + price rows. Verified by re-running: job1_prod goes from 30 → 60 DBU and + $10.00 → $19.50; job1_sdp_prod from 8 → 16 DBU. Quantity exactly doubles, + spend nearly doubles, and nothing errors. +
+
+
+ +
+
03
Join 2 · Left
5 rows
+
+
LEFT JOIN pipe_names n ON n.pipeline_id = u.usage_metadata.dlt_pipeline_id
+
+ + + + + + + + +
usage_date…job_namen.nameentity = COALESCE(job_name, n.name, '(unnamed)')
2026-07-08job1_prodNULLjob1_prod
2026-07-12job1_prodNULLjob1_prod
2026-07-12NULLjob1_sdp_prodjob1_sdp_prod
2026-07-14job1_prodNULLjob1_prod
2026-07-15NULLNULL(unnamed)
+

Jobs carry their name in the usage record; pipelines do not, which is the only reason + this join exists. The COALESCE ladder tries the job name, then the pipeline name, then a + literal fallback — so the row with neither is not dropped here, it is merely unnamed.

+
+
+ +
+
04
Filter
4 rows
+
+
WHERE u.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS + AND COALESCE(u.usage_metadata.job_id, u.usage_metadata.dlt_pipeline_id) IS NOT NULL
+
+ + + + + + + + +
usage_datesku_nameentityrow usd
2026-07-08JOBS_SERVERLESSjob1_prod3.00
2026-07-12JOBS_SERVERLESSjob1_prod7.00
2026-07-12JOBS_SERVERLESSjob1_sdp_prod2.80
2026-07-14STORAGEjob1_prodNULL
2026-07-15SQL_SERVERLESS(unnamed)2.80
+

The SQL-warehouse row has neither a job_id nor a + dlt_pipeline_id, so it is discarded — along with its $2.80. This is why the + attributed total is always less than the Databricks total, and why the cost report states the + attributed share rather than implying the breakdown is exhaustive. The gap is interactive and SQL + compute, by construction, not a defect.

+
+
+ +
+
05
Aggregate
3 rows
+
+
GROUP BY entity, kind, u.usage_unit +SUM(usage_quantity) · SUM(usage_quantity * price) · COUNT(DISTINCT usage_date)
+
+ + + + + + +
entitykindquantityunitusdactive_days
job1_prodjob30.0DBU10.0002
job1_sdp_prodpipeline8.0DBU2.8001
job1_prodjob5.0DSUNULL1
+

job1_prod appears twice — once per unit, because + usage_unit is in the GROUP BY. DBU and DSU bill at different rates and are not + comparable, so the report never totals the Quantity column. active_days = 2 comes from + COUNT(DISTINCT usage_date) over the 07-08 and 07-12 rows: it counts days, not + records, which is what makes per-active-day comparison between prod and staging possible.

+
+
+ +
+
06
Sort
3 rows
+
+
ORDER BY usd DESC
+

Descending by dollars, so the biggest spender leads. The NULL-priced storage row sorts + last — Spark places NULLs last on a DESC sort — which is the behaviour you want: an + unpriced SKU should be visible at the bottom of the table, not at the top pretending to be the + largest.

+
+
+ +

4 · Result

+
+
output · 3 rows
+
+ + + + + + +
entitykindquantityusage_unitusdactive_days
job1_prodjob30.0DBU10.0002
job1_sdp_prodpipeline8.0DBU2.8001
job1_prodjob5.0DSUNULL1
+
+

Verified output — the actual result set, not a hand-trace.

+ +

5 · What five rows proved

+
    +
  1. The price join is date-scoped, and that is load-bearing. Two price rows, one picked per + usage row. Remove the range predicates and quantity doubles to 60 DBU and spend to $19.50 — verified, + not asserted. The diagram shows those predicates as and … under the join keys; they read + as noise and are the difference between a correct number and a doubled one.
    2. +
  2. Both joins must stay LEFT. The unpriced STORAGE row survives with a NULL price. + Under INNER it would disappear silently and the report would under-count.
    3. +
  3. The attributed total is deliberately incomplete. One row of SQL-warehouse usage carries no + job or pipeline id and is filtered out. Any reconciliation against the Databricks total must expect + that gap rather than treat it as missing data.
    4. +
  4. Quantity never sums across units. job1_prod emits two rows, DBU and DSU, + because the unit is part of the grain.
    5. +
  5. A renamed pipeline reports under its current name. MAX_BY(name, change_time) + resolves pl1 to job1_sdp_prod; the superseded job1_sdp never + appears.
    6. +
+

Only the second and third are visible in the plan diagram, and only if you already know to + look at the join side and the filter. The first — the one that silently doubles the bill — needs the data.

+ +
+ job_spend_plan.sql · plan by scripts/sql_diagram.py --mode plan · SVG inlined as a data URI
+ trace executed on Databricks SQL with source tables replaced by literal VALUES
+ sample data is illustrative; the query is the real one from scripts/project_costs.py +
+ +
diff --git a/specs/tooling.md b/specs/tooling.md index be82384..cf188bc 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -145,12 +145,18 @@ update them, and do keep them in sync with the code they wrap. - **sql-diagram** — wraps `scripts/sql_diagram.py` via `make sql-diagram`: query plan or column lineage as `.mmd` + `.svg`, plus how to read each mode. -The latter two each ship an `example.md` beside the `SKILL.md`, walking a real artifact in the -skill's own `examples/` directory to show what good output looks like. Those artifacts live under +The latter two each ship a worked example showing what good output looks like, kept under `.claude/skills//examples/` rather than in `reports/`, because **all of `reports/` is gitignored generated output** — that is where the tools write, and nothing there is committed. A new example is a copy into `examples/`, never a `git add -f` out of `reports/`. +The two take different forms, and the difference is the rule worth copying. `sql-diagram` ships +`job_spend_plan.html`, which *is* the deliverable — diagram plus a data trace — so it needs no prose +companion; it had one, and the file was deleted once the page demonstrated with data what the prose +had asserted. `project-costs` ships `example.md` beside its report, because a page of cost tables +does not explain why its analysis is written the way it is. **Add a commentary file only when the +artifact cannot speak for itself**; two files narrating one artifact will drift. + One gotcha on the kit's skills. Some have a frontmatter `name:` that differs from their directory (`databricks` declares `databricks-core`; `analyze-mlflow-trace` declares `analyzing-mlflow-trace`) — **invoke by directory name**, which is what the session's skill list shows; the frontmatter name From f8421b646bf3a945a0dc7b33eb10e41b47162cff Mon Sep 17 00:00:00 2001 From: Andre Date: Thu, 6 Aug 2026 09:41:49 -0300 Subject: [PATCH 6/9] docs: commit a worked data-divergence investigation as the skill's example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The data-divergence skill was the only one of the three shipping no example. This adds one: a full investigation of prod's batch vs SDP silver tables, re-run against live prod rather than transcribed, and structured the way the skill prescribes — layer counts, key-level diff, row_commit_version to pin the blast radius, a Delta-history timeline, proven-vs-inferred, costed fixes, and an appendix of the queries. It is kept for its findings, not its format. The reported complaint was one shifted date column; three further divergences turned up, including a live gap in prod.report.order_agg (2026-07-24 missing after a failed run, never backfilled because the incremental MERGE is scoped to seed_date). Also folds in the README rephrasing of the skill bullets. Co-Authored-By: Claude Opus 5 --- .claude/skills/data-divergence/SKILL.md | 18 +- .../examples/2026-08-06-prod-batch-vs-sdp.md | 363 ++++++++++++++++++ README.md | 5 +- specs/tooling.md | 16 +- 4 files changed, 394 insertions(+), 8 deletions(-) create mode 100644 .claude/skills/data-divergence/examples/2026-08-06-prod-batch-vs-sdp.md diff --git a/.claude/skills/data-divergence/SKILL.md b/.claude/skills/data-divergence/SKILL.md index 2379e8f..6c5acd6 100644 --- a/.claude/skills/data-divergence/SKILL.md +++ b/.claude/skills/data-divergence/SKILL.md @@ -1,6 +1,6 @@ --- name: data-divergence -description: Investigate why two datasets that should agree don't — two pipelines writing the same logical table, a rollup vs the detail it aggregates, a dashboard vs its source, one environment vs another. Use when row counts, totals, or date ranges disagree and the question is what happened rather than just what differs. Covers localizing the first layer that diverges, diffing by grain and by key, reconciling across an aggregation boundary, reading Delta history and row_commit_version, why append-only tables diverge permanently, and what a fix actually costs. +description: Investigate why two datasets that should agree don't — two pipelines writing the same logical table, a rollup vs the detail it aggregates, a dashboard vs its source, one environment vs another. Use when row counts, totals, or date ranges disagree and the question is what happened rather than just what differs. Covers localizing the first layer that diverges, diffing by grain and by key, reconciling across an aggregation boundary, reading Delta history and row_commit_version, why append-only tables diverge permanently, and what a fix actually costs. See examples/ for a worked investigation. --- # Investigating a data divergence @@ -178,3 +178,19 @@ broken column **and** discards every frozen value it was carrying — you may be in one column for a divergence in another. Say which columns move, in both directions, before recommending it. Check the run state of the other path first too: resetting a table whose writer is currently failing leaves it empty. + +## The worked example + +`examples/2026-08-06-prod-batch-vs-sdp.md` is a full investigation of this repo's own `prod` catalog, +written to the shape above: layer counts, a key-level diff, `row_commit_version` to pin the blast +radius, a Delta-history timeline, then proven-vs-inferred and costed fix options. + +It is the only place this skill names real tables, and it is worth reading for what it found rather +than for the procedure. The reported complaint was one shifted date column; three further divergences +turned up, including a **live gap in a production gold table** that nobody had reported — a failed +daily run whose date-scoped incremental MERGE meant no later run ever backfilled it, while the +full-overwrite layer above it self-healed and hid the failure. Two of the four also came from a +single append-only commit freezing a column nobody intended to freeze. + +Structure a report the same way, and keep the appendix of queries: the next investigation starts by +editing them rather than by rewriting them. diff --git a/.claude/skills/data-divergence/examples/2026-08-06-prod-batch-vs-sdp.md b/.claude/skills/data-divergence/examples/2026-08-06-prod-batch-vs-sdp.md new file mode 100644 index 0000000..d2e59ed --- /dev/null +++ b/.claude/skills/data-divergence/examples/2026-08-06-prod-batch-vs-sdp.md @@ -0,0 +1,363 @@ +# Data divergence — `prod` batch vs SDP medallion paths + +**Date:** 2026-08-06 +**Investigator:** Claude Code, via the `data-divergence` skill +**Sides:** `prod.curated.order_enriched` (batch, `job1`) vs `prod.curated.order_enriched_sdp` (declarative, `job1_sdp`) +**Grain:** one row per `(order_id, item_seq)` +**Identity:** all queries run through the `databricks` MCP server, which authenticates as `template-sp` — the same +service principal `prod` runs as. Reads only; nothing in this investigation wrote to `prod`. + +--- + +## Verdict + +The two silver tables hold the same orders but disagree on **two columns** and **two blocks of rows**. Four +distinct divergences, three of them still live: + +| # | Divergence | Scope | Status | Cause | +|---|---|---|---|---| +| 1 | `order_date` shifted **13 days earlier** in SDP | 6,000,000 rows | **Live** | Initial seed anchored on run date; SDP froze the pre-reseed values | +| 2 | `country` differs | 5,364,000 rows / 447 of 500 customers | **Live** | Same mechanism — an unintended freeze | +| 3 | 5,000 orders of 2026-07-24 missing from **batch** | 15,000 rows | **Live** | Failed run + date-scoped incremental MERGE | +| 4 | 5,000 orders of 2026-06-24 **tripled** in SDP | 15,000 rows | Live, cosmetic | Source double-append; DQX caught it on the batch side only | + +Divergences 1 and 2 share one root cause and one moment. Divergence 3 is unrelated and is the residue of an +already-fixed outage. Divergence 4 is a data-quality asymmetry that is arguably working as designed. + +**The transform is not at fault.** `generate_orders.py:25` and `job1_sdp/transforms.py:49` derive the date with +byte-identical expressions: + +```python +df_order["date"].cast("date").alias("order_date") +``` + +Every other column reconciles exactly: `product_name`, `order_total`, `item_total`, `item_quantity` all show +**zero** mismatches across 6.2M joined rows. This is not corruption, a broken join, or a bad filter. It is two +tables that froze the same upstream at two different moments. + +--- + +## 1. Counting down both paths + +``` +| layer | rows | distinct keys | min date | max date | +|-----------------------------|-----------|---------------|------------|------------| +| external_source.order | 2,225,000 | 2,220,000 | 2025-06-27 | 2026-08-06 | +| raw.order (batch) | 2,215,000 | 2,215,000 | 2025-06-27 | 2026-08-06 | +| raw.order_sdp (SDP) | 2,225,000 | 2,220,000 | 2025-06-27 | 2026-08-06 | +| curated.order_enriched | 6,215,000 | 2,215,000 | 2025-06-27 | 2026-08-06 | +| curated.order_enriched_sdp | 6,230,000 | 2,220,000 | 2025-06-14 | 2026-08-06 | +| report.order_agg | 202,560 | | 2025-06-27 | 2026-08-06 | +| report.order_agg_sdp | 203,500 | | 2025-06-14 | 2026-08-06 | +``` + +Read this top to bottom and the shape of the problem is already visible: + +- **Bronze agrees on dates.** Both `raw.order` and `raw.order_sdp` start at 2025-06-27. The 13-day gap appears + for the first time at **silver**, and only on the SDP side. Everything upstream is exonerated by this one query. +- **`raw.order` is 10,000 rows lighter** than its source and has no duplicate ids, while `raw.order_sdp` copies + the source verbatim. That is DQX: the batch path quarantines, the SDP path does not. → divergence 4. +- **Gold faithfully propagates.** `report.order_agg_sdp` is a materialized view; it recomputed correctly *from + wrong inputs* and inherited the shifted window. A correct aggregation over frozen bad data is still bad data. + +The min/max pattern is the classic **"both edges shifted, middle identical"** signature: nothing is missing, +a derived column moved. + +--- + +## 2. Diffing by key + +Joining the two silvers on `(order_id, item_seq)` and histogramming the date delta: + +``` +| delta_days | n | name_diff | total_diff | otot_diff | country_diff | qty_diff | +|------------|-----------|-----------|------------|-----------|--------------|----------| +| 13 | 6,000,000 | 0 | 0 | 0 | 5,364,000 | 0 | +| 0 | 225,000 | 0 | 0 | 0 | 0 | 0 | +``` + +One bucket holds **every** shifted row, at exactly 13 days. A single-bucket histogram means a **formula**, not +corruption — a constant offset applied uniformly. And 6,000,000 is not an arbitrary number: it is precisely the +initial backfill (2,000,000 orders × 3 items). The 225,000 rows that agree are everything appended since. + +The `country_diff` column was the surprise. It was not part of the original complaint, and it rides on exactly +the same 6,000,000 rows. + +--- + +## 3. Pinning the blast radius with `row_commit_version` + +Time travel is useless here — the events are 43 days old and `delta.deletedFileRetentionDuration` is 168 hours. +`_metadata.row_commit_version` is not retention-bounded, and it attributes live rows to the commit that wrote them: + +``` +| commit | n | min date | max date | +|--------|-----------|------------|------------| +| 3 | 6,000,000 | 2025-06-14 | 2026-06-11 | ← every bad row, one commit +| 5 | 5,000 | 2026-06-24 | 2026-06-24 | +| 6 | 10,000 | 2026-06-24 | 2026-06-24 | +| 7 | 5,000 | 2026-06-25 | 2026-06-25 | +| ... | 5,000 | one per day, correct | +| 51 | 5,000 | 2026-08-06 | 2026-08-06 | +``` + +All 6,000,000 shifted rows land in **commit 3**, and **commit 5 onward is already correct**. That converts the +theory into a fact: this was a single bad write, not an ongoing drift. Nothing since 2026-06-24 12:26 has been +wrong, and nothing will retroactively fix commit 3. + +--- + +## 4. The timeline + +Delta history across four objects, all on **2026-06-24**: + +``` +12:09:32 external_source.order v0 CREATE TABLE AS SELECT 0 rows ← empty bootstrap +12:09:51 external_source.customer v1 CREATE OR REPLACE TABLE AS SELECT 500 ← countries reset to seed banding +12:09:58 external_source.order v3 CREATE OR REPLACE TABLE AS SELECT 2,000,000 ← _seed_initial, anchored 2026-06-24 +12:12:00 curated.order_enriched v0 CREATE OR REPLACE TABLE AS SELECT 6,000,000 ← batch first_run branch +12:14:51 curated.order_enriched_sdp v0 CREATE TABLE ← SDP streaming table created +12:15:22 curated.order_enriched_sdp v3 STREAMING UPDATE 6,000,000 ← the bad commit +12:19:39 external_source.order v5 WRITE / Append 5,000 ← 2026-06-24 incremental +12:21:45 curated.order_enriched v4 MERGE 5,000 +12:26:23 curated.order_enriched_sdp v5 STREAMING UPDATE 5,000 ← correct from here on +15:55:05 external_source.order v7 WRITE / Append 5,000 ← same 5,000 ids AGAIN +15:56:45 curated.order_enriched_sdp v6 STREAMING UPDATE 10,000 +15:57:24 curated.order_enriched v8 MERGE 0 ← DQX quarantined them +``` + +`external_source.order` v0 is 2026-06-24 12:09:32. **That is when the table was created, not when the data +began** — prod was dropped and rebuilt from scratch that morning, and the previous incarnation's history is +gone with it. The reseed re-anchored 363 days of synthetic history to a new `seed_date`. + +Batch silver was written at **12:12:00**, three minutes *before* SDP silver at **12:15:22** — yet batch got +2025-06-27 and SDP got 2025-06-14. The two sides read the same source table minutes apart and disagree by +13 days, and 2026-06-24 − 13 days = **2026-06-11**, the max date in SDP commit 3. The SDP path read the +**previous incarnation's** data. + +### What is proven and what is inference + +**Proven by data:** the reseed at 12:09:58; the two silver writes and their contents; that all bad rows sit in +one commit; that the prior window was anchored on 2026-06-11; that the transform is identical; that bronze is +now consistent while SDP silver is not. + +**Inference:** that the SDP pipeline's first update consumed a **stale materialization** of `raw.order_sdp` +dating from before the 12:09:58 reseed. This cannot now be confirmed. `raw.order_sdp` is a materialized view, +and `DESCRIBE HISTORY` refuses it: + +``` +[EXPECT_TABLE_NOT_VIEW.NO_ALTERNATIVE] 'DESCRIBE HISTORY' expects a table but +`prod`.`raw`.`order_sdp` is a view. +``` + +That refusal is itself informative — it tells us the object whose intermediate state we need is precisely the +one whose history we cannot read. The pipeline event log would settle it, but the repo configures no permanent +event log table, so those events aged out with the default retention. **The mechanism is the best-fitting +explanation for the evidence, not a proven fact.** The 13-day arithmetic and the single-commit blast radius are +facts regardless of which mechanism delivered the stale rows. + +--- + +## 5. Root cause: a date that is not a fact + +`src/template/job1/seed_sources.py:135`, in `_seed_initial`: + +```python +F.date_sub(F.lit(seed_date), (F.col("id") % 363).cast(IntegerType())).cast("string").alias("date"), +``` + +The entire 2M-order backfill spans "the 363 days before `seed_date`". **`seed_date` is when the initial load +ran**, so every historical date is a function of when someone last reseeded — not a stable fact about an order. +Reseed on a different day and a year of history silently slides. + +The incremental path does not have this defect. `_build_incremental_orders` derives its day offset from a fixed +constant, `_EPOCH = date(2024, 1, 1)` (line 11, used at line 196), so reruns of the same date are idempotent. +The constant already exists; `_seed_initial` simply does not use it. That asymmetry is the whole bug. + +### Why only one side could recover + +- **Batch silver** rebuilds through the `first_run` branch (`generate_orders.py:49`) whenever the table is empty — + a full `overwrite`. It re-derived everything from the post-reseed source and now matches. +- **SDP silver** is a `@dp.table` **streaming table**. It appends each row once and never revisits it. The freeze + that exists deliberately to protect `product_name` from later renames also froze `order_date` and `country`. + +**"One side fixed itself" is a clue, not a reassurance.** Batch is *newer*, not inherently *righter* — it agrees +with the current source only because it was rebuilt after the reseed. + +### The unintended freeze (divergence 2) + +An append-only table freezes **every column it appends**, not only the one the design was reasoning about. The +same commit froze `country`: + +``` +| customer_id | source_now | batch_frozen | sdp_frozen | rows | +|-------------|------------|--------------|------------|---------| +| 51 | UK | US | UK | 12,000 | +| 52 | UK | US | UK | 12,000 | +| ... | | | | | +``` + +`_seed_initial` resets `customer.country` to its banded default (ids 1–200 → US, 201–300 → UK, …), and each +incremental day MERGEs country changes onto a rotating window of customers. Batch silver, rebuilt at 12:12:00 +right after the reset, holds the **banded default**. SDP silver holds the **accumulated mutations** of the prior +incarnation. Both froze; they froze different things. + +447 of 500 customers are affected. Nobody designed `country` to be frozen — it came along for the ride. + +--- + +## 6. Divergence 3 — the missing day (batch side) + +Batch silver is missing **2026-07-24 entirely**: + +``` +| date | curated.order_enriched | ..._sdp | raw.order | +|------------|------------------------|---------|-----------| +| 2026-07-22 | 5,000 | 5,000 | 5,000 | +| 2026-07-23 | 5,000 | 5,000 | 5,000 | +| 2026-07-24 | — | 5,000 | 5,000 | +| 2026-07-25 | 5,000 | 5,000 | 5,000 | +``` + +`job1_prod` run **646670311225438** started 2026-07-24 09:01:50Z and **failed**: + +> Task `extract_source2` failed with message: Workload failed, see run output for details. This caused all +> downstream tasks to get skipped. + +That is the DQX quarantine schema-drift outage fixed by PR #52. The next run (1003939029330611, 2026-07-25 +09:01:38Z) succeeded, and everything looked healthy again — **but the hole was never filled**: + +- `raw.order` is a full `CREATE OR REPLACE` (v32 on 07-25 jumped 2,145,000 → 2,155,000, catching up *both* days), + so bronze self-healed. +- `curated.order_enriched`'s incremental branch filters `raw.order` to **`date == seed_date`** + (`generate_orders.py:62`) and MERGEs only that. The 07-25 run looked at 07-25 only. No later run ever looks + back. Silver history jumps straight from v67 (07-23) to v70 (07-25). + +The SDP path has no such hole: a streaming read consumes whatever is new, regardless of what date it carries. + +**This is a live gap in a production gold table** — `report.order_agg` under-reports 2026-07-24 — and it is +independent of divergences 1 and 2. It is also the more likely one to recur: any failed daily run leaves a +permanent hole that no subsequent run repairs. + +--- + +## 7. Divergence 4 — the tripled day (SDP side) + +`external_source.order` received the 2026-06-24 incremental batch **twice** (v5 at 12:19:39, v7 at 15:55:05), +leaving 5,000 order ids with two copies each. Downstream: + +- **Batch:** DQX's uniqueness check quarantined *both* copies — `raw.order_quarantine` holds exactly 10,000 rows + over 5,000 ids, all dated 2026-06-24. Those orders had already been merged in at 12:21:45, so batch silver + holds them once; the 15:57 MERGE inserted 0. +- **SDP:** no DQX stage. The streaming table appended 5,000 (commit 5) then 10,000 (commit 6), leaving each + `(order_id, item_seq)` present **three** times — 5,000 keys × 3 = 15,000 rows. + +This accounts for the entire 15,000-row count gap between the two silvers. It is an asymmetry by design (only +the batch path runs DQX), but it means the SDP gold table over-counts 2026-06-24. + +--- + +## 8. Impact + +| Table | Effect | +|---|---| +| `prod.curated.order_enriched_sdp` | 6M rows with `order_date` 13 days early and `country` from a dead incarnation; 2026-06-24 tripled | +| `prod.report.order_agg_sdp` | Inherits all of the above — the earliest 13 days of the window are fabricated | +| `prod.curated.order_enriched` | Missing 15,000 rows for 2026-07-24 | +| `prod.report.order_agg` | Under-reports 2026-07-24 | +| The AI/BI dashboard | Binds to the batch gold table, so it shows the 07-24 hole, not the SDP shift | + +Any side-by-side batch/SDP comparison is currently misleading in **both** directions. + +--- + +## 9. Fix options, and what each one costs + +Divergences rarely have a free repair. State what moves, in both directions, before recommending anything. + +**A. SDP full refresh — do not do this alone.** It re-derives `order_date` and `country` correctly, but a full +refresh of a streaming table discards **every frozen value it was carrying**, including `product_name`. Since +2026-06-24 the incremental seed has renamed products daily; a refresh relabels historically booked orders with +current names. That trades a divergence in two columns for a divergence in the one column the design exists to +protect — and the product-name freeze is the template's headline invariant. + +**B. `make drop env=prod` + full rebuild.** Converges both paths and clears divergences 1–4 at once. But run +against today's code it **re-triggers the root cause**: the backfill re-anchors to the new run date, and prod's +entire history moves again. Only safe after C. + +**C. Anchor `_seed_initial` on `_EPOCH` (recommended first step).** A one-line change at `seed_sources.py:135` +to derive the initial window from the same fixed constant the incremental path already uses. Makes the backfill +idempotent so a reseed reproduces identical dates. Fixes nothing already in prod on its own — it makes B safe. + +**D. Backfill 2026-07-24 into batch silver.** Independent of A–C and much cheaper. Either run `job1` with +`seed_date=2026-07-24` (the incremental MERGE is insert-only and keyed on `(order_id, item_seq)`, so it is +idempotent and will not disturb other days), or widen the incremental filter from `date == seed_date` to a +lookback window so a failed day self-heals on the next run. The second is the real fix: today, **any** failed +daily run leaves a permanent hole. + +**Recommended order: C → D → deploy → B.** C makes the rebuild safe, D closes the live gold-table gap without +waiting for a rebuild, and B is what actually converges the two paths. + +Before executing B, confirm both writers are healthy — resetting a table whose writer is failing leaves it empty. + +--- + +## 10. Standing lessons + +- **A date derived from "now" at load time is not a fact.** Anything anchored on `current_date()`, a run date or + a job parameter re-derives itself on every reload. No downstream reset fixes that permanently. +- **An append-only table freezes every column it appends.** The `country` divergence was collateral damage from + a freeze designed for `product_name`. When a pipeline deliberately freezes one attribute, enumerate what else + it is reading from the same static side. +- **A date-scoped incremental turns any failed run into a permanent hole.** The layer above self-healed because + it was a full overwrite; the layer below did not because it only ever looks at one day. +- **`DESCRIBE HISTORY` refusing an object is information.** It told us `raw.order_sdp` is a materialized view, + which is exactly why its intermediate state is unrecoverable. +- **Verify the premise, then keep looking.** The reported complaint was one shifted date column. Three more + divergences turned up, one of them a live gap in a production gold table that nobody had reported. + +--- + +## Appendix — queries + +```sql +-- 1. Count down both paths (§1) +SELECT 'external_source.order' AS layer, COUNT(*) n, COUNT(DISTINCT id) k, + MIN(date) min_d, MAX(date) max_d FROM prod.external_source.order +UNION ALL SELECT 'raw.order', COUNT(*), COUNT(DISTINCT id), MIN(date), MAX(date) FROM prod.raw.order +UNION ALL SELECT 'raw.order_sdp', COUNT(*), COUNT(DISTINCT id), MIN(date), MAX(date) FROM prod.raw.order_sdp; + +-- 2. Diff by key, with every other column checked in the same pass (§2) +SELECT datediff(b.order_date, s.order_date) AS delta_days, COUNT(*) n, + COUNT(*) FILTER (WHERE b.product_name <> s.product_name) AS name_diff, + COUNT(*) FILTER (WHERE b.item_total <> s.item_total) AS total_diff, + COUNT(*) FILTER (WHERE b.order_total <> s.order_total) AS otot_diff, + COUNT(*) FILTER (WHERE b.country <> s.country) AS country_diff, + COUNT(*) FILTER (WHERE b.item_quantity <> s.item_quantity) AS qty_diff +FROM prod.curated.order_enriched b +JOIN prod.curated.order_enriched_sdp s USING (order_id, item_seq) +GROUP BY 1 ORDER BY n DESC; + +-- 3. Pin the blast radius — not bounded by time-travel retention (§3) +SELECT _metadata.row_commit_version AS v, COUNT(*) n, MIN(order_date), MAX(order_date) +FROM prod.curated.order_enriched_sdp GROUP BY 1 ORDER BY 1; + +-- 4. Timeline. Select operationParameters.mode explicitly: an overwrite is logged as +-- WRITE with mode='Overwrite', so filtering on `operation` alone hides it. (§4) +SELECT version, timestamp, operation, operationParameters.mode AS mode, + operationMetrics.numOutputRows AS out_rows +FROM (DESCRIBE HISTORY prod.external_source.order) ORDER BY version; + +-- 5. The country freeze: source now vs what each side froze (§5) +WITH d AS ( + SELECT b.customer_id, b.country AS batch_frozen, s.country AS sdp_frozen, COUNT(*) n + FROM prod.curated.order_enriched b + JOIN prod.curated.order_enriched_sdp s USING (order_id, item_seq) + WHERE b.country <> s.country GROUP BY 1,2,3) +SELECT d.customer_id, c.country AS source_now, d.batch_frozen, d.sdp_frozen, d.n +FROM d JOIN prod.external_source.customer c ON c.id = d.customer_id ORDER BY d.customer_id; + +-- 6. Rows on one side only — locates the missing day (§6) +SELECT order_id, item_seq FROM prod.curated.order_enriched_sdp +EXCEPT SELECT order_id, item_seq FROM prod.curated.order_enriched; +``` diff --git a/README.md b/README.md index 39b382a..b0cf1bf 100644 --- a/README.md +++ b/README.md @@ -58,8 +58,9 @@ This project template demonstrates how to: - utilize the [Databricks SDK for Python](https://docs.databricks.com/en/dev-tools/sdk-python.html) to manage catalogs, schemas, workspaces, and accounts. Refer to the `scripts` folder for examples. - utilize [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) to manage permissions and get data lineage. - enforce production guardrails out of the box — identity-locked CI deploys, a health-check task, wheel version pinning, per-task timeouts, schema-drift guards, queued runs, and on-call alerting. -- track project cloud spend in USD across AWS (Cost Explorer) and Databricks ([`system.billing`](https://docs.databricks.com/aws/en/admin/system-tables/pricing)) with `make project-costs` — see an [example report](.claude/skills/project-costs/examples/2026-07-22.md). -- diagram any SQL query with [sqlglot](https://github.com/tobymao/sqlglot) — see an [example](.claude/skills/sql-diagram/examples/job_spend_plan.svg). +- use a Claude 'project-costs' skill to track project cloud spend in USD across AWS (Cost Explorer) and Databricks — see an [example](.claude/skills/project-costs/examples/2026-07-22.md). +- use a Claude 'sql-diagram' skill to diagram any SQL query with [sqlglot](https://github.com/tobymao/sqlglot) — see an [example](.claude/skills/sql-diagram/examples/job_spend_plan.svg). +- use a Claude 'data-divergence' skill to investigate why two datasets that should agree don't — see an [example](.claude/skills/data-divergence/examples/2026-08-06-prod-batch-vs-sdp.md). - utilize serverless job clusters on [Databricks Free Edition](https://docs.databricks.com/aws/en/getting-started/free-edition) to deploy your pipelines. diff --git a/specs/tooling.md b/specs/tooling.md index cf188bc..9804e85 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -137,20 +137,26 @@ is the kit's entry point for CLI, auth, and bundle work — load it first, then Committed under `.claude/skills/`, and **not** part of the kit — don't expect `install.sh` to update them, and do keep them in sync with the code they wrap. -- **data-divergence** — investigating why two datasets that should agree don't. Written generically - (no table or column names from this project), so it covers batch vs SDP, a gold rollup vs the - silver it aggregates, a dashboard tile vs its source, and prod vs staging alike. +- **data-divergence** — investigating why two datasets that should agree don't. The procedure is + written generically (no table or column names from this project), so it covers batch vs SDP, a + gold rollup vs the silver it aggregates, a dashboard tile vs its source, and prod vs staging + alike; its `examples/` report is the one place it names real `prod` tables. - **project-costs** — wraps `scripts/project_costs.py` via `make project-costs`: runs the report, then writes the analysis into its `## Analysis` placeholder. - **sql-diagram** — wraps `scripts/sql_diagram.py` via `make sql-diagram`: query plan or column lineage as `.mmd` + `.svg`, plus how to read each mode. -The latter two each ship a worked example showing what good output looks like, kept under +Each ships a worked example showing what good output looks like, kept under `.claude/skills//examples/` rather than in `reports/`, because **all of `reports/` is gitignored generated output** — that is where the tools write, and nothing there is committed. A new example is a copy into `examples/`, never a `git add -f` out of `reports/`. -The two take different forms, and the difference is the rule worth copying. `sql-diagram` ships +`data-divergence`'s example is the odd one out: the other two demonstrate a *format*, and a second +run would produce much the same document, so one example is enough. An investigation has no fixed +output — its example is kept for the **findings**, which are live defects in `prod`, and a second +investigation would be a second example rather than a replacement. + +The other two take different forms, and the difference is the rule worth copying. `sql-diagram` ships `job_spend_plan.html`, which *is* the deliverable — diagram plus a data trace — so it needs no prose companion; it had one, and the file was deleted once the page demonstrated with data what the prose had asserted. `project-costs` ships `example.md` beside its report, because a page of cost tables From e26b7e6bc4efbeac25775c2fb1febfc721d0c3aa Mon Sep 17 00:00:00 2001 From: Andre Date: Thu, 6 Aug 2026 09:48:31 -0300 Subject: [PATCH 7/9] docs: drop project-costs/example.md, the last commentary file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every principle in it was already stated generically in SKILL.md — leading with the cloud split, dollarizing spikes before judging them, using the daily
block to attribute a spike to a date, normalizing partial edge weeks, reading SQL Serverless silence as a finding, reconciling attributed spend before trusting the per-job table, and comparing per active day. The file re-narrated those against one window's numbers. Two points were not already generic and moved into SKILL.md: that a batch-vs-SDP gap becomes a finding through durability (holding across the window and repeating in a second environment) rather than size, and that the example is to be read for shape, not for figures. sql-diagram lost its example.md for the same reason last commit, so tooling.md now states the rule in its general form instead of justifying one exception. Co-Authored-By: Claude Opus 5 --- .claude/skills/project-costs/SKILL.md | 11 +++-- .claude/skills/project-costs/example.md | 60 ------------------------- specs/tooling.md | 12 ++--- 3 files changed, 13 insertions(+), 70 deletions(-) delete mode 100644 .claude/skills/project-costs/example.md diff --git a/.claude/skills/project-costs/SKILL.md b/.claude/skills/project-costs/SKILL.md index a9d24ed..431f77f 100644 --- a/.claude/skills/project-costs/SKILL.md +++ b/.claude/skills/project-costs/SKILL.md @@ -1,6 +1,6 @@ --- name: project-costs -description: Run the project cost report and write the analysis into it. Use when asked about this project's cloud spend, cost anomalies, spikes or trends, DBU/DSU consumption, per-job or per-pipeline cost, or the AWS vs Databricks split. Runs `make project-costs` (AWS Cost Explorer + Databricks system.billing), then analyses the generated report and replaces its Analysis placeholder. See example.md for a committed report read end to end. +description: Run the project cost report and write the analysis into it. Use when asked about this project's cloud spend, cost anomalies, spikes or trends, DBU/DSU consumption, per-job or per-pipeline cost, or the AWS vs Databricks split. Runs `make project-costs` (AWS Cost Explorer + Databricks system.billing), then analyses the generated report and replaces its Analysis placeholder. See examples/ for a finished report. --- # Project cost analysis @@ -69,7 +69,9 @@ to dollars before calling them big or small. the days someone deployed, so compare *per active day*, never raw totals. The `Days` column is what makes that comparison possible. - Compare `job1_*` against its `job1_sdp_*` counterpart: they produce the same medallion tables by - different execution models, so a persistent gap between them is a real finding, not noise. + different execution models, so a persistent gap between them is a real finding, not noise. What + makes it a finding is **durability** — a gap that holds across the whole window *and* repeats in a + second environment is an argument; the same gap seen once is a number. - Watch the integration-test jobs. They are easy to overlook and can rival the pipeline they test. - Reconcile before trusting: the attributed total is always *less* than the Databricks total, since SQL warehouse and interactive compute carry no `job_id`. The note under the table gives the @@ -84,8 +86,9 @@ to dollars before calling them big or small. charging the AWS account for EC2. AWS spend is therefore a *proxy for job activity*, never a measure of pipeline cost. -`example.md` walks a committed report showing what each of these sections looks like when written -against real numbers. +`examples/2026-07-22.md` is a finished report showing what these sections look like written against +real numbers. Read it for shape, not for figures — they are a snapshot of one 30-day window, and a +live run writes a fresh `reports/cost/YYYY-MM-DD.md`. ## Caveats to respect diff --git a/.claude/skills/project-costs/example.md b/.claude/skills/project-costs/example.md deleted file mode 100644 index ec75441..0000000 --- a/.claude/skills/project-costs/example.md +++ /dev/null @@ -1,60 +0,0 @@ -# Worked example — `examples/2026-07-22.md` - -A finished 30-day report, kept here rather than under `reports/` (which is entirely gitignored -generated output). Read the file itself; this page is about *why* its Analysis section is written the -way it is. Do not copy its numbers into a new report — they are a snapshot of one window, and a live -run writes to `reports/cost/YYYY-MM-DD.md`. - -## What the numbers were - -$39.81 total: Databricks $39.43 (99.0%) at list, AWS $0.38 (1.0%). Jobs Serverless 68.09 DBU -($23.83), SQL Serverless 21.57 DBU ($15.10), storage 20.66 DSU ($0.48). Attributed to jobs: $21.49 -of $39.43. - -## What the analysis did with them, and why - -**It led with the split, not with AWS.** $0.38 of AWS is noise; opening with it would bury the fact -that the entire cost conversation is about DBUs. Section order in the report follows the file; -narrative order should follow the money. - -**It converted a spike to dollars before judging it.** The AWS week of 2026-07-13 is ~5× the -surrounding baseline, which sounds alarming until you say it is $0.1925. Ratios without dollars -mislead on a project this small. - -**It used the daily `
` block to attribute the spike to a date.** The weekly pivot only says -"that week"; the daily block pinned it to 2026-07-16 and identified Cost Explorer API calls as the -driver — i.e. the report measuring itself, since Cost Explorer bills per request. That conclusion is -unreachable from stdout alone, which is why the skill insists on reading the report file. - -**It normalized the edge weeks before claiming a trend.** Raw weekly totals suggested a decline; -per-day Jobs Serverless ($1.22/day → $0.42/day → ~$0.72/day) showed a stable baseline with one -late-June spike instead. Opposite conclusion, same data. - -**It treated SQL Serverless silence as a finding.** Two and a half weeks at exactly $0.00, then -$2.90 and $3.58 on two days. The absence is the signal — nothing scheduled touches the warehouse and -nobody opens the dashboard on an ordinary day — and at $15.10 it is the second most expensive thing -in the project despite running about five days out of thirty. - -**It reconciled before trusting the per-job table.** $21.49 attributed against $39.43 total, with -the ~$17.94 gap explained by the SQL warehouse carrying no `job_id`. Because that reconciles, the -breakdown is trustworthy *as a picture of scheduled work only* — stated explicitly rather than -letting the reader assume it covers everything. - -**It compared per active day, never raw totals.** prod ran 31 days at $0.46/day; staging ran 5 days -at $1.35/day — 2.9× prod's daily burn, invisible in the raw column where prod looks far more -expensive. The `Days` column exists for exactly this. - -**It found the batch-vs-SDP gap and argued it was real.** `job1_prod` $6.58 vs `job1_sdp_prod` $4.03 -for the same medallion tables, repeated in staging ($2.48 vs $1.62). A 35–39% gap holding over 31 -days *and* across two environments is what separates a finding from noise — the durability is the -argument, not the single number. - -**It checked the integration tests.** `job1_prod_integration` at $3.78 is 57% of the pipeline it -validates, and in staging the integration test cost *more* than the job under test. Easy to skim -past; the skill calls them out because of this. - -## The shape to reproduce - -One paragraph per section, every claim carrying its number, comparisons normalized before they are -made, and absences reported as findings. No filler, no restating tables that are already in the file -directly above. diff --git a/specs/tooling.md b/specs/tooling.md index 9804e85..2a8ed6e 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -156,12 +156,12 @@ run would produce much the same document, so one example is enough. An investiga output — its example is kept for the **findings**, which are live defects in `prod`, and a second investigation would be a second example rather than a replacement. -The other two take different forms, and the difference is the rule worth copying. `sql-diagram` ships -`job_spend_plan.html`, which *is* the deliverable — diagram plus a data trace — so it needs no prose -companion; it had one, and the file was deleted once the page demonstrated with data what the prose -had asserted. `project-costs` ships `example.md` beside its report, because a page of cost tables -does not explain why its analysis is written the way it is. **Add a commentary file only when the -artifact cannot speak for itself**; two files narrating one artifact will drift. +**An example is the artifact, never a commentary file beside it.** Both `sql-diagram` and +`project-costs` once shipped an `example.md` explaining their example; both were deleted, and in each +case the explanation belonged in one of two places — the artifact itself, or the `SKILL.md` as +guidance that applies to every run, not just to the committed one. Two files narrating one artifact +will drift, and the prose is the copy that goes stale. If an example needs a companion to be +intelligible, fix the example. One gotcha on the kit's skills. Some have a frontmatter `name:` that differs from their directory (`databricks` declares `databricks-core`; `analyze-mlflow-trace` declares `analyzing-mlflow-trace`) From 2d9d640d8bcb1c9fe612d8de4b73c1d5954515de Mon Sep 17 00:00:00 2001 From: Andre Date: Thu, 6 Aug 2026 10:14:40 -0300 Subject: [PATCH 8/9] feat: gate merges on a PR description verified against the merged commit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pr-merge-description.sh copies the PR body into the merge commit message, so a description written five commits ago does not just mislead on GitHub — it becomes permanent history. This PR's own description was the worked example: it still advertised an example.md that two commits had deleted, and warned that job1_prod was failing daily with the fix unmerged, which had been false since 2026-07-25. A hook cannot judge whether prose is true. require-fresh-pr-description.sh proves the weaker thing that catches the real failure: that someone looked at the description at the exact commit being merged. The contract is a sentinel in the body, , checked against HEAD. When it does not match, the block lists what landed since it was last verified. It runs before pr-merge-description.sh, because a stale body caught after that hook is already in history. Also fixes a confirmed defect in pr-merge-description.sh: it never checked that the command was a merge, relying entirely on the settings.json `if:` filter. That filter leaks — the hook rewrote unrelated Bash commands mid-session and fired a gh pr edit network write on each one. It now self-gates on the command text like its siblings. Co-Authored-By: Claude Opus 5 --- .claude/hooks/pr-merge-description.sh | 10 +- .claude/hooks/require-fresh-pr-description.sh | 97 +++++++++++++++++++ .claude/settings.json | 7 ++ specs/tooling.md | 17 +++- 4 files changed, 126 insertions(+), 5 deletions(-) create mode 100755 .claude/hooks/require-fresh-pr-description.sh diff --git a/.claude/hooks/pr-merge-description.sh b/.claude/hooks/pr-merge-description.sh index 05fdc10..a913f9b 100755 --- a/.claude/hooks/pr-merge-description.sh +++ b/.claude/hooks/pr-merge-description.sh @@ -6,7 +6,7 @@ INPUT=$(cat) # Use Python for all JSON parsing and output (jq has snap confinement issues in this env) python3 - "$INPUT" <<'PYEOF' -import json, sys, os, subprocess, tempfile, shlex +import json, sys, os, re, subprocess, tempfile, shlex raw = sys.argv[1] try: @@ -16,6 +16,13 @@ except json.JSONDecodeError: command = data.get("tool_input", {}).get("command", "") +# Self-gate on the command text rather than trusting the settings.json `if:` filter. +# Without this the hook runs on EVERY Bash call: it appends --subject/--body-file to +# unrelated commands (breaking them) and fires a `gh pr edit` network write each time. +# Observed in practice, which is why the check is here and not only in settings.json. +if not re.search(r'\bgh\s+pr\s+merge\b', command): + sys.exit(0) + # Already has explicit body/subject — don't override user's intent if any(f in command for f in ("--body", "--subject", "--body-file")): sys.exit(0) @@ -25,7 +32,6 @@ if "--rebase" in command: sys.exit(0) # Extract PR ref (number, URL, or branch) — first non-flag token after 'gh pr merge' -import re m = re.search(r'gh pr merge\s+([^\s-]\S*)', command) pr_arg = m.group(1) if m else None diff --git a/.claude/hooks/require-fresh-pr-description.sh b/.claude/hooks/require-fresh-pr-description.sh new file mode 100755 index 0000000..541eee7 --- /dev/null +++ b/.claude/hooks/require-fresh-pr-description.sh @@ -0,0 +1,97 @@ +#!/bin/bash +# PreToolUse hook: block `gh pr merge` unless the PR description has been re-confirmed +# against the exact commit being merged. +# +# A hook cannot judge whether prose is true. What it can prove is that nobody has looked +# at the description since the branch last moved — the failure mode that actually bites, +# because pr-merge-description.sh copies the body into the merge commit message. A stale +# description does not just mislead on GitHub (where it can be edited); it becomes +# permanent git history. +# +# The contract is a sentinel line anywhere in the PR body: +# +# +# +# Whoever writes or revises the description stamps it with the branch tip they checked it +# against. If the branch advances afterwards, the sentinel no longer matches and the merge +# is blocked until someone re-reads the description and re-stamps it. Re-stamping without +# reading is possible — this enforces a deliberate act, not honesty. +# +# Self-gates on the command text rather than relying on the settings.json `if:` filter, so +# it stays correct even if that filter doesn't apply. +set -euo pipefail + +INPUT=$(cat) +COMMAND=$(echo "$INPUT" | jq -r '.tool_input.command // ""') + +# Not a merge → nothing to enforce. +echo "$COMMAND" | grep -qE '\bgh\b\s+pr\s+merge\b' || exit 0 + +# No gh, no PR, no git → don't block on an unanswerable question. +command -v gh >/dev/null 2>&1 || exit 0 +HEAD_SHA=$(git rev-parse HEAD 2>/dev/null) || exit 0 + +# First non-flag token after `gh pr merge` is the PR ref, when one is given. +PR_ARG=$(echo "$COMMAND" | sed -nE 's/.*\bgh[[:space:]]+pr[[:space:]]+merge[[:space:]]+([^-[:space:]][^[:space:]]*).*/\1/p' | head -1) + +if [ -n "$PR_ARG" ]; then + BODY=$(gh pr view "$PR_ARG" --json body -q .body 2>/dev/null) || exit 0 +else + BODY=$(gh pr view --json body -q .body 2>/dev/null) || exit 0 +fi + +STAMP=$(printf '%s' "$BODY" \ + | sed -nE 's/.*.*/\1/p' \ + | tail -1) + +# Stamp is a prefix of the tip (allows a short sha) → the description was confirmed here. +if [ -n "$STAMP" ] && [ "${HEAD_SHA#"$STAMP"}" != "$HEAD_SHA" ]; then + exit 0 +fi + +# Build the "what changed since you last looked" list, when the stamp names a real commit. +NEW_COMMITS="" +if [ -n "$STAMP" ] && git cat-file -e "${STAMP}^{commit}" 2>/dev/null; then + NEW_COMMITS=$(git log --oneline --no-decorate "${STAMP}..HEAD" 2>/dev/null | head -10) +fi + +python3 - "$HEAD_SHA" "$STAMP" "$NEW_COMMITS" <<'PYEOF' +import json, sys + +head, stamp, new_commits = sys.argv[1], sys.argv[2], sys.argv[3] +short = head[:7] + +if not stamp: + why = ( + "The PR description carries no `description-verified` stamp, so there is no evidence\n" + "anyone has checked it against what is actually being merged." + ) +else: + why = ( + f"The PR description was last verified at {stamp}, but the branch tip is now {short}.\n" + "It describes an older state of this branch." + ) + if new_commits: + why += "\n\nLanded since it was verified:\n" + "\n".join( + " " + line for line in new_commits.splitlines() + ) + +print(json.dumps({ + "continue": False, + "stopReason": ( + "Merge blocked: the PR description has not been confirmed against this commit.\n\n" + f"{why}\n\n" + "This matters more than it looks: pr-merge-description.sh copies the body into the\n" + "merge commit message, so whatever is there now becomes permanent history.\n\n" + "To clear this:\n" + " 1. Re-read the description against the branch as it stands " + "(`git log --oneline origin/main..HEAD`).\n" + " 2. Update What / Why / How / Validation / Impact in prod so they match reality —\n" + " including removing claims that were true when written and are not now.\n" + f" 3. Stamp it by putting this line at the end of the body:\n" + f" \n" + " 4. Push the updated body with `gh pr edit --body-file `, then merge.\n\n" + "Full template: .github/PULL_REQUEST_TEMPLATE.md · rule: specs/workflow.md" + ), +})) +PYEOF diff --git a/.claude/settings.json b/.claude/settings.json index 8e00d85..941267c 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -4,6 +4,13 @@ { "matcher": "Bash", "hooks": [ + { + "type": "command", + "if": "Bash(gh pr merge*)", + "command": "bash \"$CLAUDE_PROJECT_DIR/.claude/hooks/require-fresh-pr-description.sh\"", + "timeout": 15, + "statusMessage": "Checking the PR description is current..." + }, { "type": "command", "if": "Bash(gh pr merge*)", diff --git a/specs/tooling.md b/specs/tooling.md index 2a8ed6e..f391837 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -90,7 +90,7 @@ directly (or `aws` CLI / web search for the AWS and context7 cases) — but flag ## Hooks -`.claude/hooks/` holds the three shell hooks that enforce the git workflow in +`.claude/hooks/` holds the four shell hooks that enforce the git workflow in [workflow.md](workflow.md). They are **committed** — the rules they enforce are stated as project rules in `CLAUDE.md`, so shipping the scripts is what makes those statements true for a fresh clone rather than a description of one machine's setup. @@ -99,6 +99,7 @@ rather than a description of one machine's setup. |---|---|---| | `protect-main-branch.sh` | any Bash `git commit` / `git push` | blocks a commit made while on `main`, and any push targeting `main`. | | `require-changelog-entry.sh` | Bash `gh pr merge` | blocks the merge unless the branch diff touches `specs/CHANGELOG.md`, compared against `origin/main`/`main` via a merge-base (`...`) diff. | +| `require-fresh-pr-description.sh` | Bash `gh pr merge` | blocks the merge unless the PR body carries `` matching the commit being merged. | | `pr-merge-description.sh` | Bash `gh pr merge` | pushes the PR title/body to GitHub, then rewrites the command with `--subject`/`--body-file` so the description becomes the merge commit message. Skips if `--body`/`--subject`/`--body-file` or `--rebase` is already present. | The scripts are portable — no absolute paths, no secrets — and are committed mode `755`. @@ -110,8 +111,18 @@ path — that variable is what keeps the file valid in any clone, and a hardcode that would quietly break it for everyone else. Keep machine-specific hooks (an update check pointing into `~/.ai-dev-kit/`, say) in `.claude/settings.local.json` instead; the two files are merged. -`require-changelog-entry.sh` and `protect-main-branch.sh` self-gate on the command text instead of -trusting a settings-level `if:` filter, so they stay correct however they are registered. +**Every hook self-gates on the command text** instead of trusting a settings-level `if:` filter, so +they stay correct however they are registered. `pr-merge-description.sh` originally did not, and the +filter alone proved not to hold: it ran on unrelated Bash calls, appending `--subject`/`--body-file` +to commands that were not merges (breaking them) and firing a `gh pr edit` network write each time. +A settings filter is a convenience; the gate belongs in the script. + +The two merge gates are ordered deliberately — `require-fresh-pr-description.sh` runs *before* +`pr-merge-description.sh`, because the second one copies the body into the merge commit message. A +stale description caught after that point is already permanent history. Note what the freshness gate +does and does not prove: it cannot judge whether prose is accurate, only that someone re-stamped it +against the exact commit being merged. Re-stamping without reading is possible; it enforces a +deliberate act, not honesty. ## Databricks CLI From 2da44f9e414c34d61f3ffad5f13838a89d3125de Mon Sep 17 00:00:00 2001 From: Andre Date: Thu, 6 Aug 2026 10:15:20 -0300 Subject: [PATCH 9/9] docs: add the #53 CHANGELOG entry Co-Authored-By: Claude Opus 5 --- specs/CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/specs/CHANGELOG.md b/specs/CHANGELOG.md index 2f117f6..ac274a2 100644 --- a/specs/CHANGELOG.md +++ b/specs/CHANGELOG.md @@ -2,6 +2,12 @@ --- +## [#53](https://github.com/andre-salvati/databricks-template/pull/53) · 2026-08-06 · Commit the agent tooling: workflow hooks, three project skills, and a fully ignored reports/ + +Made this repo's agent tooling part of the repo instead of one developer's local config. `.claude/hooks/` and the `settings.json` registering them are committed, so the rules `CLAUDE.md` and `specs/workflow.md` state — no direct commits to `main`, a CHANGELOG entry before merge, the PR description as merge commit body — hold for a fresh clone rather than describing one machine. The two slash commands became skills (`project-costs`, `sql-diagram`) so their descriptions match on relevance instead of waiting to be typed, joined by a new `data-divergence` skill for investigating why two datasets that should agree don't. `.gitignore` un-ignores `.claude/` by name, never by wildcard, which would sweep in the Dev Kit's user-level skills. All of `reports/` is now ignored, with each skill's worked example committed beside it. A fourth hook blocks merges unless the PR body carries a `description-verified` sentinel matching the commit being merged, because that body becomes permanent history — this PR's own stale description was the case in point. + +--- + ## [#52](https://github.com/andre-salvati/databricks-template/pull/52) · 2026-07-24 · fix: unblock job1_prod (DQX quarantine schema) + rollback playbook and CI secret hardening A DQX 0.15 upgrade (#51) widened the quarantine `_errors`/`_warnings` structs, and the per-env `make drop` it prescribed was applied to staging but not prod, so `job1_prod` hard-failed on `DELTA_METADATA_MISMATCH` for a day. Fixed at the source: `raw.order_quarantine` now writes with `overwriteSchema=true`, the second intentional exception after `ops._health`, because those structs' shape belongs to the DQX version and not to the source data contract — pinning library metadata makes every dependency bump an outage. Real drift is still caught, since quarantine's business columns come from the same DataFrame written to `raw.order` under `overwriteSchema=false`. Also added a rollback playbook, built on the rule that a redeploy reverts code but never data: git history is the artifact store, and `curated.order_enriched` can only be restored, never re-derived, since it freezes `product_name` at sale time. Finally the CI client secret moved from the workflow-level `env` block, where dependency installation could read it, into the three steps that reach Databricks, and actions are pinned by SHA; OIDC would remove the secret entirely but needs an account-level API that Free Edition lacks.