diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7ac9235..866562b 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -7,5 +7,5 @@ repos: - id: ruff args: - --fix - files: ^src/ + files: ^(src|scripts)/ - id: ruff-format \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 0aa2f92..c999fe9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,10 +8,10 @@ A production-ready PySpark/Databricks ETL pipeline template using medallion arch ## Tooling: MCP servers, CLI, skills → see [`specs/tooling.md`](specs/tooling.md) -Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) (user-level, `~/.ai-dev-kit/`). MCP config (`.mcp.json`) and `.claude/` are gitignored — user-level tooling, not committed. Quick decision list (full reference in `specs/tooling.md`): +Developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) (user-level, `~/.ai-dev-kit/`). MCP config (`.mcp.json`) and `.claude/` are gitignored — user-level tooling, not committed; the exception is `.claude/commands/`, which is un-ignored and committed. Quick decision list (full reference in `specs/tooling.md`): - **Workspace / UC / Jobs / Pipelines / Apps / Serving / SQL** → prefer `mcp__databricks__*` tools over `databricks` CLI shell-outs or hand-rolled SDK scripts. -- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, but route job edits through `scripts/sdk_generate_template_job.py` + `make deploy` (never hand-edit `resources/jobs.yml`). +- **Bundle / job changes** → `databricks-bundles` / `databricks-jobs` skills, and route job edits through `scripts/sdk_generate_template_job.py` + `make deploy`. - **Library/SDK docs** (PySpark, Databricks SDK, uv, ruff) → `context7` MCP, not memory or web search. - **Cloud spend / cost analysis** → `aws-billing-cost` MCP (`AWS_PROFILE=costs`) + `/project-costs` skill. **AWS docs** → `aws-documentation` MCP. - Use the `dev` profile unless told otherwise (`prod` for prod ops). If MCP tools are unavailable, fall back to CLI/SDK and flag it. @@ -27,6 +27,9 @@ make init # One-time workspace bootstrap (SP, catalogs, schemas, gr make deploy env=dev # Generate resources/jobs.yml (jobs + SDP pipeline) + deploy bundle to target env (dev/staging/prod) make run env=dev # Run integration test job on a target env (dev or staging) make drop env=dev # Drop all medallion tables in a target env (schema migrations; staging/prod need yes=--yes) +make whoami # Print the identity the env's profile authenticates as (runs implicitly before deploy/run/drop) +make project-costs # AWS + Databricks spend report (--aws-profile costs); backs the /project-costs skill +make star-history # Regenerate the README star-history SVGs (assets/star_history*.svg) from the GitHub API ``` Run a single test file: @@ -43,7 +46,7 @@ uv run pytest tests/job1/unit_test.py::test_enrich_orders The detailed specs live in [`specs/`](specs/) — read the relevant one **before** working in that area: -- [`specs/architecture.md`](specs/architecture.md) — execution flow, CLI surface, key classes, jobs DAG, job **generation** (`scripts/sdk_generate_template_job.py` → `resources/jobs.yml`; never hand-edit), CI/CD, job-level params, deploy-time env vars, logging, production guardrails, adding a new job. +- [`specs/architecture.md`](specs/architecture.md) — execution flow, CLI surface, key classes, jobs DAG, job **generation**, CI/CD, job-level params, deploy-time env vars, logging, production guardrails, adding a new job. - [`specs/data-model.md`](specs/data-model.md) — plain-words pipeline overview, catalog/schema isolation, medallion flow, table schemas, **field naming conventions**, product-name freeze, liquid clustering, DQX/quarantine, lineage. - [`specs/workflow.md`](specs/workflow.md) — the development lifecycle (plan → branch → PR), PR description standard, production-table impact check, and the unit / integration / load test plan. - [`specs/tooling.md`](specs/tooling.md) — MCP servers (Databricks, AWS billing/docs, context7), CLI, and skills: what to reach for and when. @@ -54,7 +57,6 @@ The detailed specs live in [`specs/`](specs/) — read the relevant one **before - **Catalog-level isolation** — env separation is at the *catalog* level (`dev_` / `staging` / `prod`); the same medallion schemas (`external_source`/`raw`/`curated`/`report`/`ops`) exist in each. Staging/prod catalogs+schemas are owned by `make init`, not the runtime wheel. - **Product-name freeze** — silver freezes `product_name` onto each order line at sale time: batch via an insert-only `MERGE`, SDP via a **streaming table** (a materialized view would *restate* the name; a streaming table appends once and *freezes*). A later rename never relabels booked orders. `unit_price` is static; `total_value` in gold is `SUM(item_total)`. - **Generated job config** — `resources/jobs.yml` is generated by `scripts/sdk_generate_template_job.py`; never hand-edit it (it's gitignored). -- **Schema-drift guard** — all medallion writes use `overwriteSchema=false` (the only exception is `ops._health`). ## Constraints (things that broke us) @@ -70,7 +72,7 @@ The detailed specs live in [`specs/`](specs/) — read the relevant one **before - **Ask "should I open a new branch?" before executing a plan**, and **never commit directly to `main`** — cut a feature branch and land via PR (a hook blocks direct commits and pushes to `main`). - **Hold commits until asked.** Before merging, update the PR description (a hook uses it as the merge commit message body) following the What / Why / How / Validation / **Impact in prod** template in [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md); any table schema/data change needs the production-table impact check. - **Keep docs in sync in the same commit.** Don't ship changes to the CLI surface (`main.py:arg_parser`), runtime env vars, catalog/schema model, or production guardrails without updating `README.md`, the relevant doc under `specs/`, and this file (`CLAUDE.md`) together. -- **Add a `specs/CHANGELOG.md` entry before merging a PR** — append-only (never edit old ones), each entry **at most 3 sentences**. +- **Add a `specs/CHANGELOG.md` entry immediately before merging a PR** (not while the work is in progress — scope grows, and an early entry just gets rewritten). Append-only (never edit old ones). Each entry is **exactly 3 sentences** and **at most ~5 rendered lines** (~475 chars); keep it one unwrapped paragraph — the line cap is a length budget, not a wrap width. ## Keep It Simple @@ -79,8 +81,8 @@ Favor solutions with less code, fewer classes, and fewer abstractions. When two - Don't reintroduce `--user`, `--debug`, or `--schema` CLI args. They were removed deliberately — see PR #21. - Don't add `funcy` (or any decorator-based timing utility) to the dependencies. Use the structured logger. - Don't add `CREATE CATALOG` or `CREATE SCHEMA` calls outside the `args.env == "dev"` branch in `config.py`. Staging/prod catalogs and schemas are owned by `make init`; runtime jobs run without those privileges. -- Don't commit `resources/jobs.yml` (gitignored — regenerated on every deploy). +- Don't commit or hand-edit `resources/jobs.yml` — it's gitignored and overwritten on every deploy. Change `scripts/sdk_generate_template_job.py` instead (it generates jobs, the SDP pipeline, and the dashboard resource stanza). - Don't commit `.databricks-resources.json` (gitignored — local provisioning state, diverges per developer). -- Don't commit or hand-edit `resources/orders_dashboard_deploy.lvdash.json` (gitignored — regenerated from `orders_dashboard.lvdash.json` on every deploy with `${var.catalog}` resolved, since DABs can't substitute bundle vars inside `.lvdash.json` content). Edit the source `orders_dashboard.lvdash.json` instead. -- Don't hand-edit `resources/jobs.yml` — it is overwritten on every deploy. Change `scripts/sdk_generate_template_job.py` instead (it generates jobs, the SDP pipeline, and the dashboard resource stanza into `resources/jobs.yml`). `resources/orders_dashboard.lvdash.json` is committed and editable directly. +- Don't commit or hand-edit `resources/orders_dashboard_deploy.lvdash.json` (gitignored — regenerated on every deploy). Edit the committed `resources/orders_dashboard.lvdash.json` instead — mechanics in [`specs/data-model.md#dashboard`](specs/data-model.md#dashboard). +- Don't hand-edit `assets/star_history*.svg` — they're generated (and committed, unlike `resources/jobs.yml`) so the README renders from this repo instead of a third-party chart service. Regenerate with `make star-history`; change `scripts/star_history.py` to alter the chart. - For PySpark transformation chains, let `ruff format` shape any chain that's already multi-line (don't hand-tune it) and keep chains that fit on one line unbroken — see the convention in [`specs/architecture.md`](specs/architecture.md#code-style-pyspark-transformation-chains). diff --git a/Makefile b/Makefile index d947881..33bac1b 100644 --- a/Makefile +++ b/Makefile @@ -36,3 +36,8 @@ drop: whoami project-costs: aws-profile ?= costs project-costs: uv run python ./scripts/project_costs.py $(if $(aws-profile),--aws-profile $(aws-profile),) + +# Regenerate the README star-history chart from the GitHub API. The SVGs are committed, +# so the README renders from this repo rather than a third-party chart service. +star-history: + uv run python ./scripts/star_history.py diff --git a/README.md b/README.md index 9eff44d..7a21e8e 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# databricks-template — agentic development for Databricks + production-ready ETL +# Databricks PySpark ETL Template — production-ready pipelines, built agentically with Claude Code ![Databricks](https://img.shields.io/badge/platform-Databricks-orange?logo=databricks) ![PySpark](https://img.shields.io/badge/pyspark-4.1+-brightgreen?logo=apache-spark) @@ -9,7 +9,7 @@ ## 🚀 Overview -> Stop spending weeks on boilerplate. This PySpark project template for Databricks gives you medallion architecture, Python packaging, unit + integration + load tests, CI/CD via Declarative Automation Bundles, DQX data quality, and service-principal-based production deploys — all wired together and ready to ship. Whether you're starting a new Databricks ETL project or looking for a reference implementation of production-ready PySpark pipelines, fork this and go. +> Stop spending weeks on boilerplate. This PySpark project template for Databricks gives you medallion architecture, Python packaging, unit + integration + load tests, CI/CD, DQX data quality, service-principal-based production deploys, and much more — all wired together and ready to ship. Whether you're starting a new Databricks ETL project or looking for a reference implementation of production-ready PySpark pipelines, fork this and go. If this saves you time, a star helps others find it. Let's [connect on LinkedIn](https://www.linkedin.com/in/andresalvati/). @@ -27,7 +27,7 @@ If this saves you time, a star helps others find it. Let's [connect on LinkedIn] - Claude Code - PySpark 4.1 - Spark Declarative Pipelines (SDP) -- Python 3.12+ +- Python 3.12 - GitHub Actions - Pytest @@ -134,7 +134,7 @@ run: make deploy env=dev -8) Configure CI/CD automation with the service principal ID and secret. Configure [GitHub Actions repository secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions) (DATABRICKS_HOST, DATABRICKS_PRINCIPAL_ID, DATABRICKS_SECRET). +8) Configure CI/CD automation with the service principal ID and secret. Configure [GitHub Actions repository secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions): `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`, and `TEMPLATE_ALERT_EMAILS` (comma-separated on-call addresses). 9) (Optional) You can also execute unit tests from your preferred IDE. Here's a screenshot from [VS Code](https://code.visualstudio.com/) with [Microsoft's Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) installed. @@ -146,10 +146,12 @@ Licensed under the [Apache License 2.0](LICENSE). See [`LICENSE`](LICENSE) and [ ## Star History - + - - - Star History Chart + + + Star History Chart + +Generated from the GitHub API by `make star-history` — see [`scripts/star_history.py`](scripts/star_history.py). diff --git a/assets/star_history.svg b/assets/star_history.svg new file mode 100644 index 0000000..6d48f63 --- /dev/null +++ b/assets/star_history.svg @@ -0,0 +1,22 @@ + + +andre-salvati/databricks-template — stars over time + + +0 + +20 + +40 + +60 + +80 +Dec 2023 +Apr 2025 +Jul 2026 + + + +79 + diff --git a/assets/star_history_dark.svg b/assets/star_history_dark.svg new file mode 100644 index 0000000..fd496b3 --- /dev/null +++ b/assets/star_history_dark.svg @@ -0,0 +1,22 @@ + + +andre-salvati/databricks-template — stars over time + + +0 + +20 + +40 + +60 + +80 +Dec 2023 +Apr 2025 +Jul 2026 + + + +79 + diff --git a/pyproject.toml b/pyproject.toml index 41b6dc2..40b37fa 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -60,6 +60,11 @@ ignore = [] # spark is injected by the SDP runtime — not a normal Python import "src/template/job1_sdp/pipeline.py" = ["F821"] +[tool.ruff.lint.isort] +# `_sdk_sql` is a local sibling module in scripts/, not a third-party package — +# without this, isort files it under the databricks imports. +known-first-party = ["_sdk_sql"] + [tool.coverage.run] source = ["."] omit = ["tests/*", "setup.py"] diff --git a/scripts/sdk_analyze_job_costs.py b/scripts/sdk_analyze_job_costs.py index 69233bd..df6043a 100644 --- a/scripts/sdk_analyze_job_costs.py +++ b/scripts/sdk_analyze_job_costs.py @@ -14,10 +14,11 @@ """ import argparse +import time from datetime import datetime, timedelta + from databricks.sdk import WorkspaceClient from databricks.sdk.service.sql import StatementState -import time def get_todays_date(): @@ -55,7 +56,7 @@ def analyze_job_costs(workspace: WorkspaceClient, target_date: str): AND u.cloud = list_prices.cloud WHERE usage_date = '{target_date}' - AND + AND u.usage_metadata.job_id IS NOT NULL GROUP BY u.usage_metadata.job_id, diff --git a/scripts/sdk_init_workspace.py b/scripts/sdk_init_workspace.py index 9a3a2fa..0e19dea 100644 --- a/scripts/sdk_init_workspace.py +++ b/scripts/sdk_init_workspace.py @@ -5,7 +5,6 @@ from _sdk_sql import get_warehouse_id, run_sql - SP_DISPLAY_NAME = "template-sp" # Per-developer dev catalogs (`dev_`) and their schemas are created # on the fly by Config when a developer runs the wheel — see src/template/config.py. diff --git a/scripts/sdk_workspace_and_account.py b/scripts/sdk_workspace_and_account.py index 66ae98f..1251d9a 100644 --- a/scripts/sdk_workspace_and_account.py +++ b/scripts/sdk_workspace_and_account.py @@ -1,6 +1,4 @@ -from databricks.sdk import WorkspaceClient -from databricks.sdk import AccountClient -import requests +from databricks.sdk import AccountClient, WorkspaceClient def demoWorkspaceApi(): diff --git a/scripts/star_history.py b/scripts/star_history.py new file mode 100644 index 0000000..108658d --- /dev/null +++ b/scripts/star_history.py @@ -0,0 +1,162 @@ +""" +Render the repo's star history as committed SVGs (light + dark) for the README. + +Reads stargazer timestamps from the GitHub API via `gh` and plots them locally, so the +README depends on files in this repo rather than on a third-party chart service. +Usage: uv run python scripts/star_history.py [--repo OWNER/NAME] [--out-dir DIR] +""" + +import argparse +import json +import subprocess +from datetime import datetime, timezone +from pathlib import Path +from xml.sax.saxutils import escape + +# Slot 1 (blue) of the validated categorical palette, stepped per surface; a single +# series needs no legend, so the title carries identity and the line carries the color. +THEMES = { + "light": {"surface": "#fcfcfb", "text": "#0b0b0b", "muted": "#52514e", "grid": "#e6e6e3", "series": "#2a78d6"}, + "dark": {"surface": "#1a1a19", "text": "#ffffff", "muted": "#c3c2b7", "grid": "#33332f", "series": "#3987e5"}, +} + +WIDTH, HEIGHT = 800, 400 +PAD_L, PAD_R, PAD_T, PAD_B = 64, 88, 56, 44 +MAX_POINTS = 200 +FONT = "-apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,Arial,sans-serif" + + +def fetch_stars(repo: str) -> list[datetime]: + """Return every `starred_at`, oldest first. GitHub caps this endpoint at 40,000 stars.""" + cmd = [ + "gh", + "api", + "-H", + "Accept: application/vnd.github.star+json", + f"/repos/{repo}/stargazers?per_page=100", + "--paginate", + ] + try: + result = subprocess.run(cmd, capture_output=True, text=True) + except FileNotFoundError: + raise RuntimeError("`gh` not found — install the GitHub CLI or ensure it is on PATH.") from None + if result.returncode != 0: + raise RuntimeError(f"gh api failed for {repo}: {result.stderr.strip()}") + + records = json.loads(result.stdout) # `gh --paginate` merges the pages into one array + stars = sorted( + datetime.strptime(r["starred_at"], "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc) + for r in records + if r.get("starred_at") + ) + if not stars: + raise RuntimeError(f"{repo} has no stargazers with timestamps — nothing to plot.") + return stars + + +def nice_ticks(top: int, count: int = 4) -> list[int]: + """Round tick values (0 / 25 / 50 ...) covering 0..top.""" + raw = max(1, top / count) + magnitude = 10 ** (len(str(int(raw))) - 1) + step = next((m * magnitude for m in (1, 2, 2.5, 5, 10) if m * magnitude >= raw), raw) + ticks, value = [], 0 + while value < top + step: + ticks.append(int(value)) + value += step + return ticks + + +def build_svg(repo: str, stars: list[datetime], theme: dict[str, str]) -> str: + # Cumulative count: star N lands at index N. Anchor the series at zero on the first + # star's day so the line starts on the baseline instead of at 1. + points = [(stars[0], 0)] + [(when, i + 1) for i, when in enumerate(stars)] + if len(points) > MAX_POINTS: + stride = len(points) / MAX_POINTS + sampled = [points[int(i * stride)] for i in range(MAX_POINTS)] + points = sampled + [points[-1]] # never drop the current total + + t0, t1 = points[0][0].timestamp(), points[-1][0].timestamp() + span = max(t1 - t0, 1) + total = points[-1][1] + ticks = nice_ticks(total) + y_top = ticks[-1] + + def px(when: datetime) -> float: + return PAD_L + (when.timestamp() - t0) / span * (WIDTH - PAD_L - PAD_R) + + def py(count: int) -> float: + return HEIGHT - PAD_B - (count / y_top) * (HEIGHT - PAD_T - PAD_B) + + coords = [(px(w), py(c)) for w, c in points] + line = "M" + " L".join(f"{x:.1f},{y:.1f}" for x, y in coords) + area = f"{line} L{coords[-1][0]:.1f},{HEIGHT - PAD_B} L{coords[0][0]:.1f},{HEIGHT - PAD_B} Z" + + out = [ + f'', + f'', + f'' + f"{escape(repo)} — stars over time", + f'' + f'' + f'', + ] + + # Recessive hairline gridlines, with the y ticks carrying the values the line doesn't label. + for tick in ticks: + y = py(tick) + out.append( + f'' + ) + out.append( + f'{tick:,}' + ) + + for frac in (0, 0.5, 1): + when = datetime.fromtimestamp(t0 + span * frac, tz=timezone.utc) + anchor = "start" if frac == 0 else "middle" if frac == 0.5 else "end" + out.append( + f'{when.strftime("%b %Y")}' + ) + + out.append(f'') + out.append( + f'' + ) + + # One direct label — the current total. Sparing is what makes it readable. + end_x, end_y = coords[-1] + out.append(f'') + out.append( + f'{total:,}' + ) + out.append("") + return "\n".join(out) + "\n" + + +def main() -> None: + parser = argparse.ArgumentParser(description="Generate star-history SVGs for the README.") + parser.add_argument("--repo", default="andre-salvati/databricks-template", help="OWNER/NAME") + parser.add_argument("--out-dir", default="assets", help="directory to write the SVGs into") + args = parser.parse_args() + + stars = fetch_stars(args.repo) + out_dir = Path(args.out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + + for name, theme in THEMES.items(): + suffix = "" if name == "light" else "_dark" + path = out_dir / f"star_history{suffix}.svg" + path.write_text(build_svg(args.repo, stars, theme)) + print(f"wrote {path}") + + print(f"{len(stars)} stars — {stars[0]:%Y-%m-%d} to {stars[-1]:%Y-%m-%d}") + + +if __name__ == "__main__": + main() diff --git a/specs/CHANGELOG.md b/specs/CHANGELOG.md index 07084c2..7e25ccd 100644 --- a/specs/CHANGELOG.md +++ b/specs/CHANGELOG.md @@ -2,6 +2,12 @@ --- +## [#48](https://github.com/andre-salvati/databricks-template/pull/48) · 2026-07-16 · docs: local star-history chart, README refresh, and doc-consistency fixes + +Replaced the README star-history chart with `scripts/star_history.py` + `make star-history`, rendering committed light/dark SVGs from the GitHub API — `api.star-history.com` 503s for every repo, so it was never our markup. Fixed README step 8, which named the wrong CI secrets and silently broke CI auth for anyone forking. Also reconciled the specs with the code, de-duplicated CLAUDE.md, standardized the `#42`/`#43` headers, and widened the ruff lint gate to `scripts/`. + +--- + ## [#47](https://github.com/andre-salvati/databricks-template/pull/47) · 2026-07-16 · feat: weekly USD cost pivots + generated markdown cost report Reworked `scripts/project_costs.py` from per-day dumps into three tables — AWS by week × service, Databricks by week × SKU, and a combined by-service rollup carrying `Quantity`/`Unit`/`USD` — monetizing Databricks usage through a date-scoped `system.billing.list_prices` join on `pricing.effective_list.default`, which shows Databricks is ~99.5% of project spend ($33.91 of $34.06 at list over 30 days) and that AWS is only ever a proxy for job activity since serverless SKUs bill compute inside the DBU rate. Two properties of that join are load-bearing and easy to get wrong: it must be date-scoped because `list_prices` is a slowly-changing dimension (joining on `sku_name` alone fans out 3.5× on `JOBS_SERVERLESS_COMPUTE`), and it must be a LEFT join so an unpriced SKU shows blank rather than vanishing; per-day `ROUND()` was also dropped, since rounding then summing zeroed sub-cent SKUs like internet egress. The script now writes `cost_report/YYYY-MM-DD.md` — gitignored, with this first report force-added and linked from the README's Features list as a worked example — holding every table in markdown plus an `## Analysis` section that the `/project-costs` skill fills in, so numbers are never re-transcribed by hand. @@ -20,18 +26,19 @@ Added `specs/workflow.md` centralizing the development lifecycle (plan → ask-a --- -## [#43] · feat/freeze-product-name · 2026-06-23 · feat: freeze product_name instead of line_revenue +## [#43](https://github.com/andre-salvati/databricks-template/pull/43) · 2026-06-23 · feat: freeze product_name instead of line_revenue Removed the synthetic `line_revenue`/`unit_price_at_sale` columns — gold `total_value` is now `SUM(item_total)` (the line value the source already freezes on the order) — and re-pointed the silver insert-only-MERGE / streaming-table freeze at the mutable `product_name`, which the seed now changes by renaming 2 products per run (`Product N` → `Product N.k`); `unit_price` stays as a static attribute. The AI/BI "by product" chart and Product filter both identify a product by its latest name (consolidating by `product_id`, one line per physical product across renames, and filtering shows the full pre-/post-rename history); the frozen historical names remain in `report.order_agg` for audit. Also removed the batch `job1`'s standalone prod schedule (the SDP pipeline already had none) so `job1_prod_integration` is the single prod trigger orchestrating seed → batch + SDP, fixed `sdk_drop_tables.py` to fall back to `DROP TABLE` when a warehouse's parser rejects `DROP STREAMING TABLE`, added a `git_commit` (`${bundle.git.commit}`) deploy tag on every job so a deployed environment's exact commit is identifiable from `jobs list`, and updated unit/integration tests, schemas, the SDP pipeline, and docs. --- -## [#42] · docs/reorg-specs-tooling · 2026-06-23 · docs: rename docs/→assets/, consolidate tooling into specs/tooling.md, slim CLAUDE.md +## [#42](https://github.com/andre-salvati/databricks-template/pull/42) · 2026-06-23 · docs: rename docs/→assets/, consolidate tooling into specs/tooling.md, slim CLAUDE.md Renamed the image-only `docs/` folder to `assets/` (it held no prose, only screenshots and the CI/CD draw.io export) and repointed every `` reference in the README and specs. Added `specs/tooling.md` consolidating all four MCP servers (Databricks, aws-billing-cost, aws-documentation, context7), the Databricks CLI, and the bundled skills, then trimmed `CLAUDE.md`'s tooling section to a short every-session decision list that points at it. Moved the repo folder-structure tree out of `architecture.md` into the `specs/README.md` routing hub and flagged `CHANGELOG.md` there as append-only (don't read for context). --- + ## [#41](https://github.com/andre-salvati/databricks-template/pull/41) · 2026-06-12 · feat: raw cost DataFrames + by-service/SKU aggregation in project_costs.py `scripts/project_costs.py` now prints the raw per-day/service (AWS) and per-day/SKU (Databricks) DataFrames before each formatted daily table, plus an "AWS Costs by Service" rollup and a "Databricks Costs by SKU" rollup (quantity grouped by SKU and unit, since DBU/DSU/GB are not summable; NULL units kept, sorted within unit, AWS estimates flagged). Defaulted the `make project-costs` target to `--aws-profile costs` (the dedicated read-only Cost Explorer user), where `aws-profile=` now falls back to the default credential chain instead of hard-failing argparse. Also folded in code-review fixes: `scripts/sdk_drop_tables.py` now issues the kind-matched `DROP MATERIALIZED VIEW` / `DROP STREAMING TABLE` instead of a blind `DROP TABLE` (which aborted `make drop` mid-loop once the SDP pipeline had materialized its views/streaming tables), and `seed_sources.py` centralizes the product-category formula so the price-update path can't drift from the initial seed. diff --git a/specs/README.md b/specs/README.md index 273fd4e..07e9ddc 100644 --- a/specs/README.md +++ b/specs/README.md @@ -23,12 +23,16 @@ databricks-template/ │ ├── config.py # Config: catalogs/schemas, logging, DQX │ ├── baseTask.py # BaseTask (spark/config/logger/cluster_by) │ ├── commonSchemas.py # Canonical PySpark schemas -│ └── job1/ # extract_source1/2, generate_orders(_agg), -│ # health_check, seed_sources +│ ├── job1/ # extract_source1/2, generate_orders(_agg), +│ │ # health_check, seed_sources +│ ├── job1_sdp/ # SDP pipeline: pipeline.py (@dp defs) + transforms.py +│ └── job2/ # placeholder for a second job (.gitkeep) ├── tests/job1/ # unit_test, unit_test_sdp, integration_setup/validate ├── resources/ # jobs.yml (generated), orders_dashboard.lvdash.json (committed) ├── scripts/ # sdk_generate_template_job.py, sdk_init_workspace.py, -│ # sdk_drop_tables.py, project_costs.py, _sdk_sql.py +│ # sdk_drop_tables.py, sdk_analyze_job_costs.py, +│ # sdk_workspace_and_account.py, project_costs.py, +│ # star_history.py, _sdk_sql.py ├── specs/ # architecture / data-model / workflow / tooling (this folder) ├── assets/ # screenshots + diagrams referenced by README + specs ├── databricks.yml · pyproject.toml · Makefile · .pre-commit-config.yaml diff --git a/specs/architecture.md b/specs/architecture.md index df3c744..af8997f 100644 --- a/specs/architecture.md +++ b/specs/architecture.md @@ -22,11 +22,12 @@ does not expose custom env vars to the process). | Arg | Required | Purpose | |---|---|---| | `--task` | yes | Task key; in jobs, filled by `{{task.name}}`. | -| `--env` | yes | `dev` / `staging` / `prod` (or `local` for tests). | +| `--env` | yes | `dev` / `staging` / `prod`. Unit tests use `env=local`, but they build the args namespace directly — the parser itself rejects `local`. | | `--run-id` | no | Observability-only; filled by `{{job.run_id}}`, stamped on every log line. Defaults to `-`. | -| `--log-level` | no | `DEBUG`/`INFO`/`WARNING`, from job param `log_level` (default `INFO`). | +| `--log-level` | no | `DEBUG`/`INFO`/`WARN`/`WARNING`, from job param `log_level` (default `INFO`). | | `--quarantine-fail-ratio` | no | DQX hard-fail threshold for `extract_source2`. | | `--seed-date` | no | ISO-8601 date for `seed_sources`; empty → today at runtime. | +| `--load-test` | no | `"true"` activates load-test data volumes in the integration Setup/Validate tasks (staging-only). From job param `load_test` (default `"false"`); only the integration job passes it. | > Removed deliberately (PR #21) — do **not** reintroduce: `--user`, `--debug`, `--schema`. > Identity comes from `WorkspaceClient` at runtime with sanitization, not a CLI arg. @@ -91,18 +92,22 @@ triggers the batch `job1` (`RunJobTask`) and the `job1_sdp` pipeline (`PipelineT ## CI/CD -On every push (any branch), CI installs deps → unit tests → bundle validate → deploy to staging → -integration test on staging → (only on `main`) deploy to prod. Requires `DATABRICKS_HOST`, -`DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`, `TEMPLATE_ALERT_EMAILS` repo secrets. CLI and -action versions are pinned. Staging/prod deploy **and run as the service principal** -(identity-locked); the local dev loop runs as the developer against a per-developer catalog. +On every push, CI (`onpush.yml`) installs deps (`make sync`) → unit tests (`make unit-test`) → +uploads the coverage artifact → installs the pinned Databricks CLI → deploys to staging → +runs the integration test on staging → and, **only when `ref == main`**, deploys to prod. +Docs-only pushes trigger nothing: `paths-ignore` skips `README.md`, `CLAUDE.md`, `assets/**`, +and `specs/**`. There is **no `bundle validate` step** — `make deploy` regenerates +`resources/jobs.yml` and the bundle deploy itself is the validation gate. A `workflow_dispatch` +trigger allows re-running CI without a new commit. + +Requires `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`, and +`TEMPLATE_ALERT_EMAILS` repo secrets. CLI and action versions are pinned. CI/CD: local dev loop and the onpush.yml GitHub Actions pipeline Local steps **1–2** run as the **developer** against a per-developer `dev_` catalog; **3** -pushes / opens a PR, triggering **GitHub Actions** (`onpush.yml`) — which runs on every push and -deploys to prod only when `ref == main`. Staging and prod deploy **and run as the service +pushes / opens a PR, triggering GitHub Actions. Staging and prod deploy **and run as the service principal** — identity-locked. ## Job-level parameters (runtime, overridable per-run) @@ -146,7 +151,10 @@ Structured logging via the `template` logger (configured in `config.py:_configur warehouse before any medallion table is touched. - **Wheel version pinning** — `_project_version()` reads `pyproject.toml` to pin the wheel filename in `JobEnvironment.dependencies`, so a forgotten rebuild can't silently deploy an old wheel. -- **Retries** — 0 in dev, 2 in staging/prod, backing off `MIN_RETRY_INTERVAL_MS` (60s). +- **Retries** — 0 in dev, 2 in staging/prod, backing off `MIN_RETRY_INTERVAL_MS` (60s). Two + deliberate exceptions bypass `_retries()`: `health_check` pins `max_retries=1` in every env, and + the integration-test tasks pin `max_retries=0` even on staging (a failed assertion is a real + failure — retrying it just hides flakiness). - **Per-task `timeout_seconds`** (300s health-check, 900s extracts, 1800s transforms) so one hung task can't eat the whole job budget. - **Schema-drift guard** — all medallion writes use `overwriteSchema=false` (the only exception is diff --git a/specs/tooling.md b/specs/tooling.md index 6fd7ceb..5e2fa32 100644 --- a/specs/tooling.md +++ b/specs/tooling.md @@ -2,8 +2,10 @@ This project is developed with the [Databricks AI Dev Kit](https://github.com/databricks-solutions/ai-dev-kit) installed at the user level (`~/.ai-dev-kit/`). MCP servers and skills are **user-level tooling** — -`.mcp.json` and `.claude/` are gitignored and are **not** part of the repo. This doc records what's -wired up locally so a session knows what to reach for; it does not provision anything. +`.mcp.json` and `.claude/` are gitignored and are **not** part of the repo. The one exception is +`.claude/commands/`, which `.gitignore` un-ignores: project-specific slash commands (e.g. +`/project-costs`) are committed so every developer gets them. This doc records what's wired up +locally so a session knows what to reach for; it does not provision anything. `CLAUDE.md` carries a short decision list for every session; this is the full reference. diff --git a/src/template/job1_sdp/pipeline.py b/src/template/job1_sdp/pipeline.py index 13021ae..e5e97d9 100644 --- a/src/template/job1_sdp/pipeline.py +++ b/src/template/job1_sdp/pipeline.py @@ -12,7 +12,7 @@ external_source.order_item ──► raw.order_item_sdp (@dp.table STREAMING, append-only) raw.customer_sdp ─┐ (static dim) - raw.order_sdp ├──► curated.order_enriched_sdp (@dp.table STREAMING — freezes price) + raw.order_sdp ├──► curated.order_enriched_sdp (@dp.table STREAMING — freezes product_name) raw.product_sdp │ (static dim) raw.order_item_sdp ┘ (streaming fact)