feat(showcase-ecommerce): add 312 query entities for text-to-sql / anchor demo

[alexsku]

Summary

Adds 04-queries.json to the showcase-ecommerce datapack: 312 urn:li:query entities (1,246 MCPs total) plus the readable .sql source files they were built from. Until now the datapack had zero query entities, so the anchor-generation pipeline behind the text-to-SQL demo had no SQL to fingerprint and rank. This fills that gap.

Follows on from #195 (which added the context docs / talk-to-data scenarios for the same demo).

What's in here

• 04-queries.json — 312 query entities, each with queryProperties (SQL text + audit stamps), querySubjects (auto-detected from order_entry_db.* references), dataPlatformInstance (snowflake / b2fd91), and subTypes. Authors mapped to the 11 existing corpuser entities in 02-data.json. created timestamps spread across the data window so the corpus looks like real activity history.
• queries/ — readable source SQL alongside the build artifact:
  • 50 intent directories with a 00-intent.yaml (question, canonical definition, variant axes, data caveats) + 3–5 .sql variants each (160 files)
  • noise/ — 152 long-tail one-off queries (per-customer/order/product/promotion drill-downs, date-window investigations, DQ checks, schema exploration, abandoned half-finished queries)
  • intent-matrix.yaml — top-level index across all 50 intents
  • README.md — contributor guide, including data-realities gotchas (e.g. orders.order_status is NUMBER 0..4 not strings, mailshot is 0/1 not Y/N, customers.country_id isn't a real FK)
• index.json — one-line addition that lists 04-queries.json

Cluster vs. noise

The 160 cluster queries are organised into 50 intents (one per directory) — each intent has 3–5 variants that vary on join path (raw tables vs. analytics.order_details mart), filter idiom, CTE vs. subquery, persona writing style, and occasionally an intentional "wrong-but-equivalent" shape. These should collapse to a single anchor per intent.

The 152 noise queries are bespoke one-offs that should not cluster — they give the pipeline discrimination work and replicate real analyst-history distribution.

Demo-grade anomalies the corpus surfaces

A few real data inconsistencies that the queries pull into focus and that should make great demo moments:

• 4,028 / 5,027 promoted orders fall outside their promotion's start/end date window — all 50 promotions affected (orders_outside_promo_window)
• 1,250 In-Store Pickup orders carry a delivery_address_id they shouldn't (pickup_orders_missing_delivery_addr, per the Order Count Discrepancy runbook)
• 3 warehouses show negative avg days from order to dispatch — dispatch_date predates order_date (avg_dispatch_lead_by_warehouse)
• Negative discount rates — Hair Straightener at -73%, Vanilla Bean Ice Cream at -131% because unit_price > list_price in the data (discount_rate_by_promo)
• Order 9866 has order_total = \$210 but its line items sum to $1,000 (gap surfaced by an order_9866_header_vs_lines spot check)

Test plan

• datahub datapack load --dry-run against the local bundle — all 4 files parse, all MCPs schema-compatible (84 entity types / 726 aspects matched against current server schema)
• Every cluster-canonical executes cleanly against the real ORDER_ENTRY_DB on Snowflake during authoring; sampled noise queries also validated
• Test-load to a real DataHub instance (recommend trying contexthub before this lands)
• Anchor-generation pipeline run on the loaded corpus

🤖 Generated with Claude Code

[vercel]

@alexsku is attempting to deploy a commit to the Acryl Data Team on Vercel.

A member of the Team first needs to authorize it.

[alexsku]

@shirshanka — flagging for your review since you were driving the talk-to-data / context-docs work in #195 that this builds on. Happy to make changes.

[shirshanka]

Test-load finding — 04-queries.json rejects 1246/1246 MCPs ⚠️

Loaded the bundle locally via datahub datapack load showcase-ecommerce --url file://.../index.json --trust-custom --no-cache against a v1.1.0rc1 GMS. Files 1–3 loaded cleanly; 04-queries.json failed every MCP:

--- File 4/4: ... ---
Pipeline finished with at least 1246 failures; produced 2 events in 0.45 seconds.

Each failure was the same avro union-schema error:

Datum {'value': '{"platform":"urn:li:dataPlatform:snowflake","instance":"..."}'} cannot be parsed as union schema
  ["null", {"type": "record", "name": "GenericAspect", ... "fields": [
    {"type": "bytes", "name": "value", ...},
    {"type": "string", "name": "contentType", ...}
  ]}]

Root cause

04-queries.json MCPs have aspect: {value: "<json string>"} — missing contentType. Compare to 03-context.json (which loaded fine):

# 04-queries.json (broken)
"aspect": {"value": "{\"statement\":...}"}                    # 1 key

# 03-context.json (works)
"aspect": {"value": "{...}", "contentType": "application/json"}   # 2 keys

Without contentType, the bare-value dict can't unify against the GenericAspect record ({value: bytes, contentType: string}), so every MCP in the file is dropped.

Verification

Patched a local copy by adding "contentType": "application/json" to all 1246 aspects → reload produced 1,248 events with 0 failures. Spot-checked via GraphQL:

{ entity(urn: "urn:li:query:b2fd91.q-active_customers_90d-1-canonical_distinct") {
  ... on QueryEntity { properties { name statement { value language } } subjects { dataset { urn } } }
}}

→ returns full SQL text + correct querySubjects.dataset.urn (order_entry_db.order_entry.orders). The data itself is good; it's purely an envelope-shape issue.

Suggested fix in the generator

Wherever 04-queries.json is built, change:

"aspect": {"value": json.dumps(payload)}

to:

"aspect": {"value": json.dumps(payload), "contentType": "application/json"}

Probably worth a quick assert in whatever writes datapack JSON: assert "contentType" in mcp["aspect"].

The PR description says datahub datapack load --dry-run passed during authoring — worth a look at why; dry-run currently only checks aspectName against the entity schema (schema_compat), not the envelope shape, so this kind of issue slips past.