diff --git a/datasets/README.md b/datasets/README.md index 392b76a..379c330 100644 --- a/datasets/README.md +++ b/datasets/README.md @@ -13,6 +13,7 @@ Each dataset simulates a real-world data pipeline with tables, views (for lineag | [olist-ecommerce](./olist-ecommerce/) | Multi-table e-commerce with join relationships | 9 tables + 5 cross-table join views | `olist.db` (clean) + `olist_dirty.db` (broken joins) | ~90MB each | | [nyc-taxi](./nyc-taxi/) | 3-stage data pipeline with freshness tracking | raw → staging → mart (linear) | `nyc_taxi.db` + `nyc_taxi_pipeline.db` | ~85MB each | | [healthcare](./healthcare/) | Forking pipeline with data quality issues | raw → staging → billing + demographics (fork) | `healthcare.db` (quality issues planted) | ~2MB | +| [fiction-retail](./fiction-retail/) | Synthetic global retail — orders, fulfillment, returns | 10 flat tables (no views) | `fiction-retail.db` | ~95MB | ### When to Use What @@ -28,6 +29,9 @@ Each dataset simulates a real-world data pipeline with tables, views (for lineag **Testing data reconciliation or cross-system consistency?** → Use **olist-ecommerce** (`olist.db` + `olist_warehouse.db` + `olist_analytics.db`). Same source data in three "systems" — each with different drift. Warehouse has missing rows and precision loss. Analytics has daily rollups with inflated counts and missing days. +**Exploring a flat, wide-table retail schema with no planted issues?** +→ Use **fiction-retail** (`fiction-retail.db`). 10 clean, interconnected tables covering customers, orders, products, suppliers, inventory, warehouses, shipments, returns, and promotions. Good baseline for schema exploration, join traversal, or building demos from scratch. + **Building something else entirely?** → Any of the above. Each dataset is a complete, realistic data environment. Pick the one whose schema and pipeline shape best fits your idea. @@ -77,6 +81,21 @@ NYC Yellow Taxi trip records from the [NYC TLC](https://www.kaggle.com/datasets/ **Metadata:** Tags (daily_refresh, time_series, pii, pipeline_stage), glossary (freshness_sla, empty_load, pipeline_stage), ownership (data_platform_team). +### fiction-retail + +Fully synthetic global retail dataset — 10 interconnected tables covering the complete order lifecycle across 50k customers, 150k orders, 5k products, 500 suppliers, and 15 warehouses. + +**Committed .db files:** +- `fiction-retail.db` — Single clean database. No planted issues. All foreign keys present in the data. + +**No additional variants.** This dataset has no ETL pipeline stages or views — it's a flat analytical schema intended as a clean foundation. + +**Key tables:** `customers`, `orders`, `order_items`, `products`, `suppliers`, `inventory`, `warehouses`, `shipments`, `returns`, `promotions` + +**Metadata:** Tags (pii, financial, transactional, reference_data), glossary (order_status, customer_segment, discount_pct, reorder_threshold, return_reason_code, shipment_state), ownership (customer_team, commerce_team, catalog_team, logistics_team, marketing_team, finance_team). + +**Source:** [Kaggle — Fiction Retail E-Commerce Dataset](https://www.kaggle.com/datasets/nasalakshay/fiction-retail-e-commerce-dataset) | License: CC0 1.0 (Public Domain) + ### healthcare Synthetic patient records from [Kaggle](https://www.kaggle.com/datasets/prasad22/healthcare-dataset) — ~55k records loaded as a forking pipeline with quality issues planted. @@ -133,10 +152,16 @@ datasets/ │ ├── ingest_pipeline.yaml │ ├── add_lineage.py │ └── add_metadata.py -└── healthcare/ +├── healthcare/ +│ ├── README.md +│ ├── create_db.py +│ ├── healthcare.db ← committed +│ ├── ingest.yaml +│ ├── add_lineage.py +│ └── add_metadata.py +└── fiction-retail/ ├── README.md - ├── create_db.py - ├── healthcare.db ← committed + ├── fiction-retail.db ← committed ├── ingest.yaml ├── add_lineage.py └── add_metadata.py @@ -150,9 +175,10 @@ These datasets are derived from publicly available sources. DataHub does not own | Dataset | Original Source | License | |---|---|---| -| Brazilian E-Commerce (Olist) | [Kaggle](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce) — Olist / André Sionek | CC BY-NC-SA 4.0 | +| Brazilian E-Commerce (Olist) | [Kaggle](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce) — Olist / André Sionek | [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) — derivative work, same license (see [LICENSE](./olist-ecommerce/LICENSE)) | | NYC Taxi Trip Records | [Kaggle](https://www.kaggle.com/datasets/elemento/nyc-yellow-taxi-trip-data) — NYC TLC | Public Domain | | Healthcare Dataset | [Kaggle](https://www.kaggle.com/datasets/prasad22/healthcare-dataset) — Prasad22 | CC0 1.0 (Public Domain) | +| Fiction Retail | [Kaggle](https://www.kaggle.com/datasets/nasalakshay/fiction-retail-e-commerce-dataset) — nasalakshay | CC0 1.0 (Public Domain) | ### What We Modified diff --git a/datasets/fiction-retail/README.md b/datasets/fiction-retail/README.md new file mode 100644 index 0000000..47c7090 --- /dev/null +++ b/datasets/fiction-retail/README.md @@ -0,0 +1,194 @@ +# Fiction Retail E-Commerce Dataset + +A synthetic e-commerce dataset simulating a mid-sized global retail operation. It covers the full order lifecycle — from customer acquisition through purchasing, fulfillment, shipment, and returns — across 10 interconnected tables. + +All data is entirely fictional and generated for analytical and educational use. + +**Source:** [Kaggle — Fiction Retail E-Commerce Dataset](https://www.kaggle.com/datasets/nasalakshay/fiction-retail-e-commerce-dataset) +**License:** [CC0 1.0 Public Domain](https://creativecommons.org/publicdomain/zero/1.0/) — no attribution required, free for any use + +--- + +## What's Included + +| File | Description | +|---|---| +| `fiction-retail.db` | Single SQLite database containing all 10 tables below (~95 MB) | +| `ingest.yaml` | DataHub ingestion recipe — loads all 10 tables into DataHub | +| `add_lineage.py` | Emits FK-based lineage between tables after ingestion | +| `add_metadata.py` | Adds tags, glossary terms, and ownership after ingestion | + +### Quick Start + +```bash +# Prerequisites +pip install 'acryl-datahub[sqlalchemy,datahub-rest]' +# DataHub must be running locally + +datahub ingest -c ingest.yaml +python add_lineage.py +python add_metadata.py +``` + +--- + +## Schema + +### `customers` — 50,000 rows + +| Column | Type | Description | +|---|---|---| +| `customer_id` | TEXT | Unique customer identifier | +| `name` | TEXT | Full name | +| `email` | TEXT | Email address | +| `phone` | TEXT | Phone number | +| `signup_date` | TEXT | Date the customer registered | +| `country` | TEXT | Country of residence | +| `state` | TEXT | State / province | +| `city` | TEXT | City | +| `customer_segment` | TEXT | Segment label (e.g. Wholesale, Retail) | + +--- + +### `orders` — 150,000 rows + +| Column | Type | Description | +|---|---|---| +| `order_id` | TEXT | Unique order identifier | +| `customer_id` | TEXT | FK → `customers.customer_id` | +| `order_date` | TEXT | Date the order was placed | +| `order_status` | TEXT | Status (e.g. Delivered, Cancelled, Returned) | +| `total_amount` | REAL | Total order value in USD | +| `payment_method` | TEXT | Payment method used | +| `shipping_country` | TEXT | Destination country | +| `promo_id` | TEXT | FK → `promotions.promo_id` (nullable) | + +--- + +### `order_items` — 346,202 rows + +| Column | Type | Description | +|---|---|---| +| `order_item_id` | TEXT | Unique line-item identifier | +| `order_id` | TEXT | FK → `orders.order_id` | +| `product_id` | TEXT | FK → `products.product_id` | +| `quantity` | INTEGER | Units ordered | +| `unit_price` | REAL | Price per unit at time of order | +| `discount_pct` | REAL | Discount percentage applied | + +--- + +### `products` — 5,000 rows + +| Column | Type | Description | +|---|---|---| +| `product_id` | TEXT | Unique product identifier | +| `name` | TEXT | Product name | +| `category` | TEXT | Product category | +| `brand` | TEXT | Brand name | +| `price` | REAL | List price in USD | +| `weight_kg` | REAL | Weight in kilograms | +| `supplier_id` | TEXT | FK → `suppliers.supplier_id` | + +--- + +### `suppliers` — 500 rows + +| Column | Type | Description | +|---|---|---| +| `supplier_id` | TEXT | Unique supplier identifier | +| `name` | TEXT | Supplier name | +| `country` | TEXT | Country of operation | +| `contract_start_date` | TEXT | Date the supplier contract began | +| `status` | TEXT | Active / Inactive | + +--- + +### `inventory` — 11,476 rows + +| Column | Type | Description | +|---|---|---| +| `inventory_id` | TEXT | Unique inventory record identifier | +| `product_id` | TEXT | FK → `products.product_id` | +| `warehouse_id` | TEXT | FK → `warehouses.warehouse_id` | +| `quantity_on_hand` | INTEGER | Current stock count | +| `reserved_quantity` | INTEGER | Units reserved for open orders | +| `reorder_threshold` | INTEGER | Stock level that triggers a reorder | +| `last_restocked_date` | TEXT | Date of most recent restock | + +--- + +### `warehouses` — 15 rows + +| Column | Type | Description | +|---|---|---| +| `warehouse_id` | TEXT | Unique warehouse identifier | +| `name` | TEXT | Warehouse name | +| `city` | TEXT | City | +| `state` | TEXT | State / province | +| `country` | TEXT | Country | +| `capacity_units` | INTEGER | Maximum storage capacity in units | +| `opened_date` | TEXT | Date the warehouse opened | + +--- + +### `shipments` — 119,936 rows + +| Column | Type | Description | +|---|---|---| +| `shipment_id` | TEXT | Unique shipment identifier | +| `order_id` | TEXT | FK → `orders.order_id` | +| `warehouse_id` | TEXT | FK → `warehouses.warehouse_id` | +| `carrier` | TEXT | Carrier name | +| `tracking_number` | TEXT | Carrier tracking number | +| `shipped_date` | TEXT | Date shipped | +| `delivered_date` | TEXT | Date delivered (nullable if in transit) | +| `shipment_state` | TEXT | State of shipment (e.g. Delivered, In Transit) | + +--- + +### `returns` — 11,934 rows + +| Column | Type | Description | +|---|---|---| +| `return_id` | TEXT | Unique return identifier | +| `order_id` | TEXT | FK → `orders.order_id` | +| `product_id` | TEXT | FK → `products.product_id` | +| `return_date` | TEXT | Date the return was initiated | +| `refund_amount` | REAL | Refund value in USD | +| `return_reason_code` | TEXT | Reason code (e.g. Defective, Wrong Item) | +| `processed_by` | TEXT | Agent or system that processed the return | + +--- + +### `promotions` — 200 rows + +| Column | Type | Description | +|---|---|---| +| `promo_id` | TEXT | Unique promotion identifier | +| `promo_code` | TEXT | Alphanumeric promo code | +| `description` | TEXT | Human-readable description | +| `discount_pct` | REAL | Discount percentage | +| `valid_from` | TEXT | Promotion start date | +| `valid_until` | TEXT | Promotion end date | +| `applies_to_category` | TEXT | Product category the promo applies to (nullable = all) | +| `max_uses` | INTEGER | Maximum redemption count | +| `status` | TEXT | Active / Expired | + +--- + +## Key Relationships + +``` +customers ──< orders ──< order_items >── products >── suppliers + │ │ + │ inventory >── warehouses + │ + shipments >── warehouses + │ + returns + │ + promotions (via orders.promo_id) +``` + +All foreign keys are present in the data but not enforced at the SQLite level (no `FOREIGN KEY` constraints), consistent with a typical analytical dataset. diff --git a/datasets/fiction-retail/add_lineage.py b/datasets/fiction-retail/add_lineage.py new file mode 100644 index 0000000..6a37108 --- /dev/null +++ b/datasets/fiction-retail/add_lineage.py @@ -0,0 +1,150 @@ +#!/usr/bin/env python3 +""" +Add lineage for the fiction-retail dataset in DataHub. + +Run AFTER ingestion: + datahub ingest -c ingest.yaml + python add_lineage.py + +Supports: --dry-run + +This dataset has no ETL pipeline stages. Lineage here reflects the natural +data relationships in a retail operation — which tables depend on which +reference/dimension tables to be meaningful. For example, order_items is only +interpretable in combination with orders and products; shipments depend on +orders and warehouses. +""" + +import sys + +from datahub.emitter.mcp import MetadataChangeProposalWrapper +from datahub.emitter.rest_emitter import DatahubRestEmitter +from datahub.ingestion.graph.client import DataHubGraph, DatahubClientConfig +from datahub.metadata.schema_classes import ( + DatasetLineageTypeClass, + UpstreamClass, + UpstreamLineageClass, +) + +DATAHUB_SERVER = "http://localhost:8080" +PLATFORM = "sqlite" +DEFAULT_INSTANCE = "fiction-retail" + +# Lineage: downstream → [upstreams] +# Reflects which tables a given table draws meaning from (FK dependencies). +TABLE_LINEAGE = { + # orders reference customers and promotions + "orders": ["customers", "promotions"], + # order_items are line items that join orders and products + "order_items": ["orders", "products"], + # products are supplied by suppliers + "products": ["suppliers"], + # inventory tracks product stock levels per warehouse + "inventory": ["products", "warehouses"], + # shipments fulfil orders and ship from a warehouse + "shipments": ["orders", "warehouses"], + # returns reference a completed order and the specific product returned + "returns": ["orders", "products"], +} + + +def discover_urns(graph, platform_instance): + query = f""" + {{ search(input: {{type: DATASET, query: "{platform_instance}", start: 0, count: 100}}) {{ + searchResults {{ entity {{ urn ... on Dataset {{ name platform {{ name }} }} }} }} + }} }} + """ + result = graph.execute_graphql(query) + urn_map = {} + for item in result.get("search", {}).get("searchResults", []): + entity = item.get("entity", {}) + if entity.get("platform", {}).get("name", "") != PLATFORM: + continue + if f",{platform_instance}." not in entity.get("urn", ""): + continue + name = entity.get("name", "") + simple = name.split(".")[-1] if "." in name else name + urn_map[simple] = entity["urn"] + return urn_map + + +def emit_lineage(emitter, urn_map, dry_run=False): + count = 0 + for downstream, upstream_names in TABLE_LINEAGE.items(): + if downstream not in urn_map: + print(f" ✗ {downstream}: not found in DataHub (run ingestion first)") + continue + + upstreams = [urn_map[t] for t in upstream_names if t in urn_map] + missing = [t for t in upstream_names if t not in urn_map] + if missing: + print(f" ⚠ {downstream}: upstream(s) not found — {', '.join(missing)}") + if not upstreams: + continue + + if dry_run: + print(f" → {downstream} ← {', '.join(upstream_names)}") + count += 1 + continue + + emitter.emit(MetadataChangeProposalWrapper( + entityUrn=urn_map[downstream], + aspect=UpstreamLineageClass( + upstreams=[ + UpstreamClass(dataset=u, type=DatasetLineageTypeClass.TRANSFORMED) + for u in upstreams + ] + ), + )) + print(f" ✓ {downstream} ← {', '.join(upstream_names)}") + count += 1 + + return count + + +def main(): + dry_run = False + + for arg in sys.argv[1:]: + if arg == "--dry-run": + dry_run = True + elif arg == "--help": + print("Usage: python add_lineage.py [--dry-run]") + return + else: + print(f"Unknown flag: {arg}") + sys.exit(1) + + print(f"Connecting to DataHub at {DATAHUB_SERVER}...") + try: + graph = DataHubGraph(DatahubClientConfig(server=DATAHUB_SERVER)) + except Exception as e: + print(f" ✗ Cannot connect: {e}") + sys.exit(1) + + emitter = DatahubRestEmitter(DATAHUB_SERVER) + + print(f"\n Instance: {DEFAULT_INSTANCE}") + urn_map = discover_urns(graph, DEFAULT_INSTANCE) + if not urn_map: + print(" No datasets found — run ingestion first.") + sys.exit(1) + + print(f" Found {len(urn_map)} datasets: {', '.join(sorted(urn_map))}") + count = emit_lineage(emitter, urn_map, dry_run=dry_run) + + print(f"\n{'='*50}") + if dry_run: + print(f"DRY RUN — {count} lineage edges would be created") + else: + print(f"✅ Lineage added: {count} relationships") + print(f" customers → orders → order_items") + print(f" products → order_items, inventory, returns") + print(f" suppliers → products") + print(f" warehouses → inventory, shipments") + print(f" promotions → orders") + print(f"{'='*50}") + + +if __name__ == "__main__": + main() diff --git a/datasets/fiction-retail/add_metadata.py b/datasets/fiction-retail/add_metadata.py new file mode 100644 index 0000000..bfd70a2 --- /dev/null +++ b/datasets/fiction-retail/add_metadata.py @@ -0,0 +1,268 @@ +#!/usr/bin/env python3 +""" +Add metadata (tags, glossary terms, ownership) to the fiction-retail dataset in DataHub. + +Run AFTER ingestion + lineage: + datahub ingest -c ingest.yaml + python add_lineage.py + python add_metadata.py + +Supports: --dry-run +""" + +import sys +import time + +from datahub.emitter.mcp import MetadataChangeProposalWrapper +from datahub.emitter.rest_emitter import DatahubRestEmitter +from datahub.ingestion.graph.client import DataHubGraph, DatahubClientConfig +from datahub.metadata.schema_classes import ( + AuditStampClass, + GlobalTagsClass, + GlossaryTermAssociationClass, + GlossaryTermInfoClass, + GlossaryTermsClass, + OwnerClass, + OwnershipClass, + OwnershipTypeClass, + TagAssociationClass, + TagPropertiesClass, +) + +DATAHUB_SERVER = "http://localhost:8080" +PLATFORM = "sqlite" +DEFAULT_INSTANCE = "fiction-retail" + +# ─── Tags ─────────────────────────────────────────────────────────────────── + +TAG_DEFINITIONS = { + "pii": { + "description": "Contains personally identifiable information (customer names, emails, phone numbers, location data).", + }, + "financial": { + "description": "Contains monetary values (order totals, unit prices, discounts, refunds). Treat with appropriate access controls.", + }, + "transactional": { + "description": "Records individual business transactions (orders, purchases, shipments, returns). High row volume, append-heavy.", + }, + "reference_data": { + "description": "Slowly changing lookup data (products, suppliers, warehouses, promotions). Low row count, read-heavy.", + }, +} + +TAG_ASSIGNMENTS = { + "pii": ["customers"], + "financial": ["orders", "order_items", "returns"], + "transactional": ["orders", "order_items", "shipments", "returns"], + "reference_data": ["products", "suppliers", "warehouses", "promotions"], +} + +# ─── Glossary ──────────────────────────────────────────────────────────────── + +GLOSSARY_DEFINITIONS = { + "order_status": { + "name": "Order Status", + "definition": "Lifecycle state of an order. Typical values: Pending, Processing, Shipped, Delivered, Cancelled, Returned. Used to filter active vs completed vs failed orders in reporting.", + }, + "customer_segment": { + "name": "Customer Segment", + "definition": "Classification of a customer by purchasing behaviour or account type (e.g. Retail, Wholesale, VIP). Used to personalise pricing, promotions, and reporting cohorts.", + }, + "discount_pct": { + "name": "Discount Percentage", + "definition": "Percentage reduction applied to a line item's unit price at time of purchase. Ranges from 0.0 (no discount) to 1.0 (100% off). Used to calculate net revenue and measure promotional effectiveness.", + }, + "reorder_threshold": { + "name": "Reorder Threshold", + "definition": "The quantity_on_hand level at which a restock should be triggered for a given product in a given warehouse. When quantity_on_hand drops at or below this value, a purchase order should be raised.", + }, + "return_reason_code": { + "name": "Return Reason Code", + "definition": "Standardised code describing why a product was returned (e.g. Defective, Wrong Item, Changed Mind, Damaged in Transit). Used to categorise return volume and identify quality or fulfilment issues.", + }, + "shipment_state": { + "name": "Shipment State", + "definition": "Current status of a shipment in the carrier's pipeline (e.g. Label Created, In Transit, Out for Delivery, Delivered, Failed). Distinct from order_status — an order can be Delivered while its shipment shows a carrier exception.", + }, +} + +GLOSSARY_ASSIGNMENTS = { + "order_status": ["orders"], + "customer_segment": ["customers"], + "discount_pct": ["order_items"], + "reorder_threshold": ["inventory"], + "return_reason_code": ["returns"], + "shipment_state": ["shipments"], +} + +# ─── Ownership ─────────────────────────────────────────────────────────────── + +OWNERSHIP_ASSIGNMENTS = { + "customer_team": ["customers"], + "commerce_team": ["orders", "order_items"], + "catalog_team": ["products", "suppliers"], + "logistics_team": ["shipments", "warehouses", "inventory"], + "marketing_team": ["promotions"], + "finance_team": ["returns"], +} + + +# ─── URN discovery ─────────────────────────────────────────────────────────── + +def discover_urns(graph, platform_instance): + query = f""" + {{ search(input: {{type: DATASET, query: "{platform_instance}", start: 0, count: 100}}) {{ + searchResults {{ entity {{ urn ... on Dataset {{ name platform {{ name }} }} }} }} + }} }} + """ + result = graph.execute_graphql(query) + urn_map = {} + for item in result.get("search", {}).get("searchResults", []): + entity = item.get("entity", {}) + if entity.get("platform", {}).get("name", "") != PLATFORM: + continue + if f",{platform_instance}." not in entity.get("urn", ""): + continue + name = entity.get("name", "") + simple = name.split(".")[-1] if "." in name else name + urn_map[simple] = entity["urn"] + return urn_map + + +# ─── Emission (batch pattern: collect all per table, emit once) ─────────────── + +def create_tags(emitter): + for name, info in TAG_DEFINITIONS.items(): + emitter.emit(MetadataChangeProposalWrapper( + entityUrn=f"urn:li:tag:{name}", + aspect=TagPropertiesClass(name=name, description=info["description"]), + )) + print(f" ✓ Created tag: {name}") + + +def attach_tags(emitter, urn_map): + table_tags = {} + for tag, tables in TAG_ASSIGNMENTS.items(): + for t in tables: + if t not in urn_map: + continue + table_tags.setdefault(t, []).append(f"urn:li:tag:{tag}") + + count = 0 + for table, tag_urns in table_tags.items(): + emitter.emit(MetadataChangeProposalWrapper( + entityUrn=urn_map[table], + aspect=GlobalTagsClass(tags=[TagAssociationClass(tag=u) for u in tag_urns]), + )) + names = [u.split(":")[-1] for u in tag_urns] + print(f" ✓ {table} ← tags: {', '.join(names)}") + count += 1 + return count + + +def create_glossary(emitter): + for key, info in GLOSSARY_DEFINITIONS.items(): + emitter.emit(MetadataChangeProposalWrapper( + entityUrn=f"urn:li:glossaryTerm:{key}", + aspect=GlossaryTermInfoClass( + name=info["name"], + definition=info["definition"], + termSource="INTERNAL", + ), + )) + print(f" ✓ Created term: {info['name']}") + + +def attach_glossary(emitter, urn_map): + now_ms = int(time.time() * 1000) + table_terms = {} + for key, tables in GLOSSARY_ASSIGNMENTS.items(): + for t in tables: + if t not in urn_map: + continue + table_terms.setdefault(t, []).append(f"urn:li:glossaryTerm:{key}") + + count = 0 + for table, term_urns in table_terms.items(): + emitter.emit(MetadataChangeProposalWrapper( + entityUrn=urn_map[table], + aspect=GlossaryTermsClass( + terms=[GlossaryTermAssociationClass(urn=u) for u in term_urns], + auditStamp=AuditStampClass(time=now_ms, actor="urn:li:corpuser:datahub"), + ), + )) + names = [u.split(":")[-1] for u in term_urns] + print(f" ✓ {table} ← terms: {', '.join(names)}") + count += 1 + return count + + +def emit_ownership(emitter, urn_map): + count = 0 + for owner, tables in OWNERSHIP_ASSIGNMENTS.items(): + for t in tables: + if t not in urn_map: + continue + emitter.emit(MetadataChangeProposalWrapper( + entityUrn=urn_map[t], + aspect=OwnershipClass(owners=[ + OwnerClass( + owner=f"urn:li:corpGroup:{owner}", + type=OwnershipTypeClass.DATAOWNER, + ) + ]), + )) + print(f" ✓ owner:{owner} → {t}") + count += 1 + return count + + +def main(): + dry_run = "--dry-run" in sys.argv + + print(f"Connecting to DataHub at {DATAHUB_SERVER}...") + try: + graph = DataHubGraph(DatahubClientConfig(server=DATAHUB_SERVER)) + except Exception as e: + print(f" ✗ Cannot connect: {e}") + sys.exit(1) + + emitter = DatahubRestEmitter(DATAHUB_SERVER) + + urn_map = discover_urns(graph, DEFAULT_INSTANCE) + if not urn_map: + print(f" No datasets found for '{DEFAULT_INSTANCE}'. Run ingestion first.") + sys.exit(1) + + print(f" Found {len(urn_map)} datasets: {', '.join(sorted(urn_map))}") + + if dry_run: + print("\n Tags to create:", list(TAG_DEFINITIONS)) + print(" Glossary terms to create:", [v["name"] for v in GLOSSARY_DEFINITIONS.values()]) + print(" Owners:", list(OWNERSHIP_ASSIGNMENTS)) + return + + print(f"\n Creating tags...") + create_tags(emitter) + print(f"\n Creating glossary terms...") + create_glossary(emitter) + + print(f"\n Attaching tags...") + attach_tags(emitter, urn_map) + print(f"\n Attaching glossary terms...") + attach_glossary(emitter, urn_map) + print(f"\n Adding ownership...") + emit_ownership(emitter, urn_map) + + print(f"\n{'='*50}") + print(f"✅ Metadata complete") + print(f" Tags: pii, financial, transactional, reference_data") + print(f" Glossary: Order Status, Customer Segment, Discount Percentage,") + print(f" Reorder Threshold, Return Reason Code, Shipment State") + print(f" Owners: customer_team, commerce_team, catalog_team,") + print(f" logistics_team, marketing_team, finance_team") + print(f"{'='*50}") + + +if __name__ == "__main__": + main() diff --git a/datasets/fiction-retail/fiction-retail.db b/datasets/fiction-retail/fiction-retail.db new file mode 100644 index 0000000..ef80f7b Binary files /dev/null and b/datasets/fiction-retail/fiction-retail.db differ diff --git a/datasets/fiction-retail/ingest.yaml b/datasets/fiction-retail/ingest.yaml new file mode 100644 index 0000000..0c52256 --- /dev/null +++ b/datasets/fiction-retail/ingest.yaml @@ -0,0 +1,24 @@ +# DataHub ingestion recipe for fiction-retail.db +# +# Usage: datahub ingest -c ingest.yaml +# +# Creates: fiction-retail platform instance in DataHub +# Schema: 10 flat tables — customers, orders, order_items, products, +# suppliers, inventory, warehouses, shipments, returns, promotions +# No views or pipeline stages — run add_metadata.py after ingestion. + +source: + type: sqlalchemy + config: + connect_uri: "sqlite:///fiction-retail.db" + platform: sqlite + platform_instance: fiction-retail + include_tables: true + include_views: false + include_view_lineage: false + include_view_column_lineage: false + +sink: + type: datahub-rest + config: + server: "http://localhost:8080" diff --git a/datasets/olist-ecommerce/LICENSE b/datasets/olist-ecommerce/LICENSE new file mode 100644 index 0000000..2490735 --- /dev/null +++ b/datasets/olist-ecommerce/LICENSE @@ -0,0 +1,23 @@ +Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) + +Original dataset: Brazilian E-Commerce Public Dataset by Olist +Source: https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce +Authors: Olist / André Sionek +Original license: CC BY-NC-SA 4.0 — https://creativecommons.org/licenses/by-nc-sa/4.0/ + +This folder contains a derivative work of the above dataset. In accordance with the +ShareAlike clause of CC BY-NC-SA 4.0, this derivative work is distributed under the +same license: CC BY-NC-SA 4.0. + +Modifications made to the original data: + - SQL views added to define transformation logic and enable lineage visualization + - Variant databases derived from the source (olist_dirty.db, olist_warehouse.db, + olist_analytics.db) with planted data quality issues, ETL drift, and rollup schemas + - Ingestion recipes (YAML) for loading into a metadata platform + - Metadata scripts for attaching tags, glossary terms, and ownership + - Geolocation table subsetted to 500k rows for size manageability + +The original data has been anonymized by Olist — company and partner names in reviews +are replaced with Game of Thrones house names. + +Full license text: https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode