Skip to content

feat(datasets): add fiction-retail dataset #201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lakshay-nasa merged 1 commit into from
Jun 4, 2026
Merged

feat(datasets): add fiction-retail dataset #201

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 30 additions & 4 deletions datasets/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Each dataset simulates a real-world data pipeline with tables, views (for lineag
| [olist-ecommerce](./olist-ecommerce/) | Multi-table e-commerce with join relationships | 9 tables + 5 cross-table join views | `olist.db` (clean) + `olist_dirty.db` (broken joins) | ~90MB each |
| [nyc-taxi](./nyc-taxi/) | 3-stage data pipeline with freshness tracking | raw → staging → mart (linear) | `nyc_taxi.db` + `nyc_taxi_pipeline.db` | ~85MB each |
| [healthcare](./healthcare/) | Forking pipeline with data quality issues | raw → staging → billing + demographics (fork) | `healthcare.db` (quality issues planted) | ~2MB |
| [fiction-retail](./fiction-retail/) | Synthetic global retail — orders, fulfillment, returns | 10 flat tables (no views) | `fiction-retail.db` | ~95MB |

### When to Use What

Expand All @@ -28,6 +29,9 @@ Each dataset simulates a real-world data pipeline with tables, views (for lineag
**Testing data reconciliation or cross-system consistency?**
→ Use **olist-ecommerce** (`olist.db` + `olist_warehouse.db` + `olist_analytics.db`). Same source data in three "systems" — each with different drift. Warehouse has missing rows and precision loss. Analytics has daily rollups with inflated counts and missing days.

**Exploring a flat, wide-table retail schema with no planted issues?**
→ Use **fiction-retail** (`fiction-retail.db`). 10 clean, interconnected tables covering customers, orders, products, suppliers, inventory, warehouses, shipments, returns, and promotions. Good baseline for schema exploration, join traversal, or building demos from scratch.

**Building something else entirely?**
→ Any of the above. Each dataset is a complete, realistic data environment. Pick the one whose schema and pipeline shape best fits your idea.

Expand Down Expand Up @@ -77,6 +81,21 @@ NYC Yellow Taxi trip records from the [NYC TLC](https://www.kaggle.com/datasets/

**Metadata:** Tags (daily_refresh, time_series, pii, pipeline_stage), glossary (freshness_sla, empty_load, pipeline_stage), ownership (data_platform_team).

### fiction-retail

Fully synthetic global retail dataset — 10 interconnected tables covering the complete order lifecycle across 50k customers, 150k orders, 5k products, 500 suppliers, and 15 warehouses.

**Committed .db files:**
- `fiction-retail.db` — Single clean database. No planted issues. All foreign keys present in the data.

**No additional variants.** This dataset has no ETL pipeline stages or views — it's a flat analytical schema intended as a clean foundation.

**Key tables:** `customers`, `orders`, `order_items`, `products`, `suppliers`, `inventory`, `warehouses`, `shipments`, `returns`, `promotions`

**Metadata:** Tags (pii, financial, transactional, reference_data), glossary (order_status, customer_segment, discount_pct, reorder_threshold, return_reason_code, shipment_state), ownership (customer_team, commerce_team, catalog_team, logistics_team, marketing_team, finance_team).

**Source:** [Kaggle — Fiction Retail E-Commerce Dataset](https://www.kaggle.com/datasets/nasalakshay/fiction-retail-e-commerce-dataset) | License: CC0 1.0 (Public Domain)

### healthcare

Synthetic patient records from [Kaggle](https://www.kaggle.com/datasets/prasad22/healthcare-dataset) — ~55k records loaded as a forking pipeline with quality issues planted.
Expand Down Expand Up @@ -133,10 +152,16 @@ datasets/
│ ├── ingest_pipeline.yaml
│ ├── add_lineage.py
│ └── add_metadata.py
└── healthcare/
├── healthcare/
│ ├── README.md
│ ├── create_db.py
│ ├── healthcare.db ← committed
│ ├── ingest.yaml
│ ├── add_lineage.py
│ └── add_metadata.py
└── fiction-retail/
├── README.md
├── create_db.py
├── healthcare.db ← committed
├── fiction-retail.db ← committed
├── ingest.yaml
├── add_lineage.py
└── add_metadata.py
Expand All @@ -150,9 +175,10 @@ These datasets are derived from publicly available sources. DataHub does not own

| Dataset | Original Source | License |
|---|---|---|
| Brazilian E-Commerce (Olist) | [Kaggle](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce) — Olist / André Sionek | CC BY-NC-SA 4.0 |
| Brazilian E-Commerce (Olist) | [Kaggle](https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce) — Olist / André Sionek | [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) — derivative work, same license (see [LICENSE](./olist-ecommerce/LICENSE)) |
| NYC Taxi Trip Records | [Kaggle](https://www.kaggle.com/datasets/elemento/nyc-yellow-taxi-trip-data) — NYC TLC | Public Domain |
| Healthcare Dataset | [Kaggle](https://www.kaggle.com/datasets/prasad22/healthcare-dataset) — Prasad22 | CC0 1.0 (Public Domain) |
| Fiction Retail | [Kaggle](https://www.kaggle.com/datasets/nasalakshay/fiction-retail-e-commerce-dataset) — nasalakshay | CC0 1.0 (Public Domain) |

### What We Modified

Expand Down
194 changes: 194 additions & 0 deletions datasets/fiction-retail/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Fiction Retail E-Commerce Dataset

A synthetic e-commerce dataset simulating a mid-sized global retail operation. It covers the full order lifecycle — from customer acquisition through purchasing, fulfillment, shipment, and returns — across 10 interconnected tables.

All data is entirely fictional and generated for analytical and educational use.

**Source:** [Kaggle — Fiction Retail E-Commerce Dataset](https://www.kaggle.com/datasets/nasalakshay/fiction-retail-e-commerce-dataset)
**License:** [CC0 1.0 Public Domain](https://creativecommons.org/publicdomain/zero/1.0/) — no attribution required, free for any use

---

## What's Included

| File | Description |
|---|---|
| `fiction-retail.db` | Single SQLite database containing all 10 tables below (~95 MB) |
| `ingest.yaml` | DataHub ingestion recipe — loads all 10 tables into DataHub |
| `add_lineage.py` | Emits FK-based lineage between tables after ingestion |
| `add_metadata.py` | Adds tags, glossary terms, and ownership after ingestion |

### Quick Start

```bash
# Prerequisites
pip install 'acryl-datahub[sqlalchemy,datahub-rest]'
# DataHub must be running locally

datahub ingest -c ingest.yaml
python add_lineage.py
python add_metadata.py
```

---

## Schema

### `customers` — 50,000 rows

| Column | Type | Description |
|---|---|---|
| `customer_id` | TEXT | Unique customer identifier |
| `name` | TEXT | Full name |
| `email` | TEXT | Email address |
| `phone` | TEXT | Phone number |
| `signup_date` | TEXT | Date the customer registered |
| `country` | TEXT | Country of residence |
| `state` | TEXT | State / province |
| `city` | TEXT | City |
| `customer_segment` | TEXT | Segment label (e.g. Wholesale, Retail) |

---

### `orders` — 150,000 rows

| Column | Type | Description |
|---|---|---|
| `order_id` | TEXT | Unique order identifier |
| `customer_id` | TEXT | FK → `customers.customer_id` |
| `order_date` | TEXT | Date the order was placed |
| `order_status` | TEXT | Status (e.g. Delivered, Cancelled, Returned) |
| `total_amount` | REAL | Total order value in USD |
| `payment_method` | TEXT | Payment method used |
| `shipping_country` | TEXT | Destination country |
| `promo_id` | TEXT | FK → `promotions.promo_id` (nullable) |

---

### `order_items` — 346,202 rows

| Column | Type | Description |
|---|---|---|
| `order_item_id` | TEXT | Unique line-item identifier |
| `order_id` | TEXT | FK → `orders.order_id` |
| `product_id` | TEXT | FK → `products.product_id` |
| `quantity` | INTEGER | Units ordered |
| `unit_price` | REAL | Price per unit at time of order |
| `discount_pct` | REAL | Discount percentage applied |

---

### `products` — 5,000 rows

| Column | Type | Description |
|---|---|---|
| `product_id` | TEXT | Unique product identifier |
| `name` | TEXT | Product name |
| `category` | TEXT | Product category |
| `brand` | TEXT | Brand name |
| `price` | REAL | List price in USD |
| `weight_kg` | REAL | Weight in kilograms |
| `supplier_id` | TEXT | FK → `suppliers.supplier_id` |

---

### `suppliers` — 500 rows

| Column | Type | Description |
|---|---|---|
| `supplier_id` | TEXT | Unique supplier identifier |
| `name` | TEXT | Supplier name |
| `country` | TEXT | Country of operation |
| `contract_start_date` | TEXT | Date the supplier contract began |
| `status` | TEXT | Active / Inactive |

---

### `inventory` — 11,476 rows

| Column | Type | Description |
|---|---|---|
| `inventory_id` | TEXT | Unique inventory record identifier |
| `product_id` | TEXT | FK → `products.product_id` |
| `warehouse_id` | TEXT | FK → `warehouses.warehouse_id` |
| `quantity_on_hand` | INTEGER | Current stock count |
| `reserved_quantity` | INTEGER | Units reserved for open orders |
| `reorder_threshold` | INTEGER | Stock level that triggers a reorder |
| `last_restocked_date` | TEXT | Date of most recent restock |

---

### `warehouses` — 15 rows

| Column | Type | Description |
|---|---|---|
| `warehouse_id` | TEXT | Unique warehouse identifier |
| `name` | TEXT | Warehouse name |
| `city` | TEXT | City |
| `state` | TEXT | State / province |
| `country` | TEXT | Country |
| `capacity_units` | INTEGER | Maximum storage capacity in units |
| `opened_date` | TEXT | Date the warehouse opened |

---

### `shipments` — 119,936 rows

| Column | Type | Description |
|---|---|---|
| `shipment_id` | TEXT | Unique shipment identifier |
| `order_id` | TEXT | FK → `orders.order_id` |
| `warehouse_id` | TEXT | FK → `warehouses.warehouse_id` |
| `carrier` | TEXT | Carrier name |
| `tracking_number` | TEXT | Carrier tracking number |
| `shipped_date` | TEXT | Date shipped |
| `delivered_date` | TEXT | Date delivered (nullable if in transit) |
| `shipment_state` | TEXT | State of shipment (e.g. Delivered, In Transit) |

---

### `returns` — 11,934 rows

| Column | Type | Description |
|---|---|---|
| `return_id` | TEXT | Unique return identifier |
| `order_id` | TEXT | FK → `orders.order_id` |
| `product_id` | TEXT | FK → `products.product_id` |
| `return_date` | TEXT | Date the return was initiated |
| `refund_amount` | REAL | Refund value in USD |
| `return_reason_code` | TEXT | Reason code (e.g. Defective, Wrong Item) |
| `processed_by` | TEXT | Agent or system that processed the return |

---

### `promotions` — 200 rows

| Column | Type | Description |
|---|---|---|
| `promo_id` | TEXT | Unique promotion identifier |
| `promo_code` | TEXT | Alphanumeric promo code |
| `description` | TEXT | Human-readable description |
| `discount_pct` | REAL | Discount percentage |
| `valid_from` | TEXT | Promotion start date |
| `valid_until` | TEXT | Promotion end date |
| `applies_to_category` | TEXT | Product category the promo applies to (nullable = all) |
| `max_uses` | INTEGER | Maximum redemption count |
| `status` | TEXT | Active / Expired |

---

## Key Relationships

```
customers ──< orders ──< order_items >── products >── suppliers
│ │
│ inventory >── warehouses
│
shipments >── warehouses
│
returns
│
promotions (via orders.promo_id)
```

All foreign keys are present in the data but not enforced at the SQLite level (no `FOREIGN KEY` constraints), consistent with a typical analytical dataset.
Loading
Loading