Shopify ETL Runner

A CLI-based runner for Shopify ETL pipelines, coordinating multi-client data fetches and reports (inventory, locations, orders, purchase orders) using Mongo-backed configuration and Shopify's GraphQL API.

This repository is a cleaned-up extraction from a larger production system. It's published as-is for reference and reuse, not as a general-purpose ETL framework or drop-in product.

Note: CI is being cleaned up after extraction; some checks may be temporarily noisy.

Quick start

1. Create a virtualenv, then install: pip install . (or pip install -e .[dev] for lint/test tools).

2. Copy .env.example to .env and set your Mongo URI / API version / output dirs as needed.

3. Bootstrap Mongo (AdminDB + client DB):

   python -m shopify_etl_runner.bootstrap \
     --mongo-uri "mongodb://localhost:27017" \
     --client-db "<client_db>" \
     --client-code "<client_code>" \
     --shop-name "<store>" \
     --access-token "<token>" \
     [--stocky-api-key ...]

   See docs/config.md for expected collections/fields.

4. Run the runner:

   • Single step:

     python -m shopify_etl_runner.runner --client <client_db> \
       --step fetch_inventory

   • Grouped:

     python -m shopify_etl_runner.runner --client <client_db> --fetches
     # or
     python -m shopify_etl_runner.runner --client <client_db> --reports

   • All clients:

     python -m shopify_etl_runner.runner --all-clients --full

   • More examples: runner_commands.md.

Project layout

• runner.py - CLI entrypoint and orchestration logic.
• bootstrap.py - helper to seed AdminDB/client DB collections and configs.
• core/ - Shopify GraphQL client, Mongo helpers, logging, data transforms, report builders.
• pipelines/ - thin wrappers for individual pipeline steps used by the runner.
• docs/config.md - Mongo/config reference and bootstrap examples.

Configuration

• Environment variables: see .env.example (defaults in core/config.py).
• Config data lives in Mongo: AdminDB Clients and per-client Configurations (details in docs/config.md).
• Report outputs default to output/excel and output/csv (gitignored); override via .env or config.

Development

• Lint: ruff check .
• Format: black .
• Types: mypy .
• Tests: pytest
• See CONTRIBUTING.md for more.

License

MIT - see LICENSE.