Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,17 @@ jobs:
with:
enable-cache: true

- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: "20"

- name: Install pnpm
run: npm install -g pnpm

- name: Build frontend
run: cd frontend && pnpm install --frozen-lockfile && pnpm build

- name: Build package
run: uv build

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ data/
node_modules/
frontend/dist/
frontend/.vite/
backend/src/analytics_agent/static/

# IDE
.idea/
Expand Down
24 changes: 12 additions & 12 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,23 @@ Analytics Agent is a LangGraph-based chat agent that uses **DataHub** tools for

## Running the stack

A `justfile` at the repo root covers all common tasks. Install `just` once (`brew install just`), then:
A `Makefile` at the repo root covers all common tasks (`make` is available everywhere — no install needed):

```bash
just install # uv sync + pnpm install
just start # build frontend if stale, start backend at :8100
just port=8102 start # same on a custom port
just stop # kill the backend
just nuke # wipe the DB (start from scratch / re-trigger wizard)
just start-remote # start + print DataHub connection status
just logs # tail /tmp/analytics_agent.log
just test # unit tests
just build # force frontend rebuild
make install # uv sync + pnpm install
make start # build frontend if stale, start backend at :8100
make PORT=8102 start # same on a custom port
make stop # kill the backend
make nuke # wipe the DB (start from scratch / re-trigger wizard)
make start-remote # start + print DataHub connection status
make logs # tail /tmp/analytics_agent.log
make test # unit tests
make build # force frontend rebuild
```

`just start` automatically detects whether `frontend/src` is newer than `frontend/dist` and rebuilds only when needed.
`make start` uses Make's native dependency tracking: it rebuilds the frontend only when `frontend/src` files are newer than `frontend/dist/index.html`.

### Without just (manual)
### Without make (manual)

```bash
uv sync
Expand Down
130 changes: 130 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
PORT := 8100
DEV_PORT := 8101
LOG := /tmp/analytics_agent.log

# Load .env if present (same behaviour as just's set dotenv-load)
-include .env
export

.DEFAULT_GOAL := help

.PHONY: help install build typecheck serve dev start frontend dev-full stop \
restart logs test test-integration test-e2e start-remote check lint fix nuke

# ── Help ───────────────────────────────────────────────────────────────────────

help:
@echo "Usage: make <target> [PORT=8100]"
@echo ""
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \
awk 'BEGIN {FS = ":.*?## "}; {printf " %-20s %s\n", $$1, $$2}'

# ── Build ──────────────────────────────────────────────────────────────────────

install: ## Install all dependencies (Python + Node)
uv sync
cd frontend && pnpm install

# Make's native dependency tracking replaces just's build-if-stale recipe.
# Re-runs pnpm build only when frontend/src files are newer than the bundle.
frontend/dist/index.html: $(shell find frontend/src -type f 2>/dev/null)
cd frontend && pnpm build

build: ## Force-rebuild the frontend
cd frontend && pnpm build

typecheck: ## Type-check frontend without building
cd frontend && pnpm tsc --noEmit

# ── Development ────────────────────────────────────────────────────────────────

serve: ## Start backend in foreground with auto-reload (blocking)
uv run analytics-agent bootstrap
uv run uvicorn analytics_agent.main:app --reload --port $(DEV_PORT)

dev: frontend/dist/index.html ## Build frontend if stale, start backend with auto-reload
uv run analytics-agent bootstrap
pkill -f "analytics_agent.main" || true
nohup uv run uvicorn analytics_agent.main:app --reload --port $(DEV_PORT) > $(LOG) 2>&1 &
sleep 3 && curl -s http://localhost:$(DEV_PORT)/api/engines | head -c 120
@echo "\n→ http://localhost:$(DEV_PORT) (logs: make logs)"

start: frontend/dist/index.html ## Build frontend if stale, start backend on PORT
uv run analytics-agent bootstrap
pkill -f "analytics_agent.main" || true
nohup uv run uvicorn analytics_agent.main:app --port $(PORT) > $(LOG) 2>&1 &
sleep 3 && curl -s http://localhost:$(PORT)/api/engines | head -c 120
@echo "\n→ http://localhost:$(PORT)"

frontend: ## Start Vite dev server with HMR (use alongside 'serve')
cd frontend && pnpm dev

dev-full: ## Start backend (reload) + Vite dev server in parallel
uv run analytics-agent bootstrap
pkill -f "analytics_agent.main" || true
nohup uv run uvicorn analytics_agent.main:app --reload --port $(DEV_PORT) > $(LOG) 2>&1 &
@echo "Backend → http://localhost:$(DEV_PORT)"
cd frontend && pnpm dev

stop: ## Kill the backend
pkill -f "analytics_agent.main" || true
@echo "stopped"

restart: build stop start ## Force-rebuild frontend and restart backend

logs: ## Tail backend logs
tail -f $(LOG)

# ── Testing ────────────────────────────────────────────────────────────────────

test: ## Run unit tests
uv run pytest tests/unit/ -v

test-integration: ## Run integration tests (needs credentials in .env)
uv run pytest tests/integration/ -v -s

test-e2e: ## Run Playwright e2e tests (real backend + mock MCP tools)
npx --prefix frontend playwright test --config tests/e2e/playwright.config.ts

# ── Quality ────────────────────────────────────────────────────────────────────

check: ## Quick syntax check of the backend
uv run python -c "import analytics_agent.main"

lint: ## Lint + format check (mirrors CI)
uv run ruff check backend/src tests
uv run ruff format --check backend/src tests

fix: ## Auto-fix lint and format issues
uv run ruff check --fix backend/src tests
uv run ruff format backend/src tests

# ── Remote / DataHub ──────────────────────────────────────────────────────────

start-remote: start ## Start agent pointed at a remote DataHub instance
@echo ""
@echo " ┌─────────────────────────────────────────────────────┐"
@echo " │ Analytics Agent — Remote DataHub status │"
@echo " └─────────────────────────────────────────────────────┘"
uv run python scripts/datahub_status.py $(PORT)
@echo ""
@echo " → http://localhost:$(PORT)"

# ── Maintenance ────────────────────────────────────────────────────────────────

nuke: stop ## Wipe local DB so the onboarding wizard reappears
@DB_URL="$${DATABASE_URL:-sqlite+aiosqlite:///./data/dev.db}"; \
if echo "$$DB_URL" | grep -q "^sqlite"; then \
DB_PATH=$$(echo "$$DB_URL" | sed 's|sqlite.*:///||; s|^\./||'); \
if [ -f "$$DB_PATH" ]; then \
rm "$$DB_PATH" && echo "✓ Deleted $$DB_PATH"; \
else \
echo " (no SQLite DB found at $$DB_PATH — already clean)"; \
fi; \
else \
echo "Non-SQLite DB detected ($$DB_URL)."; \
echo "To reset manually, drop and recreate the schema your DATABASE_URL points to."; \
fi
@echo ""
@echo "Database wiped. Run 'make start' to come back up fresh."
@echo "Tip: open http://localhost:$(PORT)/#setup to force the wizard on an existing session."
60 changes: 42 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,33 @@

## ⚡ Quickstart

### Option A — pip / uvx (recommended, no Docker needed)

```bash
# Install and launch — no git clone, no repo, no Docker
pip install datahub-analytics-agent
analytics-agent quickstart

# Or with uv (no virtualenv management):
uvx datahub-analytics-agent quickstart
```

The wizard asks for your LLM provider + API key, optionally connects a data source, then starts the agent at **http://localhost:8100**. Config and the database are stored in `~/.datahub/analytics-agent/`.

Re-running `analytics-agent quickstart` detects the existing config and offers to start, reconfigure, or cancel — so it doubles as the "just start the agent" command for repeat users.

**Other server commands:**

```bash
analytics-agent start # start from existing config (no wizard)
analytics-agent stop # stop the running server
analytics-agent status # show whether server is running + URL
analytics-agent logs # tail ~/.datahub/analytics-agent/logs/agent.log
analytics-agent config # open config dir in $EDITOR or print its path
```

### Option B — Docker + sample data (full demo)

> **Requires:** Docker, DataHub CLI (`pip install acryl-datahub`), `uv`, Python 3.11+

```bash
Expand All @@ -31,12 +58,7 @@ cd analytics-agent
bash quickstart.sh
```

No `.env` editing required. The script:
- Starts a local DataHub instance (or connects to an existing one)
- Loads the Olist e-commerce sample dataset + catalog metadata
- Builds and launches Analytics Agent at **http://localhost:8100**

Open the browser — a setup wizard walks you through naming your agent, picking a model (Anthropic, OpenAI, Google, or AWS Bedrock), and entering your API key. If you already have one of those keys exported in your shell, it's picked up automatically.
The script starts a local DataHub instance, loads the Olist e-commerce sample dataset and catalog metadata, then builds and launches Analytics Agent at **http://localhost:8100**. Postgres data is persisted to `~/.datahub/analytics-agent/postgres-data/` so it survives container restarts.

**Using AWS Bedrock?** Export `LLM_PROVIDER=bedrock` before running the script. The script will verify your AWS credentials and Bedrock access before starting the container, and mount `~/.aws` read-only so boto3 picks up your profiles and SSO cache automatically.

Expand Down Expand Up @@ -68,22 +90,24 @@ Open the browser — a setup wizard walks you through naming your agent, picking

---

## Manual setup (without quickstart.sh)
## Manual setup (for contributors / development)

> This section is for hacking on the agent itself. For everyday use, `analytics-agent quickstart` is simpler.

> **Manual setup also requires:** `node` and `just` (`brew install node just`)
> **Also requires:** `node`

### 1. Clone and install

```bash
git clone https://github.com/datahub-project/analytics-agent.git
cd analytics-agent
just install # uv sync + pnpm install
just start # builds frontend, starts backend at :8100
make install # uv sync + pnpm install
make start # builds frontend, starts backend at :8100
```

Open **http://localhost:8100** — a setup wizard handles the LLM key and connections on first run.

> **Without `just`:** `uv sync && cd frontend && pnpm install && pnpm build && cd .. && uv run uvicorn analytics_agent.main:app --port 8100`
> **Without `make`:** `uv sync && cd frontend && pnpm install && pnpm build && cd .. && uv run uvicorn analytics_agent.main:app --port 8100`

### First-time setup

Expand Down Expand Up @@ -113,15 +137,15 @@ DATAHUB_GMS_URL=https://your-instance.acryl.io/gms
DATAHUB_GMS_TOKEN=eyJhbGci...
```

### Useful just tasks
### Useful make targets

| Command | What it does |
|---|---|
| `just start` | Build frontend if stale, start backend |
| `just start-remote` | Start + show DataHub connection status |
| `just nuke` | Wipe the DB and start from scratch |
| `just dev` | Hot-reload backend (use `just dev-full` for frontend HMR too) |
| `just logs` | Tail backend logs |
| `make start` | Build frontend if stale, start backend |
| `make start-remote` | Start + show DataHub connection status |
| `make nuke` | Wipe the DB and start from scratch |
| `make dev` | Hot-reload backend (use `make dev-full` for frontend HMR too) |
| `make logs` | Tail backend logs |

### Development mode (hot reload)

Expand Down Expand Up @@ -283,7 +307,7 @@ LLM_MODEL=us.anthropic.claude-sonnet-4-5-20250929-v1:0

## Database

The quickstart uses the DataHub MySQL container. For non-quickstart runs, SQLite is the default (`./data/dev.db`). Set `DATABASE_URL` in `.env` to switch backends — see `.env.example` for Postgres and SQLite formats.
The `analytics-agent quickstart` path uses SQLite at `~/.datahub/analytics-agent/data/agent.db`. The Docker quickstart uses Postgres, with data persisted to `~/.datahub/analytics-agent/postgres-data/`. For dev/Helm deployments, set `DATABASE_URL` explicitly — see `.env.example` for Postgres and SQLite formats.

---

Expand Down
13 changes: 12 additions & 1 deletion backend/src/analytics_agent/bootstrap.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,18 @@ def run_migrations() -> None:
elif "mysql" in settings.database_url:
_ensure_mysql_schema()

alembic_cfg = Config("alembic.ini")
# Locate migration scripts via package path so this works when pip-installed
# (no alembic.ini on disk). Fall back to alembic.ini in CWD for the dev/Docker
# workflow where the repo is checked out and alembic CLI is also used.
_ini = Path("alembic.ini")
if _ini.exists():
alembic_cfg = Config(str(_ini))
else:
alembic_cfg = Config()
alembic_cfg.set_main_option(
"script_location",
str(Path(__file__).parent / "db" / "alembic"),
)
command.upgrade(alembic_cfg, "head")


Expand Down
Loading
Loading