Aegis Deploy

An autonomous canary deployment control plane that shifts real traffic, evaluates release health, and restores the stable version automatically.

Open the live Zerops deployment · Watch the 46-second demo · Read the architecture

The pitch

Deployments fail at the worst possible moment. Aegis turns a release into a measured control loop: send a small percentage of traffic to a candidate, observe its real error rate and p95 latency, advance only while it is healthy, and atomically return traffic to stable when policy is violated.

The live dashboard is the product—not a static landing page. It can generate traffic, start a canary, inject deterministic faults, stream live decisions, and show the durable PostgreSQL audit trail after recovery.

Try the live demo

Live URL: https://controlplane-297d-3000.prg1.zerops.app

1. Click Reset demo to establish healthy faults and 100% stable traffic.
2. Enable the traffic generator.
3. Click Ship candidate.
4. Wait for Aegis to move from 10% to 25% candidate traffic.
5. Select Error spike.
6. Watch Aegis detect the threshold violation, roll back to 100% stable, and persist the reason and metric snapshot.

The 46-second production recording shows the complete canary rollout and automatic rollback. A WebVTT subtitle track is available for accessible playback. Five consecutive production rehearsals and their timings are documented in docs/demo-rehearsal.md.

The problem

A conventional deployment answers “did the process start?” A safe rollout must also answer:

• Is the new version serving real traffic successfully?
• Is it slower or more error-prone than the stable version?
• Do we have enough samples to make a decision?
• Can we restore known-good routing without relying on a human reaction?
• Will the reason still be available after services restart?

Aegis makes those questions explicit and auditable.

Architecture

flowchart LR
    Judge[Browser / Judge] -->|HTTPS, API, SSE| CP[controlplane]
    CP -->|weighted /demo traffic| Stable[appstable v1]
    CP -->|weighted /demo traffic| Canary[appcanary v2]
    CP -->|weights, metrics, events| Valkey[(state / Valkey)]
    CP -->|history queries| Postgres[(db / PostgreSQL)]
    Orchestrator[orchestrator worker] -->|read windows + locks| Valkey
    Orchestrator -->|health probe| Canary
    Orchestrator -->|audit transitions| Postgres
    Orchestrator -->|advance / promote / rollback| Valkey
    Valkey -->|pub/sub| CP

Loading

Only controlplane is public. The candidate, stable app, orchestrator, Valkey, and PostgreSQL services communicate through Zerops project-private hostnames. The React dashboard is built into and served by the control plane, keeping the public edge and deployment surface small.

See docs/architecture.md for component boundaries, state ownership, request paths, and recovery behavior.

Rollout control loop

The default policy is deliberately understandable:

Stage	Stable	Candidate	Observe
0	90%	10%	15 s
1	75%	25%	15 s
2	50%	50%	20 s
3	0%	100%	promote

Every two seconds, the orchestrator acquires a distributed lock, probes candidate health, loads the last 15 seconds of real proxy metrics, and evaluates the policy. It waits for at least 10 candidate samples. It rolls back when any of these are true:

• candidate error rate exceeds 15%;
• candidate p95 latency exceeds 800 ms;
• candidate health fails 3 consecutive probes.

Advance, promotion, and rollback use guarded Valkey scripts so stale workers cannot overwrite newer state. PostgreSQL transactions persist the matching audit events and rollback metric snapshot.

Zerops service mapping

Zerops service	Exposure	Responsibility
controlplane	Public HTTPS	Dashboard, API, weighted proxy, traffic generator, metrics, SSE
appstable	Private	Known-good demo app, version v1
appcanary	Private	Candidate v2 with Valkey-controlled fault injection
orchestrator	Private worker	Evaluation, distributed locking, rollout transitions
state	Private Valkey	Live weights, active deployment, metrics, locks, pub/sub
db	Private PostgreSQL	Deployment history, events, rollback snapshots

Zerops is essential to the running design: it supplies isolated Node.js runtimes, private service DNS, managed Valkey and PostgreSQL connection references, health checks, build/deploy pipelines, logs, and the single public TLS endpoint. The exact deployment and verified service IDs are recorded in docs/zerops-deployment.md.

Why Valkey and PostgreSQL?

They solve different consistency and lifetime problems:

• Valkey is the live control plane. Routing decisions need low-latency reads, expiring metric windows, a distributed worker lock, atomic compare-and- set transitions, and pub/sub for dashboard updates.
• PostgreSQL is the audit plane. Deployment status, stage transitions, reasons, thresholds, and rollback snapshots must remain queryable after metrics expire or processes restart.

Using PostgreSQL for every proxy request would add unnecessary latency. Using only Valkey would make historical evidence and recovery auditing fragile.

Run locally

Requirements: Node.js 22, npm, and Docker with Compose.

git clone https://github.com/kush1jpeg/aegis.git
cd aegis
npm ci
docker compose up -d
npm run migrate
npm run build

Start each process in its own terminal from the repository root:

# Stable v1
APP_VERSION=v1 FAULT_CONTROL_ENABLED=false PORT=3001 \
  node apps/demo-app/dist/server.js

# Candidate v2
APP_VERSION=v2 FAULT_CONTROL_ENABLED=true PORT=3002 \
VALKEY_URL=redis://127.0.0.1:6379 \
  node apps/demo-app/dist/server.js

# Orchestrator
PORT=3003 CANARY_URL=http://127.0.0.1:3002 \
VALKEY_URL=redis://127.0.0.1:6379 \
DATABASE_URL=postgresql://aegis:aegis_local@127.0.0.1:5432/aegis \
  node apps/orchestrator/dist/server.js

# Public control plane + built dashboard
PORT=3000 STABLE_URL=http://127.0.0.1:3001 \
CANARY_URL=http://127.0.0.1:3002 \
VALKEY_URL=redis://127.0.0.1:6379 \
DATABASE_URL=postgresql://aegis:aegis_local@127.0.0.1:5432/aegis \
  node apps/controlplane/dist/server.js

Open http://127.0.0.1:3000. Stop local dependencies with docker compose down. Do not add -v if you want to preserve local history.

For frontend-only development, run npm run dev -w @aegis/dashboard. API calls still require a control plane at the configured Vite proxy target.

Deploy on Zerops

The repository contains both the service topology and per-runtime build setup:

zcli project service-import zerops-import.yaml -P <project-id>
zcli service push appstable -P <project-id> --setup appstable
zcli service push appcanary -P <project-id> --setup appcanary
zcli service push controlplane -P <project-id> --setup controlplane
zcli service push orchestrator -P <project-id> --setup orchestrator

zerops-import.yaml creates the six services. zerops.yaml defines clean npm builds, minimal deploy artifacts, private URLs, generated managed-service references, health checks, and runtime commands. The control-plane command runs idempotent migrations before starting HTTP. Never replace Zerops connection references with committed credentials.

Detailed instructions and verified production evidence live in docs/zerops-deployment.md.

API summary

Method	Path	Purpose
GET	/health	Public control-plane health
GET	/demo/*	Weighted proxy to stable/candidate
GET	/api/state	Active deployment, weights, and faults
GET	/api/events	Live Server-Sent Events stream
POST	/api/deployments	Start a rollout
GET	/api/deployments	List durable history
GET	/api/deployments/:id	Deployment events and snapshots
POST	/api/deployments/:id/abort	Safely abort an active rollout
GET/PUT	/api/demo/faults	Read or change candidate faults
POST	/api/demo/reset	Return the demo to safe defaults
POST	/api/demo/traffic/start	Start bounded generated traffic
POST	/api/demo/traffic/stop	Stop generated traffic
GET	/api/demo/traffic/status	Read generator status

Testing

Run the complete critical suite with one command:

npm run verify:mvp

It performs strict type checking, linting, 74 deterministic unit/integration tests, production builds, and formatting validation. Coverage includes policy thresholds, weighted routing, metrics, SSE, API validation, atomic transitions, and a worker-level rollback flow. External checks that require Zerops are listed in docs/verification.md.

Production rehearsal command:

AEGIS_URL=https://controlplane-297d-3000.prg1.zerops.app npm run rehearse:demo

Failure handling

• Candidate proxy failures are recorded as failed candidate samples and carry request/deployment correlation IDs.
• Proxy timeouts are bounded and logged with target, internal host, timeout, and error code.
• Valkey weight refresh failure retains the last in-memory value; cold start defaults to stable-only.
• A distributed lock prevents concurrent orchestrator transitions.
• Guarded transition scripts reject stale deployment/stage ownership.
• Failed deployment audit creation compensates by resolving the active rollout.
• Restarted control planes recover weights and deployment identity; restarted workers recover active rollout state or safely roll back on violations.
• Reset aborts active work, stops traffic, restores healthy faults, and restores 100/0 routing without deleting history.

Current limitations

• One active deployment and one fixed stable/candidate pair per project.
• No authentication or multi-tenant authorization; this is a hackathon demo version.
• No dynamic build, image, or service provisioning.
• Metrics use a bounded Valkey window rather than long-term observability.
• The generated Zerops subdomain is intended for judging/demo traffic, not a custom production domain.
• The dashboard bundle is above Vite's 500 kB advisory threshold.

Future work

• Connect deployments to Git commits and Zerops build versions.
• Add authenticated projects and multiple application targets.
• Support configurable policies and longer baseline comparisons.
• Export OpenTelemetry metrics and traces.
• Add notification integrations and approval gates.
• Promote a healthy candidate into the next stable release automatically.

AI usage disclosure

OpenAI Codex was used as a coding and documentation assistant.

Repository guide

apps/controlplane   public API, proxy, metrics, SSE, dashboard host
apps/dashboard      React operator interface
apps/demo-app       stable/candidate application
apps/orchestrator   evaluator and rollout worker
packages/shared     contracts, policy, events, Valkey keys
migrations          PostgreSQL audit schema
scripts             migrations, routing sample, rehearsal, recording
docs                architecture and production evidence
zerops.yaml         build/runtime definitions
zerops-import.yaml  six-service Zerops topology

Code: https://github.com/kush1jpeg/aegis