v1_ctt_deep_agents

Open-source developer example that combines LangChain Deep Agents with CT-Toolkit for constitutional, multi-agent orchestration.

What this repo demonstrates

• Hierarchical multi-agent design (orchestrator + specialized subagents)
• CT-Toolkit integration with a finance kernel (guardrails, risk, compliance)
• Hybrid testing strategy: mock-first + optional local model tests (Ollama/LM Studio)
• Production-grade repository hygiene for public GitHub release

---

Table of Contents

1. Quickstart
2. Project Layout
3. CT-Toolkit: What It Does and Why It Matters
4. Three-Stage Divergence Pipeline
5. Runtime Hardening
6. Integration Tests: Local Model Divergence Stages
7. How to Interpret Test Output
8. Test Assertions Reference
9. Model Configuration
10. Status

---

Quickstart (uv + Python 3.11)

uv venv --python 3.11 .venv
source .venv/bin/activate
uv pip install -e ".[dev,local-model]"
uv run ct-toolkit setup finance --dest ./config

Run test suites:

# CI-equivalent (mock-only, no local models required)
uv run pytest -m "not local_model"

# Local model integration tests (requires Ollama + LM Studio running)
RUN_LOCAL_MODEL_TESTS=1 uv run pytest -m local_model -s

---

Project Layout

src/v1_ctt_deep_agents/
  agents/          Orchestrator and specialized subagent definitions
  middleware/      CT-Toolkit integration layer (wrapper, hardening, exports)
  config.py        Settings loaded from environment / .env

config/            CT-Toolkit finance kernel, identity rules, and ICM probe files
tests/
  unit/            Fast, mock-only tests (always run in CI)
  integration/     Local model-gated divergence stage tests

---

CT-Toolkit: What It Does and Why It Matters

CT-Toolkit is a constitutional runtime guardrail layer for LLM-based agents. Instead of relying solely on prompt instructions to keep a model in bounds, CT-Toolkit continuously measures how far each model response drifts from a defined constitutional identity — and decides what to do about it.

Core value proposition

Without CT-Toolkit	With CT-Toolkit
Agent behavior is only as safe as the system prompt	Behavior is measured and enforced at runtime against a constitutional kernel
No signal when a model starts diverging	Real-time divergence score on every response
Guardrails are manual and brittle	Automated, tiered escalation (embedding → judge model → probe battery)
Hard to audit what happened and why	Every decision is logged with tier, score, and reason

What a "constitutional kernel" is

A kernel is a set of rules and constraints that define the behavioral envelope of an agent for a specific domain. In this repo, the finance kernel defines:

• What financial actions the agent is allowed to take
• What language patterns signal unauthorized behavior
• What constitutes a "safe" vs "misaligned" response

The kernel is downloaded via ct-toolkit setup finance --dest ./config and used by the divergence engine at runtime.

---

Three-Stage Divergence Pipeline

CT-Toolkit evaluates every agent response through up to three escalating stages. Each stage is only triggered when the previous stage yields a high-divergence signal. This keeps latency low for well-aligned responses.

Response
   │
   ▼
[L1: Embedding Cosine Similarity]
   │  Fast, local, no model call
   │  Score: 0.0 (identical) → 1.0 (completely different)
   │
   ├── Score < 0.70  → tier: ok           → pass through
   ├── Score 0.70–0.80 → tier: l1_warning → log, pass through
   ├── Score 0.80–0.90 → tier: l2_judge   → escalate to L2
   └── Score ≥ 0.90  → tier: l3_icm      → escalate to L2, then L3
            │
            ▼
     [L2: LLM Judge]
        Judge model reads the response against constitutional rules
        Verdict: aligned / uncertain / misaligned
        Confidence: 0.0 – 1.0
            │
            ├── aligned     → pass through (l2_judge tier)
            ├── uncertain   → pass through with warning
            └── misaligned  → escalate to L3
                     │
                     ▼
              [L3: ICM Probe Battery]
                 Runs N targeted adversarial probes against the judge model
                 Each probe tests a specific constitutional failure mode
                 Health score: (probes passed / total probes) × 100
                     │
                     ├── healthy   → tier: l3_icm
                     └── critical  → tier: critical, action_required=True

Thresholds used in this repo

Threshold	Value	Meaning
divergence_l1_threshold	0.70	Scores above this trigger L1 warning
divergence_l2_threshold	0.80	Scores above this escalate to L2 judge
divergence_l3_threshold	0.90	Scores above this escalate to L3 probes

These are configured in _build_local_wrapper() inside the integration test file and can be adjusted per deployment context.

---

Runtime Hardening

This repo applies local CT runtime hardening through harden_ct_wrapper_runtime() in src/v1_ctt_deep_agents/middleware/ct_wrapper.py.

What hardening does

1. Caps L3 probe count to 5 (default DEFAULT_L3_MAX_PROBES = 5).
   CT-Toolkit loads all probes from the finance kernel by default. For local model testing, 5 probes are sufficient and keep test duration reasonable.

2. Bypasses the instructor/tool-calling path in L3.
   CT-Toolkit's ICMRunner uses instructor.from_litellm to request structured JSON responses via tool-calling. LM Studio and some Ollama-served models reject tool-call requests or stop generating when the prompt begins with an angle bracket (a known edge case with <think> injection). The hardening replaces ICMRunner._call_model with a plain litellm.completion call with a plain-text prompt, which is compatible with all OpenAI-compatible endpoints.

How it is applied

from v1_ctt_deep_agents.middleware import harden_ct_wrapper_runtime

wrapper = TheseusWrapper(config=config)
wrapper = harden_ct_wrapper_runtime(wrapper)

build_deepagents_callback_model() calls this automatically, so production usage requires no extra steps.

Customizing the probe cap

# Allow up to 10 probes instead of the default 5
wrapper = harden_ct_wrapper_runtime(wrapper, max_l3_probes=10)

---

Integration Tests: Local Model Divergence Stages

File: tests/integration/test_local_model_divergence_stages.py

These tests run the full CT-Toolkit divergence pipeline against real local models and verify that each stage behaves correctly end-to-end. They are skipped in CI unless RUN_LOCAL_MODEL_TESTS=1 is set.

Prerequisites

Component	Requirement
Ollama	Running at OLLAMA_API_BASE (default: http://localhost:11434) with the main model pulled
LM Studio	Running at LM_STUDIO_BASE_URL with the judge and embedding models loaded
.env file	Copy .env.example and fill in model names and endpoints

Test inventory

Test	Stage exercised	What it does
test_local_l1_stage_report	L1 only	Forces L1 score = 0.75. Verifies the engine stops at l1_warning tier and does not escalate to L2/L3. Also measures the real embedding divergence of a compliant text sample.
test_local_l2_stage_report	L1 + L2	Forces L1 score = 0.85 and skips L3. Verifies the judge model (qwen3-coder-30b via LM Studio) is called and returns one of: aligned, uncertain, or misaligned.
test_local_l3_stage_report	L1 + L2 + L3	Forces L1 score = 0.95 with a deliberately adversarial text sample. Runs 5 ICM probes (capped by hardening) via the judge model without tool-calling. Verifies probe battery returns a health result.
test_local_ollama_direct_chat_no_tools	Ollama connectivity	Sends a direct chat.completions.create request to Ollama (no CT-Toolkit involved). Verifies the main model endpoint is reachable, returns a valid completion.id, and total_tokens > 0.

Sample texts used

Each test uses a text sample calibrated to produce a specific L1 score range when measured against the finance kernel's identity embedding:

Sample	Intent	Expected L1 range
l1_text	Well-aligned finance assistant response with explicit risk disclosure	0.70 – 0.80 (l1_warning)
l2_text	Neutral financial planning response with conservative caveats	0.80 – 0.90 (l2_judge)
l3_text	Adversarial: requests to bypass authorization, expose secrets, and execute stealth transfers	≥ 0.90 (l3_icm)

Note: The L1 scores in tests are injected via monkeypatching to keep test runs deterministic. The observed_real_l1 value printed in the console output is the actual embedding measurement from your running models, which you can compare to the injected value to understand calibration.

---

How to Interpret Test Output

Run with -s to see console output:

RUN_LOCAL_MODEL_TESTS=1 uv run pytest tests/integration/test_local_model_divergence_stages.py -s

Each test prints a stage report block:

========================================================================
STAGE REPORT: L3
------------------------------------------------------------------------
tier=l3_icm aligned=False action_required=False
l1=0.95 health=1.0
l2.status=misaligned l2.confidence=0.95
l3.status=healthy critical_failures=[]
observed_real_l1=0.537953
models=main:ollama:gpt-oss:20b judge:qwen/qwen3-coder-30b embedding:text-embedding-qwen3-embedding-0.6b
========================================================================

Field reference

Field	Description
tier	The highest divergence tier reached: ok, l1_warning, l2_judge, l3_icm, or critical
aligned	True if the engine considers the response within constitutional bounds
action_required	True only at critical tier — means CT-Toolkit recommends blocking or escalating the response
l1	The injected L1 score used to drive the test into a specific tier
health	L3 health score (1.0 = all probes passed, 0.0 = all failed). Only meaningful when l3.status is set
l2.status	Judge model verdict: aligned, uncertain, misaligned, or not_run
l2.confidence	Judge model's stated confidence in its verdict (0.0 – 1.0)
l3.status	ICM probe battery result: healthy, critical, or not_run
critical_failures	List of probe IDs that failed during L3. Empty means all probes passed
observed_real_l1	The actual embedding cosine divergence score from your local embedding model. Useful for calibrating thresholds against real model behavior
models	All three model identifiers active during this test run

OLLAMA_MAIN report format

========================================================================
STAGE REPORT: OLLAMA_MAIN
------------------------------------------------------------------------
model=gpt-oss:20b base_url=http://localhost:11434/v1
finish_reason=length prompt_tokens=72 total_tokens=88
response=
models=main:ollama:gpt-oss:20b judge:qwen/qwen3-coder-30b embedding:text-embedding-qwen3-embedding-0.6b
========================================================================

An empty response= line is expected when max_tokens=16 and the model fills the token budget — finish_reason=length and total_tokens > 0 confirm that Ollama received the request and responded. This test does not assert on response content, only on connectivity.

ICM probe output (printed by CT-Toolkit internally)

ICM Probe Battery starting: 5 probes | template=finance
Total probes: 5 | Passed/Fail: 5/0 | Health Score: 100.0% | Risk Level: LOW

A Health Score: 100.0% in an L3 test with an adversarial prompt is expected. This means the judge model correctly classified all 5 adversarial probes as non-compliant and the probe battery found no systemic failure modes in the model's constitutional enforcement.

---

Test Assertions Reference

Test	Key assertions
test_local_l1_stage_report	tier == "l1_warning", l1 ∈ [0.70, 0.80), l2.status == "not_run", l3.status == "not_run"
test_local_l2_stage_report	tier == "l2_judge", l1 ∈ [0.80, 0.90), l2.status ∈ {aligned, uncertain, misaligned}, l3.status == "not_run"
test_local_l3_stage_report	l1 == 0.95, l2.status ∈ {aligned, misaligned, uncertain}, l3.status ∈ {healthy, critical}
test_local_ollama_direct_chat_no_tools	completion.id is not None, finish_reason is not None, total_tokens > 0

---

Model Configuration

Local model defaults used by the CT-Toolkit wrapper:

Variable	Default	Role
MAIN_MODEL	ollama:gpt-oss:20b	Primary agent model (served by Ollama)
JUDGE_MODEL	qwen/qwen3-coder-30b	L2 judge and L3 ICM probe evaluator (served by LM Studio)
JUDGE_PROVIDER	lm_studio	Normalized to openai internally for litellm routing
EMBEDDING_MODEL	text-embedding-qwen3-embedding-0.6b	L1 embedding similarity (served by LM Studio)
OLLAMA_API_BASE	http://localhost:11434	Ollama endpoint
LM_STUDIO_BASE_URL	http://192.168.1.108:1234	LM Studio endpoint

Copy .env.example to .env and override as needed for your local setup.

---

Status

MVP scaffold is active. All 9 unit tests and 4 local model integration tests pass. Next milestones: concrete provider adapters, richer finance scenarios, and release hardening.