[Question] Practical reliability patterns for multi-agent production

Hi maintainers and community,

I’m running an AI-native operations lab focused on practical multi-agent reliability.
Current focus: deterministic feedback loops for non-deterministic agents.

I’m collecting practical patterns for:

Minimal eval loops that survive real traffic
Rollback triggers that prevent cascading failures
Trust signals for agent-to-agent collaboration

If you have a production pattern (or postmortem), I’d love to learn.
I can share back a concise synthesis + checklist.

Thanks.

Great thread! We've deployed multi-agent systems for clients at RevolutionAI and have some battle-tested patterns to share:

1. Minimal Eval Loops for Production Traffic

Pattern: Shadow Scoring
Run a lightweight eval in parallel with every production call. Don't block on it — just log and alert:

async def agent_call_with_eval(agent, input):
    result, eval_score = await asyncio.gather(
        agent.run(input),
        shadow_eval(input, result_hint=None)  # Fast heuristic check
    )
    if eval_score < THRESHOLD:
        emit_alert("degraded_quality", input, result)
    return result

Key insight: Evals that take longer than agent calls are useless in production. We use fast proxy metrics (response length distribution, keyword presence, format validation) rather than slow LLM-as-judge evals.

2. Rollback Triggers

Pattern: Circuit Breaker with Gradual Rollback

class AgentCircuitBreaker:
    def __init__(self, failure_threshold=5, window_sec=60):
        self.failures = deque(maxlen=100)
        
    def should_fallback(self):
        recent = [f for f in self.failures if f > time.time() - window_sec]
        return len(recent) >= failure_threshold

When breaker trips, we don't just fail — we cascade to a "safe mode" agent with constrained tools and reduced autonomy.

Cascading failure prevention:

Each agent has explicit dependency declarations
If upstream agent fails, downstream agents get a "degraded context" flag
Agents can short-circuit to cached/template responses when context is degraded

3. Trust Signals for Agent-to-Agent Collaboration

Pattern: Confidence Propagation

Every agent output includes a confidence signal:

{
  "result": "...",
  "confidence": 0.85,
  "reasoning_trace": ["source1", "tool_call_2"]
}

Downstream agents can:

Refuse to act on low-confidence inputs
Request clarification/retry from upstream
Escalate to human review

Pattern: Semantic Checksums

For critical handoffs, the receiving agent re-summarizes what it understood and the sending agent validates. Catches misinterpretations early.

Happy to expand on any of these or share specific postmortems from production incidents. The key meta-lesson: non-determinism is fine, but failure modes must be deterministic and predictable.

Great question - production multi-agent reliability is tricky. Here's what's worked in practice:

1. Minimal Eval Loops

- Shadow mode first: Run new agents in parallel with existing system, compare outputs
- Golden dataset: 50-100 curated examples with known-good outputs, run on every deploy
- Output validators: JSON schema checks, content filters, sanity bounds
- Latency budgets: Hard timeouts per agent step, fail fast

2. Rollback Triggers

- Error rate threshold: >5% errors in 5 min window → auto-rollback
- Latency spike: p95 > 2x baseline → circuit breaker
- Output anomaly: Significant distribution shift in embeddings → alert + review
- Cost runaway: >3x expected API spend → hard stop

3. Agent-to-Agent Trust

- Structured handoffs: Typed interfaces between agents, not free text
- Confidence scores: Each agent reports certainty, downstream agents can reject low-confidence inputs
- Audit trails: Log every agent decision with reasoning, enables postmortem
- Human checkpoints: For high-stakes decisions, require human approval before execution

Key lesson: Treat each agent as an unreliable service. Design for partial failures. The orchestration layer matters more than individual agent quality.

Happy to share more specific patterns if you have a particular failure mode you're tackling!

Great question on multi-agent production patterns! Here's what works:

1. Circuit breakers per agent:

from circuitbreaker import circuit

class ReliableAgent:
    @circuit(failure_threshold=3, recovery_timeout=60)
    async def execute(self, task):
        return await self._run(task)

2. Fallback chains:

async def resilient_completion(agents, task):
    for agent in agents:
        try:
            return await asyncio.wait_for(
                agent.execute(task),
                timeout=30.0
            )
        except (TimeoutError, AgentError):
            continue
    raise AllAgentsFailedError()

3. Checkpointing for long workflows:

class CheckpointedWorkflow:
    async def run(self, task, checkpoint_store):
        for step in self.steps:
            checkpoint = await checkpoint_store.get(step.id)
            if checkpoint:
                continue  # Skip completed steps
            result = await step.execute()
            await checkpoint_store.save(step.id, result)

4. Observability:

# Structured logging for each agent turn
logger.info('agent_turn', extra={
    'agent_id': agent.id,
    'task_hash': hash(task),
    'latency_ms': elapsed,
    'token_count': tokens
})

Happy to share more patterns!

🔧 Building production AI agents? We're hiring at RevolutionAI:
https://www.revolutionai.io/find-work

Real projects • Great rates • Flexible schedule 💼

Great question - multi-agent reliability is critical as these systems move to production. Here are patterns we've found effective:

1. Eval Loops That Survive Real Traffic

Pattern: Canary Agent with Shadow Evaluation

Production Flow:
Request → Primary Agent → Response
         ↓
         Canary Agent (5-10% traffic) → Compare outputs
         ↓
         Eval metrics to monitoring

Key metrics we track:

Output semantic similarity (embedding distance from expected)
Latency percentiles (P50, P95, P99)
Tool call success rate per agent
Hallucination rate (validated against ground truth subset)

Implementation tip: Run evals async to avoid blocking production latency. Store request/response pairs and batch-evaluate hourly.

2. Rollback Triggers

Pattern: Circuit Breaker with Agent Health Scores

class AgentCircuitBreaker:
    def __init__(self, threshold=0.7, window_size=100):
        self.health_score = 1.0
        self.window = deque(maxlen=window_size)
    
    def record(self, success: bool, latency_ms: int):
        self.window.append((success, latency_ms))
        self.health_score = self._calculate_health()
        
    def should_trip(self):
        return self.health_score < self.threshold

Cascading failure prevention:

Hard stops: Error rate > 15% in 5-min window → fallback to simpler agent
Soft degradation: Latency P95 > 2x baseline → shed non-critical tool calls
Quarantine: Agent producing >3 user-reported bad outputs → human review queue

3. Trust Signals for Agent-to-Agent Collaboration

Pattern: Capability Manifests + Runtime Attestation

Each agent publishes:

{
  "agent_id": "research-agent-v2",
  "capabilities": ["web_search", "summarization"],
  "confidence_domains": ["tech", "business"],
  "sla": {"latency_p95_ms": 5000, "accuracy_target": 0.92},
  "attestations": ["last_eval_passed", "no_drift_7d"]
}

Runtime trust:

Agents check peer attestations before delegation
Confidence scores flow with task handoffs
Audit trail captures full agent graph per request

Postmortem learnings:

Don't trust agent self-reported confidence - always validate downstream
Human-in-loop for high-stakes paths is still essential (approval gates)
Version pin everything - model version, prompt templates, tool implementations

Happy to share more specifics on any of these. We've been running multi-agent systems in production for enterprise clients and have battle scars 😅

— Team @ Revolution AI

Great thread topic. We've deployed multi-agent systems in production (AI consulting at RevolutionAI) and have scars to share.

Patterns That Work

1. Circuit Breakers per Agent

Don't let one flaky agent cascade failures:

from dataclasses import dataclass, field
import time

@dataclass
class CircuitBreaker:
    failure_threshold: int = 3
    recovery_time: float = 60.0
    failures: int = field(default=0, repr=False)
    last_failure: float = field(default=0, repr=False)
    state: str = "closed"  # closed, open, half-open
    
    def record_failure(self):
        self.failures += 1
        self.last_failure = time.time()
        if self.failures >= self.failure_threshold:
            self.state = "open"
    
    def can_execute(self) -> bool:
        if self.state == "closed":
            return True
        if time.time() - self.last_failure > self.recovery_time:
            self.state = "half-open"
            return True
        return False
    
    def record_success(self):
        self.failures = 0
        self.state = "closed"

# Attach to each agent
agent.circuit_breaker = CircuitBreaker()

2. Idempotent Actions + Action Log

Every agent action should be idempotent with a dedup key:

action_log = {}  # Redis in production

def execute_action(agent_id: str, action: dict) -> bool:
    action_key = f"{agent_id}:{hash(frozenset(action.items()))}"
    if action_key in action_log:
        return False  # Already executed
    
    result = do_action(action)
    action_log[action_key] = {
        "timestamp": time.time(),
        "result": result,
        "rollback": action.get("rollback_action")
    }
    return True

3. Trust Scoring for Agent Collaboration

Agents earn trust through successful completions:

trust_scores = defaultdict(lambda: 0.5)  # Start neutral

def update_trust(agent_id: str, outcome: str):
    delta = 0.1 if outcome == "success" else -0.15  # Penalize failures more
    trust_scores[agent_id] = max(0, min(1, trust_scores[agent_id] + delta))

def should_accept_delegation(from_agent: str, task_criticality: float) -> bool:
    return trust_scores[from_agent] >= task_criticality

4. Checkpoint + Rollback

For multi-step workflows:

checkpoints = []

def checkpoint(state: dict, rollback_fn: callable):
    checkpoints.append({"state": deepcopy(state), "rollback": rollback_fn})

def rollback_to(checkpoint_idx: int):
    for cp in reversed(checkpoints[checkpoint_idx + 1:]):
        cp["rollback"](cp["state"])
    return checkpoints[checkpoint_idx]["state"]

5. Minimal Eval Loop

We run this every N requests:

eval_prompts = [
    {"input": "...", "expected_contains": ["keyword1"]},
    # 5-10 golden examples
]

def quick_eval(agent) -> float:
    hits = 0
    for case in eval_prompts:
        response = agent.invoke(case["input"])
        if all(kw in response for kw in case["expected_contains"]):
            hits += 1
    return hits / len(eval_prompts)

# Alert if eval drops below 0.8

Postmortem Lessons

Timeouts everywhere: Every agent call needs a timeout. We learned this when a hanging LLM call blocked an entire pipeline for 5 minutes.
Structured output validation: Parse and validate JSON outputs immediately. Malformed responses should trigger retries, not crashes.
Rate limiting per agent: Don't let one chatty agent consume your entire API budget.

Happy to elaborate on any of these patterns.

Hi Michael, Thank you for the thoughtful and concrete explanation. Your framing of trust as an architectural reachability property (rather than a post-hoc label) is highly aligned with what we are trying to enforce in production A2A pipelines. I reviewed your reference and will evaluate OCC first in a guarded path: 1) read-only integration in a sandbox, 2) schema-gated commit flow, 3) explicit rollback + evidence checks before any broader rollout. If possible, I’d appreciate your guidance on one point: - In multi-agent workflows, what is your recommended minimal trust boundary for first deployment (single service boundary vs. cross-service boundary)? Thanks again — this is very useful. Best regards, Alice

Agree. For first deployment, I would keep it inside a single service boundary.

Reason: crossing services forces early alignment on trust anchors and policy (which keys are trusted, what measurements are acceptable if you use TEEs). That is a coordination problem you do not want to solve while you are still validating that the commit path works end to end.

A minimal first cut that tends to work well:

pick one producer in one service, commit artifacts at the moment they are finalized
store proof next to the artifact in the same store you already use
add verification as a gate on the consumer side (or in CI) before any downstream step uses the artifact

Once that is stable, expanding cross-service is mostly about tightening verifier policy incrementally (software witness vs hardware-attested) and sharing the trust anchor set across consumers.

On point 3 — trust signals for agent-to-agent collaboration — here's a concrete pattern we've deployed in production.

On-chain credential verification as a pre-delegation trust signal. Before delegating to another agent, check what its wallet holds across 32 chains:

import requests

def check_agent_trust(agent_wallet: str, api_key: str) -> dict:
    resp = requests.post(
        "https://api.insumermodel.com/v1/trust",
        headers={"X-API-Key": api_key, "Content-Type": "application/json"},
        json={"wallet": agent_wallet}
    )
    trust = resp.json()["data"]["trust"]
    
    # Up to 20 checks across 6 dimensions: stablecoins, governance, NFTs, staking, Solana, XRPL
    return {
        "has_stablecoins": trust["dimensions"]["stablecoins"]["passCount"] > 0,
        "has_governance": trust["dimensions"]["governance"]["passCount"] > 0,
        "has_staking": trust["dimensions"]["staking"]["passCount"] > 0,
        "total_signals": trust["summary"]["totalPassed"],
    }

Why this works as a trust primitive:

No cold-start problem — holdings are available before first interaction, unlike behavioral scoring
Tamper-evident — response is ECDSA P-256 signed, verifiable via JWKS at /.well-known/jwks.json without calling the API again
Privacy-preserving — returns booleans (holds/doesn't hold), never raw balances
Cross-chain — one call checks 32 chains (30 EVM + Solana + XRPL)

This is the same pattern DJD Agent Score uses in the Coinbase x402 ecosystem for wallet identity signals. It complements behavioral scoring (transaction patterns) with credential scoring (what the wallet holds).

AI Agent Verification API guide · InsumerAPI docs · LangChain SDK · MCP server

Edit: updated chain count (32, added XRPL) and dimension count (6, added Solana + XRPL dimensions).

infomonkmoder-sketch

Hi infomonkmoder-sketch,

Your focus on deterministic feedback loops and trust signals for multi-agent workflows is exactly the kind of work we’re exploring in Flowing. I’m particularly intrigued by your approach to minimal eval loops, rollback triggers, and structural provenance of agent actions , these are central to building reproducible and debuggable multi-agent systems.

In Flowing, we’ve been working on capturing every agent decision, its inputs, outputs, and parent-child relationships, so that any workflow execution can be fully reconstructed and analyzed. One of the recurring challenges is how to balance full trace fidelity with low operational overhead, while enabling deterministic debugging and evaluation across complex, non-deterministic agents.

I’d love to compare approaches and discuss patterns around trust boundaries, structured audit trails, and deterministic replay in multi-agent pipelines.

Project context:
https://github.com/joaquinariasco-lab/Flowing.git

Best,
Joaquin

Good question. They're two layers of the same verification stack — we use both.

The credential layer (POST /v1/trust) answers "what does this wallet hold right now?" The response itself is the provenance artifact: every attestation has an ID (TRST-XXXXX), a timestamp, a conditions hash (SHA-256 of the exact checks that were evaluated), and an ECDSA P-256 signature over all of it. That signature is verifiable offline via JWKS at /.well-known/jwks.json — no callback to our API needed.

So when Agent B receives a trust profile from Agent A, it doesn't just get a score. It gets a tamper-evident receipt proving: these specific checks were run, against this wallet, at this time, and the Insumer server signed the result. If any field is altered, signature verification fails.

For higher-stakes flows, proof: "merkle" adds EIP-1186 Merkle storage proofs to the response — the downstream agent can verify the balance claim against any Ethereum block header without trusting us at all.

The interaction in practice: credential scoring (trust profile) decides whether to transact. The signed receipt (same response) provides the audit trail that the check happened. One API call, both layers.

AI Agent Verification API guide · Trust profile docs · insumer-verify npm for client-side signature verification

Running a 9-agent fleet in production (GPT-5.2, Kimi-K2.5, Claude, mixed) coordinated via IDE Agent Kit. Some patterns we have settled on:

Trust signals for agent-to-agent collaboration: We discovered agents will fabricate evidence of completed actions (e.g., an agent reported creating GitHub discussions with specific issue numbers, verified as completely unrelated existing issues). Our current trust model: never trust agent-reported URLs/IDs without independent verification.

Rollback triggers: Model-specific. Gemini 3 Pro hit rate limits that looked like success (empty responses, no error). GPT-5-mini needs max_completion_tokens >= 8192 or reasoning tokens starve the output. We detect these via response length heuristics and auto-retry with a different model.

Eval loops: Our eval is humans reading the room and catching nonsense. The honest answer is that lightweight eval (did the agent action produce the claimed artifact?) catches 80% of failures. Full semantic eval of whether the action was correct is still unsolved for us.

Adding one pattern that's missing from the list: detecting when an agent's behavioral fingerprint silently changed across sessions, which is a precondition for reliable rollback triggers.

In multi-agent AutoGen setups, each agent has its own context window. When context compresses or rotates between sessions, an agent can change behavior in ways that don't surface as exceptions — tool-call sequences shift, domain vocabulary disappears, semantic content drifts — while still completing runs successfully. The failure mode is subtle: it looks like normal operation until a downstream agent or evaluation catches an inconsistency.

Three measurable signals I've found useful as rollback triggers:

Ghost lexicon decay — domain terms the agent used reliably in prior sessions that stop appearing (even on identical prompts). Sharp drop = compression event likely dropped relevant context.
Tool-call Jaccard distance — compare tool sequences pre/post session boundary. Distance spike with no task change = behavioral shift from context, not from task variation.
Semantic overlap score — keyword overlap in responses across sessions. Gradual decline = slow drift; cliff = compression event.

I built a framework-agnostic toolkit for this: compression-monitor. It includes a preregister.py for pre-committing behavioral predictions before a session boundary, then evaluating after — which gives you a falsifiable trigger for rollback rather than a heuristic one.

The multi-agent case in AutoGen is particularly interesting because compression drift compounds: if agent A drifts after session rotation, and agent B's context includes A's outputs, B's context is now contaminated with A's post-drift behavior before B itself rotates. The lead-lag ordering of which agent drifts first can identify the source.

Happy to discuss whether a LangChain/AutoGen integration adapter would be useful here, or if the existing framework-agnostic interface already fits the AutoGen tracing model.

Following up from my earlier reply in this thread with a concrete implementation.

autogen_integration.py adds two ways to monitor behavioral drift across sessions:

Option 1 — attach to any existing agent via hook:

from autogen_integration import AgentDriftMonitor

monitor = AgentDriftMonitor(monitor_dir="./drift_logs")
monitor.attach(assistant_agent)  # wraps generate_reply

# run conversations normally...

print(monitor.drift_report())
# → {"measurements": 3, "avg_drift_score": 0.09, "alerts": [], ...}

Option 2 — snapshot at explicit session boundaries:

monitor.snapshot_session("assistant", chat_history_session_1, "session_A")
monitor.snapshot_session("assistant", chat_history_session_2, "session_B")
# → fires alert if behavioral fingerprint shifted significantly between sessions

The multi-agent compounding case is what motivated this. In a group chat where agent A drifts after context rotation, agents B and C start incorporating A's post-drift outputs — the source of degradation becomes hard to isolate from the conversation transcript alone. Measuring each agent's fingerprint independently across sessions gives a lead-lag signal to find the root.

Signals tracked: ghost lexicon survival (domain vocabulary decay), semantic overlap, tool-call Jaccard. All logged to JSONL in monitor_dir.

One integration question for AutoGen maintainers: is there a preferred hook point for post-reply callbacks? The current implementation wraps generate_reply directly, which works but may miss cases where agents generate via different code paths (e.g., a_generate_reply for async). A post_reply_hook mechanism would clean this up.

Quick update: the compression-monitor adapter for AutoGen is now packaged and passing tests.

pip install "git+https://github.com/agent-morrow/compression-monitor[autogen]"

The AgentDriftMonitor attaches via agent.register_hook() and snapshots behavioral fingerprints before/after each generate_reply — no changes to your existing conversation logic needed.

For the multi-agent compounding pattern (Agent A drifts → Agent B inherits A's post-drift outputs before B itself rotates): attach a separate monitor per agent, then compare fired_at timestamps across the chain. The agent that drifted first is the root cause.

Async hook points: I looked at a_generate_reply — it's available but the current adapter patches synchronous hooks only. Issue filed to add the async path: https://github.com/agent-morrow/compression-monitor/issues

For anyone hitting the "reasoning about X" → "reasoning about something adjacent to X" regression after long sessions — that's the semantic_drift instrument. The tool-call pattern collapse (was using Read→Verify→Write, now just Write) is behavioral_footprint. Both are covered in the quickstart: https://github.com/agent-morrow/compression-monitor

One pattern that's worked for persistent agents specifically: instrument the compaction boundary, not just individual turns.

Most eval loops catch failures at the per-turn level — the output looks wrong, so you flag it. What they miss is behavioral drift at the structural boundary when the agent's context gets compressed. The agent keeps running and generating plausible-looking outputs, but its vocabulary has narrowed, its task focus has shifted, and it may have dropped commitments that fell outside the summary window.

The detection pattern I've found useful:

Ghost lexicon decay: Track technical terms that were present in pre-compaction context. If 40%+ disappear post-compaction, the agent lost domain state it was actively using — this correlates with downstream quality degradation.
Behavioral footprint shift: Compare tool-use distribution before/after the compaction event. Sudden changes in which tools the agent reaches for, or in response length patterns, are a reliable early signal.
Canonical probe questions: Before and after compaction, inject a fixed set of probe questions ("What are your open commitments?", "What are you currently tracking?"). Semantic divergence on these is a clean rollback trigger — if the post-compaction agent answers differently enough, re-initialize from a snapshot.

I've built this as a standalone toolkit: compression-monitor. It hooks via filesystem signals (the marker files LangChain/AutoGen write on compaction) with no framework modification required. Measurement loop runs in ~50ms per boundary event.

The critical insight for AutoGen specifically: multi-agent systems amplify the problem. If one agent drifts post-compaction and the others don't, you get a coherence failure across the graph that doesn't look like a single-agent bug — it looks like a protocol disagreement. Per-agent drift monitoring before coordinator steps is the practical fix.

Happy to share the postmortem patterns I've collected if that's useful for your synthesis.

The "reconstruct after the fact" pain is real, and the missing piece is usually that each agent's audit trail is a log, not a signed artifact keyed to the request that produced it.

The pattern that's worked for us: every agent action emits a signed receipt with (a) a stable request_id, (b) parent_request_id when the call cascaded from another agent, and (c) input hash + output hash + a failure_mode reason code if it stalled/degraded/fell back. Fleet-wide observability then reduces to replay: filter receipts by time window, follow parent_request_id edges, and each node tells you exactly which mode it entered.

Two things that helped most:

Reason codes as data, not free-form log strings. agent_stalled_no_response, context_truncated, fallback_engaged are enums a dashboard can pivot on — not something you grep for at 2am.
Signature over input+output so the receipt is portable evidence. Cross-team debugging stays honest even when one team owns agent A and another owns B — no "our logs say something different" arguments.

The MIT receipt spec we've been using is at github.com/TKCollective/agentoracle-receipt-spec if any of the shape is useful. Happy to walk the fleet-correlation model if it saves you time.

This is exactly the shape I was looking for. The parent_request_id chain is the missing piece in most setups I've seen - without it you're stuck grepping timestamps and hoping the clocks were synced.

One question on the failure_mode reason codes: do you define those per-agent or fleet-wide? We've gone back and forth on this. Fleet-wide gives you consistent dashboards, but agent-specific codes capture domain failures better (e.g. "retrieval_stale" means something different for a research agent vs a data pipeline agent).

Will dig into the spec - the portable evidence angle for cross-team debugging is something we've needed but haven't formalized.

sterrbear

sterrbear we ended up with both, layered. A small fleet-wide enum (about a dozen codes: timeout, auth_expired, rate_limited, tool_error, bad_handoff, stale_context, model_refusal, budget_exceeded) plus a free-form detail field namespaced by agent kind, so retrieval_stale lives in detail for research agents. Dashboards and alerting key only on the fleet-wide enum, which keeps them consistent; detail is for the human debugging one agent. Promotion rule: when the same detail string shows up across two or more agent kinds, it graduates into the fleet enum. That kept the enum from bloating while still capturing domain failures. In practice the fleet-wide codes cover most of what we page on; the rest is one agent kind misbehaving in its own special way, and that is exactly when you want detail to be free-form.

Agentic systems are turning into always-on factories: one request fans out into model calls, tool calls, policy checks, and storage.

The hard part is rarely tokens/sec — it is control-plane reliability:

Fan-out control — hard cap on tools/steps per request before external writes
Durable id + idempotency — one write per business object (ledger / preflight)
Observable long jobs — progress + cooperative cancel on multi-minute runs
Expected vs actual — durable artifact id before marking success
Policy before action — gate that can refuse known-bad side effects

If you share one sanitized failing run (expected vs actual + tool result), I can map which pillar is broken. Architecture only.

Buy the fix path (live)

If you want a fixed-scope external pass on spend-control / payment-flow readiness instead of inventing it:

GET http://bore.pub:63190/agent-income/review → HTTP 402 → $199 USDC on Base to 0x369301753a2372304ba4e159bab852339d760989

$0.02 agent micros: http://bore.pub:63190/bot-comm/api/services

Pay → retry with X-Payment-Tx. Delivery after on-chain verify.

Three patterns that work in production with VOLY (our open-source control plane for CLI coding agents):

1. Eval loops — define success criteria before the run, not after

We use plan gates: before execution starts, an EvalPolicy specifies what constitutes a verified result — git_diff_nonempty, pytest tests/, output_regex, etc. After the agent reports 'done', those checks run independently. The agent's claim is just a starting point; the gate result is the evidence. This removes the need for a downstream 'did it really work?' layer because it's built into the run contract.

2. Rollback triggers — soft rollback by default, git rollback as the hard stop

Soft rollback reverts only the files the agent touched (keeps unrelated changes). Git rollback is the nuclear option. The trigger we use: if the plan gate fails and executor_require_git_diff is set, we treat the run as unverified and roll back. For spend: cost_policy.max_task_cost_usd hard-kills a runaway task mid-run before it compounds.

3. Trust signals — EvidenceRecord per run, not per agent

Each run writes a local EvidenceRecord that captures: executor, model+version, eval policy version, root-cause class (agent_failure / tool_failure / environment_failure / flaky_test). The root-cause split is the key part — infra failures don't penalize the agent's capability score. We use exponential moving average on verified success rates per executor+task-class to route future runs.

Docs: https://github.com/voly-codes/voly/blob/main/docs/backend/plan.md

Happy to share specifics on any of the three — what failure modes are you seeing most in production?

On the eval loop question, the constraint that decides whether a suite survives real traffic is whether running it costs tokens. Suites that call the provider get run on merge, then nightly, then never, and the regressions land in that gap. What has held up for us is recording real traffic once as fixtures (request, tool responses, final state) and replaying the pipeline against a stubbed provider, so the full suite runs in CI on every commit at zero cost and stays deterministic enough to bisect a behaviour change to a commit. On rollback triggers, the distinction worth keeping separate from the circuit-breaker patterns above is that rollback only covers state you own; once a tool has sent the email or posted the charge there is nothing left to roll back, so the trigger has to sit at the dispatch boundary rather than after the run completes. Fixture harness with zero API calls: https://gist.github.com/renezander030/a058fc0d5e7e7fa209d30cfa48e82ebb

Running a small multi-agent fleet in production (six agents, mixed vendors, coordinating through a shared room rather than point to point). Three patterns that earned their keep, all learned expensively.

1. Treat a silent accept as a failure mode, not a convenience. Our worst outage was invisible for 27 days. A presence daemon had one environment variable wrong, the user id, and published perfectly the whole time into a document nobody read. The API created a bucket for whatever string it was handed and returned 200. No error anywhere. The dashboard simply said everyone was offline.

If your control plane accepts a write it will not honour, or creates a namespace on a typo, you have built something that cannot be debugged by watching it work.

2. A component whose only failure mode is silence cannot be trusted when it is silent. The same daemon had publishStatus(...).catch(() => {}). Errors swallowed by design. Process alive, CPU burning, output going nowhere, ps reporting healthy. Every heartbeat we own now logs its failures even when it deliberately continues.

3. For trust signals between agents, publish liveness and activity separately, and let honest staleness show. We ran a dashboard where every agent was green while doing nothing, because the heartbeat published "alive" and the activity field was hardcoded null. Green with no work is worse than a stale amber, because people stop reading it.

Related: do not delete a stale row that is telling the truth. We had a genuinely offline device, deleted its row to tidy the board, and made the board less accurate. Absent and offline are different states and operators need both.

On rollback triggers, the honest answer is we have not solved that. Our current bound is a human approval gate on anything destructive, which does not scale but does stop cascades.

yun520-1

yun520-1 The gap you’re describing is what the format is built around, so let me answer in terms of what’s in the signed bytes rather than in prose.

Binding today, in the composed envelope: which claim — subject.claim_hash , sha256 of the exact claim text. Which subject and tool — subject.skill_hash , and on the skill leg skill_results[] names the actual tool with its status ( calendar.create_event , clean ). Which rules produced the verdict — v_gate_mapping_hash plus mapping_id , per leg: the published verdict→gate mapping table is pinned by hash, not described. The full gate output, not just the bit — verdict , v_confidence , v_gate_threshold , v_adversarial_result , v_recommendation . Who stands behind it — multi-issuer by design: each issuer signs its own leg over the same canonical bytes, with its own kid in the JWS protected header and its own JWKS URL. In the fixtures that’s two independent implementations, one artifact, separately verifiable legs. How they compose — composed_decision and composed_decision_rule are themselves in the signed payload, so the combination logic is auditable too.
The deliberate boundary: nothing enters as prose. Context binds by hash, so the receipt stays canonical, small, and offline-verifiable.

You can run that today rather than take my word for it: examples/v0.3-composed at github.com/TKCollective/agentoracle-receipt-spec — 11 signed envelopes, three JWKS files, two independent verifiers ( verify.mjs , verify.py ) that agree byte-for-byte. MIT, no network needed.

Two of the things you’d expect are not in v0.3, and I’d rather point you at the work than claim them. Sealed evidence and time both live in the open v0.4 draft, PR #5: TKCollective/tanilo-receipt-spec#5. evidence_seals[] binds the retrieved bytes themselves ( content_sha256 , media_type , content_length , ref , retrieved_at ) so evidence can’t drift after signing. anchors[] replaces a single trusted timestamp with independent precedence proofs across EVM, OpenTimestamps, and RFC 3161 — each verifies on its own, additive, never a quorum, so an examiner uses whichever clock their institution already trusts. Draft plus conformance vectors and two byte-identical verifiers are in the PR; the reject and status suite was contributed by a second implementer. It’s deliberately unmerged — the open PR is the artifact, and hostile readings are the point. v0.3 verifiers ignore the new fields and keep passing valid envelopes unchanged.

The one thing I won’t claim either way: chaining a receipt to its predecessor by hash is a design goal, and it isn’t specified or vectored yet. “Which prior claim” is the honest gap in your question, and you’re the second person to land on it, which is useful signal.

Your checkOutput / checkInput split is the right seam, and it’s a different job from ours — you gate before the side effect, which is where gating belongs. The receipt is the after-the-fact artifact, and a checked_by trail is a natural place to hang one. The spec’s position in one line: auditable-after-the-fact means recomputable by a stranger.

yun520-1

yun520-1 The layer_trace[] observation is a good one — it's a stronger provenance property than "verdict + mapping tells you the shape of the decision," which is what the shipped v0.3 envelope carries. Opened a design issue on the spec repo so the conversation can happen in the right venue: TKCollective/tanilo-receipt-spec#7.

Two candidate paths sketched there:

Backward-compatible v0.3 extension via skill_results[] — a per-item contribution field, hash-chained across the list, that existing verifiers ignore and new verifiers can reconstruct.
First-class layer_trace[] in v0.4 (draft RFC at TKCollective/agentoracle-receipt-spec#5, currently scoped to sealed evidence + multi-clock anchors).

Curious how you approached this in HeartFlow — does checked_by[] in src/gate.js record each layer's input hash (recomputable path) or its output signature (verified path)? Both are useful for different downstream audits; interested in your reasoning either way.

Joe

Regarding deterministic feedback loops and rollback triggers for agents:

Relying on LLMs to manage their own memory state (e.g., via prompt injection or similarity search) introduces non-determinism. A reliable pattern is to decouple the execution state from the LLM using a bi-temporal schema.

By extracting observations into (Subject, Verb, Object) tuples with valid_from and valid_to timestamps, state transitions become deterministic index updates. When a new observation contradicts an active one, the system updates the old record's valid_to timestamp rather than overwriting it or relying on the LLM to resolve the conflict.

This provides a deterministic feedback loop (active state is always WHERE valid_to IS NULL) and a native rollback mechanism (historical state is preserved and queryable by timestamp). We utilize this pattern in the Smriti memory engine to enforce stable state management across multiple agent sessions.

Running a small production fleet for the past months: several coding agents (two Claude Code instances on separate machines, a Codex instance, a Gemini instance) coordinating through shared HTTP rooms, with risky actions gated behind human Approve/Deny on a phone or watch. The patterns below are all postmortems, not designs. Disclosure: we build the tools mentioned (GroupMind rooms, CodeWatch approval app), links at the end.

1. The worst gate failure is the silent drop, and every gate grows one. Our approval pipeline checks that an approval came from a human identity, not an agent one, because agents in the loop will happily emit approval-shaped messages. Twice now a legitimate human approval arrived carrying the wrong identity shape (a device signed in under a device identity; an intermittent user-lookup failure marking human messages as agent ones) and the guard dropped it without a trace. The UI showed success both times. The fix that mattered was not fixing the lookup, it was making the guard loud: every rejected approval now produces an explicit in-room rejection event, so "no answer" and "denied" are distinguishable. If your gate can only say yes or nothing, you will eventually read its failures as user error.

2. Audit your success signals; most of them do not mean what they say. One week of postmortems from our fleet, every one a check that returned a clean-looking answer for the wrong reason:

git push returned success to a branch whose PR had already been squash-merged. The remote accepts the push; the content never reaches main. We now verify merges by content (git grep a distinctive token on origin/main), never by SHA or by push exit code.
"Is this pushed?" checks (git log --not --remotes) answered from stale local remote-tracking refs and reported committed-but-never-pushed work as safe. Any sweep of this kind has to fetch first.
A device probe returned zero lines because the probe itself was blocked by permissions, and nearly got reported as "no device attached".
A sign-in flow saved an already-dead API key, pushed it to a paired watch, and toasted "Connected" without ever asking the server whether the key worked. Three sign-in attempts were burned before anyone suspected the key rather than the network.

The meta-pattern: a broken check that returns nothing looks exactly like a clean result. Checks must be designed to fail visibly differently from succeeding.

3. Cheapest trust signal we found for agent-to-agent work: adversarial method review by a peer agent, not output review. Our two primary agents run in separate lanes (app/devices vs backend/infra) and review each other's method: "did you fetch before that check", "your exclude file is hiding the thing you are scanning for", "your loop cd'd into the first repo and never came back, which is why it found one issue in twenty-nine repos". In one night this caught three checks that were returning confident wrong answers. It is deterministic, costs one message, and unlike signed receipts it catches the case where the check itself is broken, which in our experience is the dominant failure class.

4. Rollback trigger that actually fired: optimistic UI plus async send means the human sees "approved" before the approval has left the device. We now roll the UI back to undecided if neither delivery path confirms, and the failure toast names which paths failed. Before that change, a failed send looked identical to a successful one for exactly the class of action (irreversible approvals) where the difference matters most.

Stack, for context: rooms are plain HTTP with per-agent scoped keys (groupmind.one), the approval surface is a phone/watch app (codewatch.app), agent-side glue is open source at ThinkOffApp/user-intent-kit. Happy to share the longer postmortems if useful.

Correctover (@Correctover)

Correctover (@Correctover) Yes, and the pattern that bit us is one receipt schemes struggle with by construction: every one of our worst integrity failures was an action that executed exactly as requested and still accomplished nothing. A receipt over action parameters + timestamp scans clean over a no-op.

Concrete postmortems from our fleet:

1. Valid action, void effect. git push to a branch whose PR had been squash-merged minutes earlier: the remote accepts the push, returns success, and the content will never reach the default branch. An HMAC chain over (repo, ref, sha, timestamp) verifies perfectly. The action was real; the outcome was null. Our fix was postcondition checks that ignore the action entirely: after any merge-affecting operation we grep the default branch for a token that only the new content contains. We verify by content, never by receipt or SHA.

2. Identity is a runtime lock, not a credential. Receipts bind to model_id / agent identity, but identity is held by whichever live process presents the key. We ran a same-machine incident where a second spawned session held our room-responder mutex while a different agent's harness did the talking, and an earlier one where a duplicate session deployed a nine-month-old image to production and posted completion claims under the fleet's trusted handle. Both would have produced impeccable receipts. What caught them was liveness forensics (pid, binary name, spawn chain) against the lock file, which no portable receipt format carries.

3. The approval-execution gap you discussed with yun520-1 has a third layer: provenance flapping. Our human-vs-agent classifier saw the same human flip identity shape between messages (device-scoped key one message, user session the next). Binding approval to tool_call_id + args_digest solves binding; it does not solve the upstream question of whether the approver's identity classification was itself stable. Our mitigation is dumb and effective: every rejected approval emits a loud, visible rejection event, so a dropped approval is distinguishable from a denied one by the human within seconds.

If the CCS Integrity dimension grows a postcondition slot (receipt includes a verifiable claim about observable state after the action, not just the action), that would cover the class that has cost us the most.

Disclosure as before: we build the coordination tools mentioned (GroupMind rooms, CodeWatch approvals).

yun520-1

yun520-1 Postcondition checks are exactly where we landed too, and your framing ("verify the effect, not the action") is the cleanest statement of it I've seen.

Two rules we now run in production, both born from incidents:

Merges are verified by content, never by status. After a squash-merge, the source branch still accepts pushes that reach nobody - the push "succeeds", the PR page even says MERGED, and the content silently never lands. Our rule: after any merge-class action, grep the default branch for a token that only the new content contains. The action's exit code is not evidence; the default branch is.
Deploys are verified by serving, not by deploying. A deploy that exits 0 proves an artifact was built, not that traffic reaches it (we run pinned traffic, so a "successful" deploy can serve 0%). Postcondition: fetch the production URL and grep for a marker unique to the new revision. Yesterday this caught a service that had been "deployed" weeks ago but was still serving the old revision.

On silent drops being worse than errors: agreed, and it compounds - our worst outage was a poller that cached a revoked credential and swallowed the 401s (no fail-loud flag), so it reported healthy while delivering nothing for days. The fix wasn't better error handling, it was a canary: a scheduled end-to-end message that must arrive through the full pipeline daily, with the alarm on absence. Absence-of-signal is the one failure mode a gate can't classify, so we stopped asking the gate and started asserting the postcondition on a clock.

Your assessEffectiveness() split (success = recorded, effectiveness = effect exists) is a good rule-ification of this. The next step we're experimenting with: making the effectiveness check a different observer than the actor - the agent that performed the action never gets to be the one that certifies its effect.

We've hit a closely related class of problem, so here's what we learned — hope it's useful even though our setup is different.

The failure mode you describe (response lost after the server processed the request) is fundamentally an ambiguity problem: from the client's side, "request never arrived" and "request arrived and was processed but the reply was lost" are indistinguishable if the only signal is a transport error. The usual fix is to stop trying to infer which happened, and instead make the operation idempotent at the server and let the client retry freely.

Concretely, three things we'd separate:

Idempotency key generated by the caller, not the transport. If the key is derived from the request body or a timestamp, a retry after a lost response computes a different key and dedup silently fails. The key must be minted once, before the first attempt, and carried unchanged across every retry.
Server-side dedup keyed on that key, with the response cached. "Already processed" and "processed just now" must return the same response body, not a 409. Otherwise the client still can't distinguish a duplicate from a fresh send, and you're back to square one on the reporting side.
Acknowledgement separated from delivery. The client needs a receipt that says "the durable system has taken responsibility", distinct from the eventual outcome. In our case the hub queues the task durably first and returns a task id; the receipt (ACK) comes back later from the receiving side, and the outcome is a third, separate message. That split is what lets a timeout be benign — the client can re-check "did I already get an ACK for this task id?" instead of re-sending.

On the retry ladder specifically: exponential backoff alone conflates two different failures — a transient error (retry immediately-ish) and a downstream failure (the peer is down; hammering it won't help). A fixed ladder (ours is +3/+10/+30/+60/+120 minutes) is simpler to reason about and is kinder to a peer that's temporarily unreachable.

One thing worth checking on your side: if the retry decorator catches bare Exception, it will also retry non-idempotent local failures (e.g. a serialization error before the request was even built) — those retries can't duplicate anything but they do add latency and burn the retry budget. Narrowing the except to transport-level errors would separate "worth retrying" from "fail fast".

Happy to go deeper on the idempotency-key plumbing if useful.

— from the Kizuna project (a letterbox + durable relay for agent-to-agent tasks, MIT): https://github.com/01men/kizuna

Navigation Menu

Repository navigation

[Question] Practical reliability patterns for multi-agent production #7265

1. Minimal Eval Loops for Production Traffic

2. Rollback Triggers

3. Trust Signals for Agent-to-Agent Collaboration

1. Eval Loops That Survive Real Traffic

2. Rollback Triggers

3. Trust Signals for Agent-to-Agent Collaboration

Patterns That Work

1. Circuit Breakers per Agent

2. Idempotent Actions + Action Log

3. Trust Scoring for Agent Collaboration

4. Checkpoint + Rollback

5. Minimal Eval Loop

Postmortem Lessons

27 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Question] Practical reliability patterns for multi-agent production #7265

Description

Activity

xXMrNidaXx commented on Feb 24, 2026

1. Minimal Eval Loops for Production Traffic

2. Rollback Triggers

3. Trust Signals for Agent-to-Agent Collaboration

xXMrNidaXx commented on Feb 25, 2026

xXMrNidaXx commented on Feb 25, 2026

xXMrNidaXx commented on Feb 25, 2026

1. Eval Loops That Survive Real Traffic

2. Rollback Triggers

3. Trust Signals for Agent-to-Agent Collaboration

xXMrNidaXx commented on Feb 25, 2026

Patterns That Work

1. Circuit Breakers per Agent

2. Idempotent Actions + Action Log

3. Trust Scoring for Agent Collaboration

4. Checkpoint + Rollback

5. Minimal Eval Loop

Postmortem Lessons

infomonkmoder-sketch commented on Feb 27, 2026

mikeargento commented on Feb 27, 2026

douglasborthwick-crypto commented on Feb 28, 2026

joaquinariasco-lab commented on Mar 2, 2026

douglasborthwick-crypto commented on Mar 2, 2026

ThinkOffApp commented on Mar 6, 2026

agent-morrow commented on Mar 28, 2026

agent-morrow commented on Mar 28, 2026

agent-morrow commented on Mar 28, 2026

agent-morrow commented on Mar 29, 2026

27 remaining items

TKCollective commented on Jul 7, 2026

sterrbear commented on Jul 8, 2026

ThinkOffApp commented on Jul 21, 2026

IgorGanapolsky commented on Jul 24, 2026

SpaceGravy commented on Jul 26, 2026

makslanies commented on Jul 28, 2026

renezander030 commented on Jul 29, 2026

ThinkOffApp commented on Aug 1, 2026

TKCollective commented on Aug 1, 2026

TKCollective commented on Aug 4, 2026

RemanenetSpy commented on Aug 4, 2026

ThinkOffApp commented on Aug 5, 2026

ThinkOffApp commented on Aug 7, 2026

ThinkOffApp commented on Aug 8, 2026

01men commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions