Agent Performance Report - Week of 2026-10-21

[github-actions[bot]]

Agent Performance Report - 2026-10-21

Executive Summary

• Metrics freshness: metrics/latest.json is current again (2026-10-21T03:05Z) after a long stale period. Collection is partial: per-workflow safe-output outcomes, engagement and PR merge rates are null, so quality/effectiveness scores are not computed this run; this report is run-outcome based only.
• Runs (24h): 549 total, 538 executed, 408 successful, 130 failed → 75.8% success. 193 of 345 workflows executed; 0 skipped / 0 action_required.
• Ecosystem safe-output tool calls: 356 calls (add_comment 204, assign_to_agent 123, create_issue 107, noop 144, missing_tool 87, create_discussion 30, create_pull_request 15). These are call counts, not confirmed accepted outputs.
• Tokens: 62.6M; AIC 13,609.

Lowest Success Rates (executed runs only)

Workflow	Executed	Success	Avg duration
ai-moderator	10	10%	~83 min
smoke-copilot	9	22%	11.7 min
smoke-copilot-sdk-routed-claude	3	33%	5.5 min
smoke-pi-routed	3	33%	5.5 min
contribution-check	6	50%	7 min
smoke-copilot-sub-agents	6	50%	8.5 min
pr-code-quality-reviewer	28	54%	7.1 min
test-quality-sentinel	32	56%	6.9 min
mattpocock-skills-reviewer	31	58%	10.9 min
design-decision-gate	30	60%	7.3 min
ponytail-reviewer	34	62%	5.7 min

Key Findings

• PR-review cluster is the largest absolute failure source: the five PR-triggered reviewers (test-quality-sentinel, mattpocock-skills-reviewer, ponytail-reviewer, pr-code-quality-reviewer, design-decision-gate) account for 65 of 130 failures (50%) at 54-62% success each. Failures this uniform across different prompts point to a shared cause (engine/model, shared imports or PR-event handling) rather than five separate prompt defects.
• ai-moderator: 9/10 failed with ~83 min average runtime — likely timeouts; highest-priority single-workflow regression.
• Smoke workflows (copilot, routed variants) fail frequently; likely engine/routing issues rather than agent quality.
• missing_tool calls (87) and 11 report_incomplete indicate agents hitting tool gaps; worth triaging by workflow once attribution is available.
• Behavior patterns (label set: over-creation, under-creation, repetition, scope-creep, inconsistency): no reliable classification possible without per-workflow output counts; ci-reviewer cluster flagged as inconsistency (≈40% failure, same prompt style).

Prompt Improvement Initiative

A bottom-10 prompt-deficiency audit requires per-workflow quality scores, which are null this run. Mattpocock Skills Reviewer and Design Decision Gate remain redesign-vs-deprecation candidates: both sit at ≤60% success with ~7-11 min runtimes. Recommendation: first fix shared failure cause; if success stays <70% afterward, reduce scope (single rubric, explicit success criteria, early noop) or deprecate.

Coordination / Memory Hygiene

Root causes in shared-alerts.md (EACCES sandbox regression #61952, model_not_supported_error) date from September and were not re-verified this run; they should be treated as stale until checked against current run logs.

Recommendations

1. High: Pull logs for the 65 failed PR-reviewer runs and identify the shared error; fix once.
2. High: Investigate ai-moderator timeouts (cap runtime, check engine/model).
3. Medium: Fix Metrics Collector attribution so per-workflow safe-output outcomes and engagement are populated; enables real quality scoring next run.
4. Medium: Triage the 87 missing_tool calls by workflow.

Next Steps

Re-run quality scoring once per-workflow metrics are available; re-verify stale shared-alert root causes.

Analysis window: 2026-10-20T03:05Z – 2026-10-21T03:05Z

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · copilot · auto · 18.1 AIC · ⌖ 0.902 AIC · ⊞ 18.5K · ◷

• expires on Oct 22, 2026, 4:59 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-22T12:59:26.314Z.

Closed by Workflow