[detection-analysis] Detection Analysis Report — 2026-10-17

[github-actions[bot]]

Summary

• Window evaluated: last 24 full hours ending 2026-10-17T23:15Z (UTC)
• Total runs analyzed: 483 of 618 downloaded run directories (135 excluded — no aw_info.json, meaning the agent job never ran far enough to emit it; out of scope for this report's detection rules)
• Detection-enabled runs: 423 (87.6%)
• Regular runs (gh-aw-detection: false): 60 (12.4%)
• Misconfigured workflows found: 1
• Coverage gap: logs for 2026-10-16T23:17:54Z – 2026-10-17T00:05:21Z (~48 min at the start of the window) were not collected — the MCP log fetch hit its count/storage limit before reaching that slice. All other analysis below is scoped to the 483 classifiable runs actually retrieved.

Warning

One workflow (Smoke Copilot) has gh-aw-detection: false with run volume above the misconfiguration threshold.

Comparison Chart

Metric	Regular (detection: false)	Detection-enabled
Runs	60	423
Success rate	66.67%	92.43%
Failures	20	32
Avg. tokens/run	84,092	124,566

Regular runs are almost entirely engine smoke/conformance tests (Smoke Copilot, Smoke Claude Routed, Smoke Codex, Smoke Cursor, etc.) that intentionally opt out of detection since they exercise raw engine plumbing rather than untrusted content — this explains most of the lower success rate and token usage in that group, not a defect in the detection feature itself.

Misconfigured Workflows

Workflow	Misconfiguration type	Run count	Recommended fix
Smoke Copilot	gh-aw-detection: false with >3 runs in window	5	.github/workflows/smoke-copilot.md sets features.gh-aw-detection: false explicitly. This appears to be a deliberate choice for a build/pass-rate smoke test, but it exceeds the high-volume threshold this check flags on. Confirm the opt-out is intentional; if so, no action needed beyond documenting why, otherwise remove the override so detection runs by default. Example run: §38987143187.

No runs matched the "detection job itself failed while enabled" rule (rule 2), and no workflow alternated between enabled/disabled within the window (rule 3).

One adjacent observation (not flagged, informational only): run §38992703035 (Smoke Copilot, detection explicitly disabled) shows its detection job with conclusion: failure. Inspection shows this job still executes unconditionally (gated internally by a detection_guard step) even when the feature is disabled, and here the failure traces to a shared infrastructure step, "Download container images," which failed identically in the agent job in the same run — a transient infra issue, not a detection-logic defect.

View All Run Metrics

Workflows with more than one run in the window (28 of 188 distinct workflows seen):

Workflow	Total runs	Detection runs	Regular runs	Success rate	Avg tokens	Engine(s)
PR Sous Chef	99	99	0	100.0%	215	pi
Issue Monster	46	46	0	97.8%	144,031	pi
Daily Trajectory Grader Implementer	45	45	0	97.8%	6,514	copilot
Daily Go Test Parallelizer	12	12	0	100.0%	941,911	codex
Matt Pocock Skills Reviewer	12	12	0	100.0%	48,667	copilot
Ponytail Reviewer	12	12	0	91.7%	124,839	codex
Test Quality Sentinel	12	12	0	100.0%	10,128	copilot
Design Decision Gate 🏗️	11	11	0	100.0%	1,993	pi
PR Code Quality Reviewer	11	11	0	100.0%	157,736	copilot
Auto-Triage Issues	8	8	0	100.0%	138,876	codex
Contribution Check	6	6	0	100.0%	19,776	copilot
[aw] Failure Investigator (6h)	6	6	0	66.7%	266	claude
Smoke Copilot	5	0	5	60.0%	8,681	copilot
Code Scanning Fixer	4	4	0	100.0%	10,843	copilot
PR Triage Agent	4	4	0	100.0%	70,835	copilot
Smoke Claude Dynamic Workflow	4	4	0	100.0%	1,532	claude
Avenger	3	3	0	100.0%	282,470	copilot
Smoke Copilot Small	3	0	3	100.0%	7,224	copilot
Daily Credit Limit Test	2	2	0	100.0%	24,709	pi
ESLint Refiner	2	2	0	50.0%	1,201	claude
Smoke Claude Routed	2	0	2	100.0%	1,846	claude
Smoke Claude on Copilot	2	2	0	100.0%	680	claude
Smoke Copilot - AOAI (Entra)	2	0	2	0.0%	0	copilot
Smoke Copilot - AOAI (apikey)	2	0	2	0.0%	0	copilot
Smoke Copilot SDK Routed	2	0	2	100.0%	2,524	copilot
Smoke OTEL	2	0	2	100.0%	505,546	codex
Smoke Pi Routed Claude	2	0	2	100.0%	3,551	pi
Smoke Pi Sub Agents	2	0	2	100.0%	36,028	pi

The remaining 160 workflows each had exactly 1 run in the window: 120 succeeded, 40 failed/other (75% success rate), spread across copilot, codex, claude, pi, aider, goose, agy, cursor, opencode, crush, deepseek-harness, and gemini engines. None of these single-run workflows meet the >3-runs-in-window threshold used for misconfiguration rule 1.

View Historical Trend

Based on 19 accumulated daily snapshots in cache-memory (7 unique calendar days after de-duplication), showing regular vs. detection-enabled run counts per day with a shaded band for detection success rate.

Recommendations

1. Confirm intent for Smoke Copilot. It explicitly sets gh-aw-detection: false in .github/workflows/smoke-copilot.md and runs frequently enough (5×/24h) to trip the high-volume misconfiguration threshold. If this is a deliberate smoke-test scoping decision (likely, given its purpose), no code change is needed — just confirm. If not intentional, drop the override.
2. No detection-job defects found. Rule 2 (detection enabled + detection job itself failing) matched zero runs this window — the detection feature's job-level reliability looks healthy.
3. Re-run analysis once the ~48-minute collection gap closes (2026-10-16T23:17:54Z–2026-10-17T00:05:21Z) if a complete 24h picture is needed; today's conclusions are based on the 483 runs that were retrievable within the MCP log budget.

References:

• §38987143187 — Smoke Copilot, example of the flagged misconfiguration
• §38992703035 — Smoke Copilot, informational: detection job failed due to shared infra issue while detection was disabled
• §39046717920 — Smoke Copilot, latest run in window with the same gh-aw-detection: false setting

Generated by 🔎 Detection Analysis Report · claude · agent · 152.3 AIC · ⌖ 0.777 AIC · ⊞ 6.6K · ◷

• expires on Oct 20, 2026, 3:31 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Detection Analysis Report.

A newer discussion is available at Discussion #70034.