[agent-job-health] Agent Job Health Monitor — 2026-10-17: fleet rate 5.27%, chronic redact-secrets issue filed

[github-actions[bot]]

Warning

One novel failure cluster (chronic, 10+ days unaddressed) triggered an issue this run; another novel cluster (Engine Conformance Test Matrix) is recurring but below the auto-issue bar for today's count. No fleet-wide regression.

Summary

• Window evaluated: 2026-10-17T00:06 UTC - 2026-10-17T23:16 UTC (~23.2h, close to the intended full 24h ending at this workflow's start)
• total_runs: 531 (completed; 4 additional runs were still in_progress at collection time and excluded), distinct_workflows: 194
• agent_failures: 28 unique runs where the agent job itself failed (verified via failure_clusters[].signal.job=="agent", de-duplicated by run ID since clusters overlap)
• Run-weighted fleet failure rate: 28 / 531 = 5.27%
• Workflow-weighted rate: median 0%, mean 11.1% across 191 workflows with completed runs this window (26 of them had >=1 agent-job failure) - the large median/mean gap confirms the fleet number is driven by a long tail of one-off failures across many different low-volume workflows, not a broad regression affecting most of the fleet.
• Tracked vs. novel: GitHub issue search was blocked by an integrity policy for the entire run (the 5th+ consecutive Agent Job Health Monitor run to hit this), so live verification wasn't possible. Based on this monitor's own history file, ~3 clusters (18 runs) look like already-known chronic signatures; the rest (10 runs across 2 clusters + 10 one-off singles) could not be matched to an open issue.
• Trend vs. previous entry (2026-10-15): 3.76% -> 5.27%, a +1.5 point move - not a >10-point regression, and within the normal fluctuation band seen across the last several months of this history (range ~2.4%-19.7%).

Failure Rate by Step

View table (agent-job failures only, 28 runs across 18 distinct failing steps)

Failing step	Runs	Share of agent_failures
Execute GitHub Copilot CLI	8	28.6%
Assert engine conformance	3	10.7%
Execute Codex CLI	2	7.1%
Execute Pi CLI	2	7.1%
Redact secrets in logs	2	7.1%
Checkout github/gh-aw-side-repo into gh-aw-side-repo	2	7.1%
Verify agent session artifact	2	7.1%
Download container images	1	3.6%
(failed job, no step detail)	1	3.6%
Execute Crush CLI	1	3.6%
Execute Gemini CLI	1	3.6%
Build documentation	1	3.6%
Assert Goose smoke evidence	1	3.6%
Execute Claude Code CLI	1	3.6%
Assert sub-agent model and wire API evidence	1	3.6%
Checkout drive-memory file share (default)	1	3.6%
Execute DeepSeek Harness	1	3.6%

Note: counts above are de-duplicated per cluster; two runs appear in two clusters each (a Copilot-CLI run also flagged "Verify agent session artifact", and the DeepSeek Harness run also flagged "Redact secrets in logs" as a likely cascading symptom), which is why the table sums to slightly more than 28.

Tracked Failures

View table (unverified - see caveat below)

Workflow(s)	Failing runs	Existing issue (per history, not verified live)
daily-squid-image-scan, firewall-escape, smoke-copilot-aoai-apikey, smoke-copilot-aoai-entra	6	Chronic "Execute GitHub Copilot CLI" signature, historically tracked as #54186 / #55413 / #56694
smoke-copilot-mai, smoke-work-queue	2	Same Copilot-CLI family as above
smoke-azure-codex, smoke-codex	2	Chronic "Execute Codex CLI" signature, historically tracked as #54242 / #65950
daily-multi-device-docs-tester, smoke-azure-pi	2	Chronic Pi-CLI family, historically associated with #65909 (rate limiting) - today's failures were not confirmed as the same root cause

[!CAUTION]
GitHub issue search returned 0 results with items removed by an "integrity policy" on every query this run - the same block hit on 2026-10-09 through 2026-10-15 runs of this monitor. The "tracked" classifications above are inferred from this monitor's own history file, not confirmed against live issue state. Treat them as a starting point for triage, not a guarantee the issues are still open or cover today's exact runs.

Novel Failure Clusters

1. Engine Conformance Test Matrix (claude + codex + deepseek-harness) - simultaneous failure

• Count: 3 runs, 3 distinct workflows (engine-conformance-claude, engine-conformance-codex, engine-conformance-deepseek-harness)
• Failing step: "Assert engine conformance" (agent job), driver_exit
• Engine: claude / codex / deepseek-harness respectively
• Representative run: 39021474793 (deepseek-harness variant); siblings 39021458069 (codex), 39021443537 (claude) - all three started within 30 seconds of each other (09:31:14-09:31:43 UTC)
• Suspected root cause: a shared assertion/fixture in the Engine Conformance Dispatcher's matrix, not an individual engine regression (three unrelated engines failed the same assertion in the same minute)
• Blast radius: single dispatcher workflow, engine-conformance-wide
• This exact pattern ("claude+codex+deepseek-harness variants all failed within same ~10min window") was already flagged as novel/untracked in this monitor's 2026-10-15 report. It has now recurred with no evidence of a fix or tracking issue. Recommended: open a tracking issue if one doesn't already exist.

2. "Redact secrets in logs" step failure (driver_exit) - chronic, now ticketed

• Count: 2 runs, 2 distinct workflows (artifacts-summary, smoke-deepseek-harness)
• This is the longest-running untracked signature this monitor has observed: first seen 2026-10-07 (4 runs/4 workflows), confirmed again 2026-10-15 (4 runs/4 workflows), and again today (2 runs/2 workflows) - 10+ days of recurrence across different workflows each time with no confirmed tracking issue.
• Representative run: 39025245757 (artifacts-summary) - failed cleanly in this step alone. The other instance, 38987303689 (smoke-deepseek-harness), failed "Execute DeepSeek Harness" first, so the redaction failure there may be a cascading symptom rather than independent.
• Issue opened this run: see Recommendations below.

3. Ten singleton one-off failures (Download container images / Crush / Gemini / Claude Code / DeepSeek Harness CLI execution, Build documentation, Goose smoke evidence, sub-agent model assertion, drive-memory checkout, side-repo checkout x2) - each affects exactly 1 workflow with 1 failing run this window. These look like isolated smoke-test flakiness rather than a systemic cluster; none individually meets the 2-workflow / 10%-of-runs bar for an issue. Listed in the step table above for visibility.

Schedule Heartbeat

Checked a 29-workflow sample of schedule-triggered workflows (not exhaustive - roughly 249 .md files under .github/workflows/ contain a schedule: key, far more than could be verified individually this run). All large-gap cases found resolve to an already-known cause:

Workflow	Last observed run	Expected cadence	Gap	State
Daily Hippo Learn	2026-07-27	daily	~82 days	disabled_manually
GEO Optimizer Daily Audit (daily-geo-optimizer)	2026-08-18	daily (per name; actual cron is weekly)	~60 days	disabled_manually
Slide Deck Maintainer	2026-09-23	weekly	~24 days	disabled_manually

No blind spots detected among active workflows. Three workflows whose names start with daily- (daily-architecture-diagram) or sound daily (mcp-inspector, blog-auditor, schema-feature-coverage) turned out, on checking their actual frontmatter, to be scheduled weekly, not daily - their 3.5-5.7 day gaps are well within the 15-day weekly blind-spot threshold. This is a reminder that workflow names are not a reliable proxy for cron cadence; this report verified cadence from frontmatter before flagging anything.

Per-Workflow Breakdown

View top offenders by agent-job failure count

Workflow	Runs	Agent failures	Rate
Smoke Copilot - AOAI (apikey)	2	2	100%
Smoke Copilot - AOAI (Entra)	2	2	100%
gh-aw work-queue (artifacts-summary)	1	1	100%
Daily Grader Audit	1	1	100%
Multi-Device Docs Tester	1	1	100%
Daily Security Observability Report	1	1	100%
Daily Container Image Security Scan	1	1	100%
Engine Conformance (claude / codex / deepseek-harness dispatches)	1 each	1 each	100%
The Great Escapi (firewall-escape)	2	1	50%
Smoke Azure Codex / Azure PI / Codex / Copilot MAI / Copilot Sub Agents / Crush / Create-Cross-Repo-PR / DeepSeek Harness / Drive / Gemini / Goose / Update-Cross-Repo-PR / Work Queue / Unbloat Docs	1-2 each	1 each	20-100%
Smoke Copilot	5	1	20%

191 workflows had completed runs this window; 26 had >=1 agent-job failure (165 had zero).

Recommendations

1. Investigate the Engine Conformance Test Matrix simultaneous-failure pattern (claude + codex + deepseek-harness all failing "Assert engine conformance" within the same ~30s window). This is the second time this monitor has flagged it (first on 2026-10-15); it points at a shared fixture/assertion bug in the conformance dispatcher rather than three independent engine regressions.
2. An issue was opened this run for the "Redact secrets in logs" driver_exit chronic signature, given its 10+ day unaddressed recurrence across 2026-10-07 / 2026-10-15 / 2026-10-17. Given the security sensitivity of the redaction step, prioritize triage; verify whether a duplicate issue already exists since search tooling was unavailable to check.
3. Restore GitHub issue search access for this monitor - the integrity-policy block on search_issues/list_issues has now affected 5+ consecutive runs, materially weakening the Tracked vs. Novel classification and risking duplicate issue creation. This is itself worth a bug report against the monitoring pipeline / integrity-policy configuration.
4. No action needed on the fleet rate itself (5.27%, +1.5pp vs. last report, within normal range) or on the three disabled-but-silent daily/weekly workflows (Hippo Learn, GEO Optimizer, Slide Deck Maintainer) - their disabled_manually state already explains the gap; re-enable only if the disabling was unintentional.

Generated by 🩺 Agent Job Health Monitor · claude · agent · 535.6 AIC · ⌖ 0.94 AIC · ⊞ 6.9K · ◷

• expires on Oct 20, 2026, 3:42 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agent Job Health Monitor.

A newer discussion is available at Discussion #70043.