Auto-suppress TypeSpec validation failures by harvesting TypeSpec Validation - All runs

[iscai-msft]

Summary

Adds a scheduled and manually triggerable agentic workflow that applies simple, well-understood TypeSpec suppressions automatically.

Rather than recompiling every spec, it harvests the logs of an existing "TypeSpec Validation - All" (TSV-All) run. TSV-All already runs 4×/day on both main and typespec-next, so there is no reason to pay for a second full compile — and harvesting guarantees the agent reacts to exactly what CI observed.

TSV-All itself is not modified. This pipeline only reads its logs.

File	Role
typespec-suppression-harvest.yaml	Deterministic. Resolves a TSV-All run, downloads its ubuntu job logs, extracts per-folder failures. Never edits the repo.
src/harvest-tsv-failures.js	Parses ##[group]Validating <folder> blocks out of the logs and classifies each failure.
typespec-auto-suppression.md (+ generated .lock.yml)	Agentic. Runs once per branch, reads the harvest report, opens a single PR.

Why harvest instead of recompile

TSV-All shards specs across 3 ubuntu + 3 windows jobs per ref. An independent scan would have duplicated all of that. Harvesting instead:

• costs about a minute of log parsing instead of a full repo-wide tsp compile
• can never disagree with CI, because it is CI's output
• still yields one combined view per branch, since all three shards of a ref are parsed together

The parser was validated against real TSV-All logs (run 32135569949): 196 spec folders parsed, 62 already-suppressed correctly skipped. Synthetic failures injected into that log were classified correctly into eligible / infrastructure / unclassified.

One PR per branch

The harvest job runs per ref (main, typespec-next), then dispatch triggers the agent once per ref — so all three shards are united into a single PR per branch, not one per shard. Ubuntu only, as requested; a Windows-only failure should be fixed, not auto-suppressed.

Schedule

Runs weekly, Mondays 03:00 UTC, shortly after the TSV-All 00:00 scheduled run:

schedule:
  - cron: "0 3 * * 1"

Weekly rather than daily on purpose: suppression churn is slow-moving, and daily runs would open PRs faster than they can be reviewed. For daily, change the cron to "0 3 * * *".

How to invoke it manually

Harvest the latest completed scheduled TSV-All run, both branches:

gh workflow run typespec-suppression-harvest.yaml --repo Azure/azure-rest-api-specs

Harvest a specific TSV-All run — the main reason you'd trigger this by hand, e.g. right after a TSV-All run went red:

gh workflow run typespec-suppression-harvest.yaml --repo Azure/azure-rest-api-specs \
  -f run-id=32135569949

Limit to one branch, or skip opening PRs entirely and just produce the failure report:

gh workflow run typespec-suppression-harvest.yaml --repo Azure/azure-rest-api-specs -f refs=default
gh workflow run typespec-suppression-harvest.yaml --repo Azure/azure-rest-api-specs -f dispatch-agent=false

From the UI: Actions → TypeSpec Auto-Suppression - Harvest → Run workflow.

Re-run only the agent against an existing harvest:

gh workflow run typespec-auto-suppression.lock.yml --repo Azure/azure-rest-api-specs \
  --ref main -f target-ref=main -f harvest-run-id=<harvest run id> -f ref-key=default

Inputs:

Workflow	Input	Values	Meaning
Harvest	run-id	TSV-All run ID	Run to harvest (default: latest completed scheduled run)
Harvest	refs	both (default), default, next	Which branches to harvest
Harvest	dispatch-agent	true (default), false	Whether to trigger the agent afterwards
Agent	target-ref	main / typespec-next	Branch harvested, and the PR base
Agent	harvest-run-id	run id	Harvest run whose artifact to consume
Agent	ref-key	default / next	Matrix key used to select the artifact

Note: only the scheduled TSV-All trigger covers typespec-next, so harvesting next from a push-triggered run will correctly report that no matching jobs exist.

Two suppression mechanisms

The agent picks the right mechanism per failure rather than forcing everything into suppressions.yaml.

Inline #suppress in the .tsp source — preferred whenever the compiler reported an exact location, following the pattern in #45367:

#suppress "@azure-tools/typespec-azure-core/no-openapi-client-extensions" "Existing API contract: retain this x-ms-* OpenAPI extension for compatibility."
@doc("The resource instance.")
@Http.bodyRoot

This is scoped to the single declaration that deviates, instead of blanket-suppressing a whole folder.

A diagnostic qualifies when its rule id comes from a TypeSpec linter package (@azure-tools/... or @typespec/...). There is deliberately no hardcoded rule allowlist — one would go stale every time a linter adds a rule.

The rule id is the gate, not severity. #suppress takes a rule id as its argument, so a diagnostic without one cannot be suppressed inline even in principle. Severity turns out to be useless here: TSV compiles with --warn-as-error, so every linter warning prints as error — across every run harvested so far there were zero warning-severity diagnostics. A severity-based gate would have suppressed nothing, ever. Rule-less compile errors are still parsed (as rule: null) so that they disqualify their folder. The agent is additionally told to leave any diagnostic that looks like a real API design problem (ARM resource shape, provisioning state, envelope properties, versioning) for a human.

Entry in specification/suppressions.yaml — used only for failures that belong to a folder as a whole and have no single source location: FolderStructure/MustUseV2, Compile/ExtraSwagger, SdkTspConfigValidation.

The harvest step parses each <file>:<line>:<col> - error|warning <rule> diagnostic out of the logs, so the agent gets exact insertion points rather than having to guess.

Importantly, a folder is only inline-eligible when every diagnostic in it is inline-suppressible. A folder mixing a suppressible warning with a real compiler error goes to a human, since suppressing the warning would leave it failing anyway. This is covered by a test case.

Safety constraints

• Only specification/suppressions.yaml and .tsp files under specification/ may be edited, and only to add a suppression. tspconfig.yaml, package.json, eng/, and workflows are off-limits.
• Every added line must be a #suppress directive, a suppressions.yaml entry, or the result of tsp format on a folder that was edited. The agent must read its own diff line by line and revert anything that changes a model, operation, property, decorator, or doc comment.
• Existing suppressions may only be added to, never rewritten or deleted, in either mechanism.
• A diagnostic with no linter rule id is never eligible — a plain compile error means the spec is broken, and it is surfaced to a human.
• Infrastructure flake (ETIMEDOUT, npm ERR!, runner shutdown, and similar) is detected and reported as "needs a rerun", never suppressed.
• The harvest step only pre-classifies; the agent must confirm each one against the log excerpt.
• PRs open as draft so they don't ping every affected spec author.
• gh-aw's built-in protected-files policy independently blocks package.json and lockfile edits.
• The agent settles one folder at a time: apply the suppressions, run tsp format, re-run npx tsv, and react to the result. TSV has its own format check, so an inserted directive that does not match the formatter's layout would otherwise trade a suppression failure for a formatting one.
• That loop is capped at three passes per folder. Re-validation is cheap (6s for a small folder, 19s for AppService), so the cap is not about compute — it is about scope. Unbounded iteration turns "suppress the reported set" into "make this folder pass at any cost", which is the pressure that produces over-broad suppressions. Folders that hit the cap are listed in the PR body under "Needs a second pass" so a reviewer sees them without reading the diff.
• Across every pass the agent may only suppress a diagnostic the compiler just reported — never one it expects to fire next, and never by widening an existing directive's scope.
• The agent must verify with get-suppressions and re-run npx tsv on every folder it touched before opening a PR, and emits a no-op rather than an empty or broken PR.

Validation gate

Generated PRs must still pass TypeSpec Validation - All — all three shards on both Ubuntu and Windows. The prompt explicitly reminds the agent that one green shard proves nothing about the others.

Validation against real runs

The parser was developed against hand-written logs, which turned out to prove very little — every real run since has exposed a distinct bug class. Each was found by cross-checking parser output against the raw logs and fixed:

Run	Ref	Folders	Failures	Eligible	What it caught
31362706542	typespec-next	589	29	29	ANSI colour codes, UTF-8 BOM, --warn-as-error
31137822383	default	589	1	0	correctly rejects an uncommitted regenerated service.yaml
31137822383	typespec-next	588	0	0	clean no-op
31079418235	typespec-next	589	71	66	failures hidden behind an existing Suppressed: line

The highest-impact bug was ANSI colour codes: the compiler colourises diagnostics, so file:line:col is never contiguous plain text and the regex matched zero real diagnostics while passing every synthetic test. The most recent one dropped 8 real failures because a folder can carry a suppression for one rule and still fail on another, and the parser checked Suppressed: before checking for failure.

End-to-end proof

Parsing correctly is only half the claim — the suppressions also have to actually fix the folders. For run 31079418235 this was verified by checking out typespec-next at 7316bb7 (the commit that run validated) with its package-lock.json regenerated to next versions exactly as the job does, since these rules only fire against prerelease compiler versions:

Folder	Diagnostics	npx tsv after
automation/Automation.Management	1	exit 0
.../Reservations/Reservations	2, across 2 files	exit 0
.../Compute/ComputeDisk	2, across 2 files	exit 0
.../Microsoft.Web/AppService	113, across 3 files	exit 0

Every folder reproduced its reported failure at the exact reported line, and reached npx tsv exit 0 after suppressing. Three behaviours surfaced that the prompt now handles explicitly:

• Suppressing alone is not enough. Reservations and ComputeDisk hit 0 compile errors but still failed the Format rule. Hence the tsp format step.
• Walking up to find the insertion point can overshoot. In AppService one directive landed above an @@override(...) belonging to the previous declaration, leaving the original diagnostic firing. The placement rule now names the terminators to stop at.
• Fixing the harvested set can reveal masked diagnostics. AppService surfaced models.tsp:8627, absent from the report because the compiler stopped before reaching it. Suppressing it too brought the folder to exit 0. This is why the agent iterates rather than making a single pass: it distinguishes a repeated diagnostic (a placement mistake to fix) from a new one (a masked diagnostic to suppress if eligible).

The 5 rejected folders in that run are all specification/cognitiveservices/* failing tsp format — a reformat commit, which no #suppress can fix, so refusing them is correct.

Note that harvest reads only the ubuntu shards. A Windows-only failure should be fixed rather than hidden, so it is deliberately never auto-suppressed.

Notes for reviewers

• .lock.yml is generated by gh aw compile; edit the .md and recompile.
• The Protected Files check fails because this adds files under .github/* — the expected known failure for workflow changes.
• The typespec-next dispatch will fail with "workflow not found" until this also lands on typespec-next, since gh workflow run --ref requires the lock file to exist on that branch. Noted in a comment in the harvest workflow.

[github-actions]

Next Steps to Merge

⌛ Please wait. Next steps to merge this PR are being evaluated by automation. ⌛

Comment generated by summarize-checks workflow run.

[microsoft-github-policy-service]

Hi @iscai-msft. Your PR has had no activity for 14 days and is marked as stale. If no further updates are made within 14 days, this PR will automatically be closed. To keep this PR active, please leave a comment as the PR author or remove the no-recent-activity label.