Discussion: Output Format section after the classifier hook — still all load-bearing?

[akfrick-de]

Discussion: Output Format section after the classifier hook — still all load-bearing?

Repo: danielmiessler/PAI (or current public fork)
File: PAI/PAI_SYSTEM_PROMPT.md → section ## Output Format (CRITICAL — MANDATORY — ZERO EXCEPTIONS)
Type: Question / discussion (not a bug report, not a hard PR proposal)

Context

Running a private fork of PAI 5.0 with the Sonnet classifier hook (hooks/PromptProcessing.hook.ts) actively writing MODE: MINIMAL|NATIVE|ALGORITHM and TIER: E1..E5 to additionalContext on every prompt. The classifier-driven mode-selection seems to be doing the heavy lifting structurally — the model reads MODE/TIER and picks the template.

That made me look hard at the existing ## Output Format section in PAI_SYSTEM_PROMPT.md. It's about 15 lines and reads (paraphrased):

1. Authority statement: "highest enforcement priority", "CRITICAL FAILURE regardless of underlying correctness"
2. Every-response statement: "every response uses exactly one output format from CLAUDE.md"
3. Eight "Hard requirements" bullets (header first, fields populated, closing-line last, no freeform prose around/between fields, no conversational sentences as content, exploratory/recommendations/opinions/plan-presentations/acknowledgments all use format, answering a question is not a license to drop format, apologies/errors also use format)
4. "Self-check before emitting any response" line
5. "Recurring failure pattern" reflection about freeform-prose-drift on interesting topics

Observation

Many of the 8 Hard-requirement bullets restate the same underlying principle: use the format, all the time, no exceptions. The "Recurring failure pattern" paragraph reads as post-hoc reflection rather than enforceable rule — it tells the model what mistake it tends to make, without specifying an action different from "use the format".

In my fork, with the classifier writing the explicit MODE line, I notice:

• The mode-selection itself is no longer ambiguous (the classifier decided)
• Format-compliance failures I've seen this session were drift inside a chosen mode (e.g., freeform prose before the header), which is mostly addressed by ONE bullet: "Mode header is the first visible token, closing line is the last"
• The "every-response-including-follow-ups" qualifiers feel like they were added against specific past failures

Two readable failure-history hypotheses

A. Pre-classifier history. Before the classifier hook existed, the section had to do double duty: (1) tell the model to choose a mode, (2) tell it to obey the chosen mode's template. The 8-bullet enumeration was the model's only enforcement. Now with the classifier, half of that work is structurally upstream — but the section was never retrenched.

B. Failure-pattern accretion. Each bullet corresponds to a specific mistake observed across versions: "answering a question is not a license to drop format" → seen models drop format when answering direct questions. "Apologies use the format too" → seen apologies revert to prose. Removing any one bullet risks re-introducing that mistake on the next model version.

If (B) is the actual reason, the section is doing more than it looks — it's a checklist of failure-modes-seen-in-the-wild and each bullet is load-bearing against a specific drift mode. Trimming it would be a regression.

What I would love to know

1. Is hypothesis (A), (B), neither, or both?
2. For each Hard-requirements bullet, is there a remembered failure case that drove it? If yes, the bullet earns its weight independent of the others. If no, it's restating defaults.
3. With the classifier hook now writing MODE/TIER, does the Output Format section want to evolve — or does experience say "leave it, the classifier is one more belt, not a replacement for any suspender"?

Sketch of a leaner alternative (for discussion, not a PR)

Not proposing this — putting it here only so the question has something concrete to argue against:

**This rule has the highest enforcement priority. Every response uses exactly one output format: ALGORITHM, NATIVE, or MINIMAL (templates in CLAUDE.md). Violating is CRITICAL FAILURE regardless of underlying correctness.**

- Mode header is the first visible token, closing line is the last.
- All content lives inside template fields — no freeform prose around or between them.
- Follow-ups, questions, apologies, errors — same rule, no exceptions.
- Self-check pre-emit: header first, closing line last, fields populated.

That removes the explicit enumerations and the recurring-failure-pattern paragraph. My instinct says "smaller, sharper". Your data may say "the long version is what stops the recurring drift".

What I'm doing in the meantime

Keeping the section untouched in my fork. If the data on this side starts to suggest the long version is doing work, I'll know.

If you have a moment to share the failure history, the call gets a lot easier for everyone running a PAI fork.

---

Draft authored 2026-05-18 during a BitterPillEngineering audit on a private fork. Not yet posted upstream — staged for review before submitting.
*Draft reviewed 2026-05-18 by me (Arne Frick). Please bear with me, as this is my first Github Issue / Question report at all :-) *

[jmmarkiewicz]

Great first issue, Arne — and a genuinely useful analysis. BitterPillEngineering doing its job.

Short answer: both (A) and (B). The section predates the classifier hook and was never retrenched (A), AND each bullet maps to a specific observed failure (B).

Some concrete history:

• "answering a question is not a license to drop format" — recurring Sonnet-era failure where direct Q&A would revert to conversational prose
• "apologies/errors also use the format" — Opus would break format specifically when apologizing for a mistake, switching to freeform "I'm sorry, I should have..."
• "no conversational sentences masquerading as content" — models would technically use the template but fill fields with prose preambles ("Let me explain what I found...")
• The "recurring failure pattern" paragraph — still load-bearing against Opus drift on interesting topics; when the work is genuinely engaging, format compliance is the first thing to slip

Your 4-bullet sketch is cleaner and I think it covers bullets 1-3 and the self-check well. The one thing I'd keep from the original is the "recurring failure pattern" paragraph — it's doing different work than the structural rules. The structural rules say what; that paragraph says when you're most likely to forget. That metacognitive nudge has measurably reduced drift in my runs.

With the classifier hook active, you're right that mode selection is now deterministic upstream. The remaining job of the Output Format section is mode compliance — staying inside the template once the mode is chosen. That's a smaller job, and your leaner version scopes it correctly.

TL;DR: trim with confidence, but keep the failure-pattern reflection. And if you do trim, watch your next 20 sessions for regression on the specific cases above before calling it clean.

[akfrick-de]

@jmmarkiewicz — thank you. The Selection-vs-Compliance distinction is the conceptual hook I hadn't isolated; framing it that way scopes the section's remaining job correctly. And the four model-version-failure pairs are exactly the earned-weight evidence I was looking for.

Plan I'm taking from this:

1. Apply the four-bullet sketch, keep the recurring-failure-pattern paragraph verbatim — your point that what and when you're most likely to forget are different categories of nudge is the load-bearing piece I would have lost.
2. Instrument the three named failure modes — Q&A-prose-revert, apology-prose-break, prose-preamble-in-fields — as testable hypotheses, and watch the next 20 sessions for regressions.
3. If any of the three drifts back, walk back the specific bullet rather than the whole structure.

I'll post back once the sessions are in. If others running PAI forks have parallel data, the failure-modes-per-model-version mapping seems worth a shared corpus — that's the kind of knowledge that decays fast without contributors.