Real-LLM (qwen3.5:4b) doesn't reproduce canonical NAFEMS W-AR-02 firing pattern on aero COU1

[cloudronin]

Observation

The canonical NAFEMS-aero narrative (encoded in the hand-crafted tests/fixtures/extract/aero-cou1-imported.jsonld and exercised by TestAeroWeakenerPipelineFromFixture::test_cou1_w_ar_02_fires_on_gap_factors):

• COU1 (Accepted, with level-gap factors) → W-AR-02 fires on each level gap
• COU2 (Not Accepted, no level-gap factors) → W-AR-02 stays at zero
• diff(cou1, cou2) → highlights the W-AR-02 divergence

The May 25 real-LLM e2e run with ollama/qwen3.5:4b did NOT reproduce this on either COU:

	Expected (hand-crafted)	Observed (qwen3.5:4b)
COU1 firings	W-AR-02 (n≥1) + others	W-AL-02, W-AR-05×2, W-CON-04, W-EP-04, W-NASA-02, W-NASA-03, W-NASA-06, W-ON-02 — no W-AR-02
COU2 firings	(mostly clean)	W-AL-02, W-AR-05×3, W-CON-04, W-EP-04×19, W-NASA-01, W-NASA-04, W-ON-02, W-SI-02 — no W-AR-02
Diff highlight	W-AR-02 divergence	W-SI-02 divergence (1 total)

The LLM is producing semantically reasonable extractions — both COUs surface real weakener firings — but the factor-level structure that drives W-AR-02 specifically isn't being extracted reliably. W-AR-02 fires on level gaps (required > achieved) — qwen is likely either:

1. Not extracting the requiredLevel / achievedLevel pair at all (so the rule has no inputs)
2. Extracting them but always equal (no gap)
3. Extracting them but missing the pack-specific edges that trigger this particular pattern

Why this matters

The W-AR-02 pattern on cou1 is the headline divergence in the NAFEMS demo deck (per docs/v0.5-morrison-deltas.md and the pre-built fixtures). If the production extract pipeline doesn't reliably hit it on the canonical evidence, the demo narrative falls apart whenever someone runs uofa extract instead of using the pre-built jsonld.

Suggested investigation

1. Inspect factor-level extraction accuracy on aero COU1: run dev/tools/scripts/score_extraction.py --pack nasa-7009b --case cou1 --model ollama/qwen3.5:4b and diff the factor_level_accuracy against the W-AR-02-firing prerequisite. The accuracy log already tracks this metric; the question is whether the levels qwen extracts pin to required==achieved or actually surface gaps.
2. Compare prompts: the V&V 40 prompt asks for required_level and achieved_level per factor. Is qwen returning these reliably for NASA cou1?
3. Try a larger model as a control (claude-sonnet-4-6 or gpt-4o) to see whether this is a qwen-specific limitation or a prompt/spec issue.
4. Consider eval-loop iteration: the v4-kv-adaptive-ctx prompt got factor_level_accuracy = 0.947 on nasa cou1, which is high but not perfect. The misses might be concentrated on the few factors that would trigger W-AR-02.

Test-side context

The real-LLM e2e in tests/test_aero_full_pipeline_e2e.py::TestAeroFullPipelineE2ERealLLM was initially written to assert the canonical pattern (commits f78675b → d90c52c). After today's run revealed the divergence, the assertions were loosened to "produces ≥1 firing" and "diff detects ≥1 divergence" — chain-plumbing, not semantic correctness. This issue tracks the gap so the loosened assertions don't paper over a real quality regression.

The hand-crafted TestAeroWeakenerPipelineFromFixture continues to guard the canonical pattern against a known-good input.

Discovered during

UOFA_RUN_REAL_LLM=1 python -m pytest tests/test_aero_full_pipeline_e2e.py::TestAeroFullPipelineE2ERealLLM — May 25, 2026 demo-prep run.

[cloudronin]

Update: Claude data point (2026-05-25 follow-up to #23 fix)

After fixing the Anthropic empty-extraction bug in 7b0f41c, I was able to run the same e2e against anthropic/claude-sonnet-4-5. Claude also doesn't fire W-AR-02 on aero COU1:

	COU1	COU2
Canonical (hand-crafted fixture)	W-AR-02 fires	(mostly clean)
qwen3.5:4b (original observation)	W-AL-02, W-AR-05, W-CON-04, W-EP-04, W-NASA-02, W-NASA-03, W-NASA-06, W-ON-02 — no W-AR-02	W-AL-02, W-AR-05×3, W-CON-04, W-EP-04×19, W-NASA-01, W-NASA-04, W-ON-02, W-SI-02 — no W-AR-02
claude-sonnet-4-5 (new data)	W-AL-02, W-AR-05×3, W-CON-04, W-NASA-02, W-NASA-03, W-NASA-06, W-ON-02, W-SI-02 — no W-AR-02	W-AL-02, W-AR-05×3, W-CON-04, W-EP-04×5, W-NASA-02, W-NASA-03, W-NASA-06, W-ON-02 — no W-AR-02

Reframing this issue: this isn't qwen-specific quality drift. It's a systemic gap between the canonical demo narrative (hand-crafted fixtures the spec writers built to demonstrate W-AR-02) and what any LLM extracts from the actual evidence files.

The two non-trivial overlaps between qwen and Claude:

• Both fire W-AL-02, W-AR-05, W-CON-04, W-NASA-02, W-NASA-03, W-NASA-06, W-ON-02 on COU1
• Both miss W-AR-02 entirely
• Both fire W-EP-04 heavily on COU2 (5–19 hits) but NOT on COU1

This is convergent: two unrelated models extract similar weakener profiles from the same evidence. The W-AR-02 absence is therefore not LLM variance — it's that the evidence files don't carry the data shape W-AR-02 requires (required_level > achieved_level on assessed factors). The hand-crafted fixture was constructed specifically to have those gaps; the real evidence apparently doesn't.

Recommended investigation reframing:

1. Compare the hand-crafted aero-cou1-imported.jsonld factor structure against what Claude/qwen extract from aero-evidence-cou1/. Specifically: does the evidence narrative describe required-vs-achieved level gaps that W-AR-02 needs, and if so, are LLMs failing to extract them, or is the gap not in the source text at all?
2. If the source text doesn't carry the gaps: either edit the evidence narrative to make the gaps explicit, or shift the demo narrative to a weakener pattern that DOES emerge naturally (W-EP-04 is a strong candidate — both models fire it heavily on COU2).
3. If the source text does carry the gaps but LLMs miss them: tighten the prompt for the per-factor required_level / achieved_level extraction.

This expands the scope of the issue from "qwen quality regression" to "demo narrative ↔ real-evidence ↔ extraction-pipeline alignment work."