Parent
Part of #46.
Relationships
Depends on #50, #52, #54, #55, #56, #57, #58, #59, #60 and #62.
Goal
Create a representative orchestration scenario suite that lets v1.7 be aggressive about deleting prompt prose while detecting real behavioural regressions.
The suite should test outcomes and control boundaries rather than exact chain-of-thought or prompt wording.
Required scenarios
Cover at least:
Simple change
A straightforward change that can move through planning → implementation → review → completion without unnecessary fan-out.
Design-heavy change
A change where proposal/spec/design/tasks dependencies matter and custom artifact eligibility must be respected.
Parallel-worthy implementation
A change with genuinely independent implementation surfaces where the Coordinator can choose parallel lanes and the runtime validates the assignments.
Non-parallel implementation
Multiple tasks exist, but a coherent single Implementer is a valid/better model choice. This protects against accidentally turning v1.7 into a deterministic fan-out scheduler.
Remediation loop
Reviewer identifies an implementation defect → targeted remediation → re-review with stable finding identity.
Planning correction
A review/remediation finding points back to an approved planning artifact and routes to the appropriate planning role rather than being papered over in implementation.
Resume/recovery
Start from partially completed durable OpenSpec state and reconstruct legal actions/Todo/current dispatch state without relying on stale ephemeral projections.
Material blocker
The model recognizes a genuine unresolved decision/evidence gap and uses the intended escalation/blocking path.
Background/parallel failure
A specialist dispatch fails or returns malformed output and the workflow recovers without corrupting durable state or exceeding capacity.
Test layers
Use a pragmatic mix:
- deterministic fixtures for status/lifecycle/assignment/Todo/progress mechanics;
- integration tests for host/tool/dispatch boundaries;
- model-driven eval scenarios only where model judgement is the thing under test.
Avoid making normal CI depend on expensive nondeterministic model runs if a deterministic contract test proves the same property.
Assertions
The suite should specifically catch over-determinism as well as under-specification. For example:
- parallel-capable does not mean parallel-required;
- multiple eligible actions do not imply a hard-coded preferred action;
- runtime rejects illegal assignments but does not repartition them;
- Todo/progress failures do not become workflow failures;
- review evidence semantics survive prompt simplification.
Acceptance criteria
Out of scope
- Measuring production latency/cost itself.
- Requiring one specific model's exact prose or reasoning path.
Parent
Part of #46.
Relationships
Depends on #50, #52, #54, #55, #56, #57, #58, #59, #60 and #62.
Goal
Create a representative orchestration scenario suite that lets v1.7 be aggressive about deleting prompt prose while detecting real behavioural regressions.
The suite should test outcomes and control boundaries rather than exact chain-of-thought or prompt wording.
Required scenarios
Cover at least:
Simple change
A straightforward change that can move through planning → implementation → review → completion without unnecessary fan-out.
Design-heavy change
A change where proposal/spec/design/tasks dependencies matter and custom artifact eligibility must be respected.
Parallel-worthy implementation
A change with genuinely independent implementation surfaces where the Coordinator can choose parallel lanes and the runtime validates the assignments.
Non-parallel implementation
Multiple tasks exist, but a coherent single Implementer is a valid/better model choice. This protects against accidentally turning v1.7 into a deterministic fan-out scheduler.
Remediation loop
Reviewer identifies an implementation defect → targeted remediation → re-review with stable finding identity.
Planning correction
A review/remediation finding points back to an approved planning artifact and routes to the appropriate planning role rather than being papered over in implementation.
Resume/recovery
Start from partially completed durable OpenSpec state and reconstruct legal actions/Todo/current dispatch state without relying on stale ephemeral projections.
Material blocker
The model recognizes a genuine unresolved decision/evidence gap and uses the intended escalation/blocking path.
Background/parallel failure
A specialist dispatch fails or returns malformed output and the workflow recovers without corrupting durable state or exceeding capacity.
Test layers
Use a pragmatic mix:
Avoid making normal CI depend on expensive nondeterministic model runs if a deterministic contract test proves the same property.
Assertions
The suite should specifically catch over-determinism as well as under-specification. For example:
Acceptance criteria
Out of scope