Skip to content

[Spec] Simplify MCO into an LLM-driven raw-output orchestrator #107

Description

@tt-a1i

Problem Statement

MCO 当前把通用多 Agent 编排和结构化代码审查混在同一条主链路中。Provider CLI 由子进程执行后,MCO 还会等待完整输出、解析固定 findings schema、校验字段、去重合并 finding、计算严重度与共识、生成 decision 和多种派生 artifact,最后才把结果写到 stdout。

从用户视角,这带来三个直接问题:

  1. 即使 Agent 已经能理解自然语言和 Markdown,仍被要求生成固定字段;少字段或格式偏差会把本来可读、可用的回答判为解析失败。
  2. stdout 往往要等整条链路完成才产生内容。长任务运行二十分钟以上时,外部重定向文件仍可能是 0 字节,看起来像进程卡死。
  3. 通用执行、审查、辩论和综合都受到 findings 数据模型约束,增加了子进程、artifact、解析、合并和兼容代码,却没有提升上层 Agent 理解结果的能力。

MCO 应回到清晰边界:负责选择 Agent、模型和调用实例,调度阶段、处理权限、超时、取消、状态和结果传输;回答内容由 LLM理解。MCO 不应解析或判断回答中的 finding,也不应对回答做隐藏的总结、截断或决策。

Solution

将 MCO 重构为 LLM 驱动、原始回答优先的多 Agent 编排器。

每个 Agent 可以正常输出自然语言或 Markdown。Adapter 只负责把 Provider CLI 的协议事件解包为正式回答的实时文本增量和最终回答,不解析回答语义。MCO 默认实时输出回答,按调用实例区分来源,并在最终 JSON、JSONL 和可选 artifact 中保留完整原文。

基本执行单位从 Provider 改为 Agent invocation:一个 invocation 由唯一实例 ID、Provider、Model 和可选别名组成。同一个 Provider 可以在一次任务中启动多个不同模型,也可以通过不同别名重复启动同一模型。

mco review 保留为兼容且易懂的薄入口。它默认只读,但不再注入 findings schema,也不改写用户显式提供的 prompt。chaindebatesynthesize 由 LLM读取前序原文完成;MCO 只组织阶段和上下文文件,不解析观点、投票或 finding。

默认不持久化 artifact。多阶段任务在临时目录中以 Markdown 文件传递完整上下文,任务结束后清理。显式启用持久化时,保存每个阶段、每个 invocation 的原始 Markdown、聚合阅读入口 result.md 和仅包含运行元数据的 run.json

任务状态只描述运行完整性:所有 invocation 成功为 complete;至少一个成功且至少一个失败为 partial;没有任何成功或发生确定的任务级配置错误为 failed。内容质量由调用 MCO 的上层 Agent 判断。

User Stories

  1. As an Agent caller, I want MCO to return each Agent's normal answer without a findings schema, so that I can understand the result directly.
  2. As an Agent caller, I want output to appear while the Provider is still running, so that a long task does not look stuck.
  3. As a shell user, I want redirected stdout to receive live answer content, so that a growing output file proves the task is making progress.
  4. As a single-Provider user, I want the Agent answer streamed without source decorations inside the answer, so that the content remains natural.
  5. As a multi-Provider user, I want the terminal to show which invocation currently produced output, so that concurrent responses remain understandable.
  6. As an automation author, I want a final --json envelope containing invocation identity, status and original output, so that I can consume results without parsing terminal presentation.
  7. As an automation author, I want --stream jsonl to emit source-qualified output_delta events, so that I can reconstruct every invocation's exact answer in real time.
  8. As an Agent caller, I want stdout reserved for answers or the selected machine protocol, so that progress and diagnostics do not corrupt redirected results.
  9. As an operator, I want progress, warnings and diagnostic messages on stderr, so that I can observe execution independently from answer delivery.
  10. As a Pi user, I want to invoke multiple Pi models in the same task, so that I can compare their independent reasoning.
  11. As a caller, I want every invocation to have a unique ID, Provider and Model, so that one model failure is not reported as a failure of every invocation using that Provider.
  12. As a CLI user, I want a repeatable --agent [alias=]provider:model option, so that multi-model tasks are concise and explicit.
  13. As an existing user, I want --providers to remain a compatible shorthand for one default-model invocation per Provider, so that simple existing calls continue to work.
  14. As a caller repeating the same model, I want distinct aliases to identify the invocations, so that their output, state and artifacts cannot collide.
  15. As a user with a malformed invocation specification, I want MCO to reject the entire task before starting any model, so that I do not incur partial cost from an avoidable configuration error.
  16. As a user of a model absent from an incomplete discovery list, I want MCO to attempt execution when the model cannot be conclusively rejected, so that discovery limitations do not block valid models.
  17. As a user running multiple models, I want runtime failures isolated to the affected invocation, so that healthy invocations can finish.
  18. As a caller receiving partial results, I want successful answers returned even when another invocation fails, so that useful work is never discarded.
  19. As an automation author, I want complete, partial and failed to be distinct states with distinct exit behavior, so that I can react to operational completeness without inferring content quality.
  20. As a user, I want an invocation-level stall timeout and hard timeout, so that one stuck model does not indefinitely block others.
  21. As a user, I want a task-level cancellation and optional global hard timeout, so that I can stop all remaining work intentionally.
  22. As a user, I want already-produced output preserved after timeout, cancellation or partial failure, so that diagnostics and useful partial answers survive.
  23. As a reviewer, I want mco review to default to read-only execution, so that review does not unexpectedly modify the repository.
  24. As a reviewer, I want my explicit review prompt forwarded unchanged, so that hidden schemas and templates do not alter the task.
  25. As a reviewer omitting a prompt, I want a short natural-language default review prompt, so that the convenience command remains useful.
  26. As a user enabling perspectives or division, I want all prompt additions explicit and visible in dry-run output, so that MCO never silently changes Agent instructions.
  27. As a chain-mode user, I want later LLMs to read earlier answers as source material, so that they can critique or extend them without MCO parsing the content.
  28. As a debate-mode user, I want LLMs to identify disagreements and challenge prior answers themselves, so that debate is model-driven rather than finding-driven.
  29. As a synthesis-mode user, I want a selected LLM to read all prior outputs and write a normal final answer, so that synthesis can use the LLM's full understanding.
  30. As a multi-stage user, I want prior outputs passed through Markdown files, so that long contexts are not duplicated into command arguments or silently truncated.
  31. As a multi-stage user, I want a manifest describing every input file and failed or missing invocation, so that downstream LLMs know whether their context is complete.
  32. As a multi-stage user, I want later stages to continue when at least one valid prior output exists, so that one failed model does not cancel useful analysis.
  33. As a multi-stage user, I want the overall task to remain partial when an earlier invocation failed, even if synthesis succeeds, so that a final answer cannot hide incomplete execution.
  34. As a security-conscious user, I want debate and synthesis to run read-only and treat prior outputs as untrusted reference material, so that model-produced instructions do not silently gain write authority.
  35. As a developer who wants code changes, I want write execution to be an explicit separate stage, so that analysis and mutation have a visible boundary.
  36. As a user running parallel write Agents, I want the MCO Skill to guide me to divide ownership and warn about overlapping edits, so that I can manage conflict risk without mandatory worktree orchestration.
  37. As a user, I want MCO to avoid creating or managing worktrees, so that repository layout remains under my control.
  38. As an artifact user, I want each invocation's answer stored as Markdown, so that the saved result is readable and preserves the original response.
  39. As an artifact user, I want a deterministic result.md organized by stage and declaration order, so that I can read and diff runs consistently.
  40. As an artifact user, I want live completion order in the terminal but stable declaration order in final documents, so that responsiveness does not make artifacts nondeterministic.
  41. As an artifact user, I want failed invocations represented by status and a concise error rather than fabricated answer text, so that missing output remains explicit.
  42. As an artifact user, I want final synthesis displayed first in result.md while retaining all stage outputs afterward, so that conclusions remain easy to find and auditable.
  43. As a privacy-conscious user, I want temporary context files deleted when a task ends, so that model outputs do not accumulate without consent.
  44. As a user requesting persistent artifacts, I want explicit save flags to preserve the task directory, so that retention is intentional.
  45. As a caller not saving artifacts, I want artifact_root to be null after completion, so that APIs do not return paths that have already been deleted.
  46. As an operator debugging a failed invocation, I want complete Provider stderr available in verbose mode or saved diagnostics, so that failures remain diagnosable without polluting the answer.
  47. As a user of a Provider CLI with structured events, I want Adapter transport decoding to expose only official answer deltas and final responses, so that protocol noise does not become user content.
  48. As a user, I want MCO never to infer token usage or status from natural-language answers, so that metadata is trustworthy.
  49. As an observability user, I want Provider-reported token usage retained when reliable and otherwise set to null, so that missing telemetry is explicit.
  50. As a maintainer, I want the findings parser, merger, consensus engine and decision model removed from the execution path, so that MCO has one coherent responsibility.
  51. As a maintainer, I want mco run and thin mco review to share a common execution engine, so that generic execution is not implemented through a review-specific function.
  52. As a maintainer, I want runtime types centered on Run, Stage, AgentInvocation, Output and Status, so that the code reflects the product domain.
  53. As a user of removed findings flags, I want a clear migration error, so that a breaking change is understandable rather than silently ignored.
  54. As a maintainer, I want old internal findings-oriented Python APIs treated as private and removable, so that the redesign is not constrained by undocumented imports.
  55. As a user, I want historical artifacts left untouched as ordinary files, so that the breaking release does not require destructive migrations.

Implementation Decisions

  • MCO's product boundary is orchestration and transport. It does not parse, normalize, merge, score or judge answer semantics.
  • The canonical execution unit is an Agent invocation with a unique invocation ID, Provider, Model and optional user alias.
  • Add a repeatable --agent [alias=]provider:model interface. An omitted alias is generated from Provider and Model. Repeated uses of the same Provider and Model require distinct aliases.
  • Keep --providers as a compatibility shorthand that creates one invocation per Provider using its default model. New documentation should teach --agent as the full interface.
  • Perform task-wide preflight validation before starting any invocation. Confirmed syntax, duplicate alias, unknown Provider, permission and scope errors fail the whole task without launching models. Models that cannot be conclusively validated because discovery is incomplete may run as unverified.
  • Runtime failures are invocation-scoped. They do not cancel sibling invocations unless the user cancels the task or a global hard timeout expires.
  • Define operational task states: complete when every required invocation succeeds; partial when at least one succeeds and at least one fails, times out or is cancelled; failed when none succeeds or task-level preflight fails.
  • Use exit code 0 for complete. Assign a dedicated nonzero exit code for partial, distinct from failed/configuration errors. The exact numeric value must be documented in the public CLI contract.
  • Always return successful invocation outputs, including when the task is partial or failed after producing partial data.
  • Default text mode streams official Agent answer content as it becomes available. A single invocation needs no source decoration. With multiple invocations, the terminal prints a source header when the active output source changes; it does not prefix or rewrite every answer line.
  • --json emits one valid final document after task completion. It is a thin transport envelope and may wait until completion.
  • --stream jsonl emits timestamped lifecycle and output_delta events carrying stage and invocation identity. Concatenating deltas for an invocation must reproduce its official answer exactly.
  • stdout contains only the selected answer/protocol surface. MCO progress, warnings and Provider diagnostics go to stderr.
  • Adapter parsing is limited to Provider protocol decoding: extracting official response deltas, final response, reliable usage metadata and terminal status. It must not parse content semantics.
  • Provider stderr is not answer content. Failures expose a concise normalized operational reason; full diagnostics are available through verbose output or persisted diagnostic artifacts.
  • mco review remains a thin command that defaults to read-only. It forwards an explicit prompt unchanged and no longer adds a findings schema. When no prompt is supplied, it may use a short natural-language review prompt.
  • Explicit coordination features may add instructions, but all additions must be visible in dry-run output.
  • Chain, debate and synthesis are LLM-driven stages. They provide prior stage outputs to later LLMs and preserve every stage's answer; MCO does not extract claims, findings or votes.
  • Multi-stage context is always file-backed. MCO writes prior official answers to Markdown files plus a context manifest, then gives the next invocation absolute file paths and a clear instruction to read them.
  • File-backed context directories are temporarily added to the invocation's read-only scope. A Provider unable to read context files fails explicitly with context_file_unsupported; MCO does not silently truncate or substitute a summary.
  • The context manifest records successful, failed and missing prior invocations. Later stages proceed when at least one valid input exists; they stop when no valid prior output exists.
  • Debate and synthesis always default to read-only and treat previous Agent outputs as untrusted reference material. Repository mutation requires a separate explicit write stage.
  • MCO does not create worktrees and does not prevent parallel write invocations. The bundled Skill must guide callers to partition ownership and warn about overlapping file edits and conflict risk.
  • Default execution uses a temporary task directory for raw answers and multi-stage context. It remains available for the duration of the task and is cleaned after completion, failure or cancellation.
  • Persistent artifact writing is opt-in through the existing explicit save/result-mode surfaces. Persisted output consists of per-stage/per-invocation Markdown files, a stable result.md, minimal run.json, and optional raw diagnostics.
  • result.md is deterministic by stage order and CLI declaration order, regardless of actual completion order. A final synthesis is presented first for readability, followed by the full stage record. Source headings are presentation metadata; answer bodies remain unchanged.
  • Temporary executions report artifact_root: null after cleanup. Persistent executions report valid retained paths.
  • run.json, JSON and JSONL contain only operational metadata: task, stage, invocation ID, Provider, Model, timestamps, duration, status, exit code, concise error, output or persistent output path, and reliable optional token usage.
  • Remove semantic fields and states including finding, severity, confidence, consensus, decision and content-derived INCONCLUSIVE.
  • Remove findings schema enforcement, parsing, validation, normalization, deduplication, consensus scoring, decision generation, findings persistence and passive finding lifecycle behavior from the new runtime.
  • Remove findings-oriented CLI surfaces and formats, including the findings command, findings artifacts, Markdown-PR rendering and SARIF generation, in a clearly documented breaking release.
  • Removed legacy flags fail with a concise migration message instead of being silently ignored.
  • Historical artifact files are not migrated or deleted.
  • Treat runtime Python modules and findings-oriented request/result types as internal implementation details. The new internal domain model centers on Run, Stage, AgentInvocation, Output and Status.
  • mco run and mco review share one execution engine. Review is a permissions/prompt-default preset rather than a separate semantic result pipeline.
  • Update the bundled Skill to teach invocation identity, task division for parallel writes, conflict warnings, output surfaces, partial status and LLM-driven stages.

Testing Decisions

  • The primary test seam is the public CLI boundary. Tests invoke the CLI with deterministic fake Agent adapters and assert externally visible stdout, stderr, exit codes, lifecycle events and artifacts. This is intentionally the highest practical seam and should carry most acceptance coverage.
  • Good tests assert observable behavior rather than private orchestration methods. They should prove that answers arrive, remain attributable and can be reconstructed; they should not assert internal thread layout or helper call counts.
  • The CLI seam covers invocation parsing, aliases, duplicate-model behavior, preflight fail-fast, model discovery uncertainty, runtime isolation, task states and exit codes.
  • The CLI seam verifies real-time text delivery by emitting delayed deltas from fake invocations and asserting that stdout changes before task completion.
  • The CLI seam verifies that single-invocation text is not decorated inside the answer and that multi-invocation terminal source switches remain understandable without changing stored output.
  • The CLI seam verifies stdout/stderr separation under success, warning, Provider diagnostic, timeout and cancellation scenarios.
  • The CLI seam verifies final JSON shape and confirms that each output matches the official Agent answer exactly.
  • The CLI seam verifies JSONL lifecycle ordering and confirms that concatenated output_delta values reproduce every invocation's answer exactly, including interleaved concurrent deltas.
  • The CLI seam exercises complete, partial and failed tasks, including one failed model among several invocations sharing the same Provider.
  • The CLI seam verifies that successful outputs survive sibling failure, task cancellation and timeout.
  • The CLI seam verifies per-invocation timeout isolation, global cancellation and signal cleanup.
  • The CLI seam verifies review prompt behavior: explicit prompts are unchanged, the default review prompt is natural language, and review stages receive read-only permissions.
  • The CLI seam verifies LLM-driven chain, debate and synthesis using fake Agents that read the provided context files and emit normal Markdown answers.
  • The CLI seam verifies partial-stage continuation, accurate context manifests, all-inputs-failed stopping behavior and preservation of overall partial status after successful synthesis.
  • The CLI seam verifies temporary context lifetime during execution and cleanup after complete, partial, failed and cancelled runs.
  • The CLI seam verifies persistent artifact opt-in, per-invocation Markdown fidelity, deterministic result.md, minimal run.json, failure sections and final-synthesis placement.
  • The CLI seam verifies that temporary runs return a null artifact root and persistent runs return valid paths.
  • The CLI seam verifies explicit migration errors for removed findings commands, flags and formats.
  • Thin Adapter contract tests remain only where Provider protocols differ. They feed representative CLI protocol events into an Adapter and assert official answer deltas, final response, operational status and reliable usage extraction.
  • Adapter contract tests must not contain findings payloads or test semantic interpretation.
  • Existing streaming, CLI JSON contract, quiet/output-mode, signal cancellation, execution-mode, Provider model, chain/divide and Adapter contract suites are prior art to consolidate into the new CLI-centric seam.
  • Existing findings parsing, consensus, confidence, decision, findings-memory and findings-format tests should be removed or replaced, not mechanically preserved behind compatibility helpers.
  • Include a regression test for the motivating failure: while a fake Agent runs longer than a polling interval and emits output incrementally, redirected stdout becomes non-empty before the task completes.

Out of Scope

  • MCO will not determine whether an Agent answer contains a valid bug, security issue or recommendation.
  • MCO will not merge duplicate claims or findings across answers.
  • MCO will not calculate severity, confidence, consensus or pass/fail decisions from content.
  • MCO will not silently truncate, sample or summarize context.
  • MCO will not provide implicit LLM compression. Any future compression must be a visible, explicit LLM stage with its own preserved output.
  • MCO will not create, manage or merge Git worktrees.
  • MCO will not prevent users from running parallel write invocations against one worktree.
  • MCO will not automatically resolve file conflicts caused by parallel write Agents.
  • MCO will not preserve undocumented internal Python findings APIs.
  • MCO will not migrate or delete historical findings artifacts.
  • MCO will not treat Provider CLI protocol logs as end-user answer content.
  • MCO will not add new content-aware output formats to replace Markdown-PR or SARIF in this redesign.

Further Notes

  • This is intentionally a breaking simplification rather than an additive compatibility layer. Keeping the old semantic pipeline behind the new output surface would preserve the latency, failure modes and maintenance burden that motivated the redesign.
  • The key trust boundary is explicit: MCO understands process state and transport protocols; LLMs understand answer content.
  • A Provider subprocess is still required when MCO orchestrates Provider CLIs. The redesign does not eliminate process startup, but it removes content-schema coupling and makes progress visible as soon as official answer deltas exist.
  • Multi-stage file-backed context should use task-private names derived from sanitized invocation IDs. Temporary and persistent directories must remain within the execution scope explicitly granted to downstream read-only stages.
  • The bundled Skill is part of the product behavior for parallel write guidance. It should recommend explicit ownership division and warn about overlapping edits without presenting worktrees as mandatory.
  • The confirmed primary acceptance seam is the CLI boundary with deterministic fake Agent adapters; Provider-specific protocol tests are supplemental and deliberately narrow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentReady for an agent to implement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions