Skip to content

cli: stdout/stderr stream discipline is documented but unpinned by any test #953

Description

@bpowers

Problem

simlin-cli's contract is that STDOUT carries only the TSV result table and every diagnostic (warnings included) goes to STDERR. This is documented in two places:

  • print_formatted_error's rustdoc in src/simlin-cli/src/main.rs
  • the "Diagnostic reporting" section of src/simlin-cli/CLAUDE.md

An interleaved diagnostic on stdout corrupts a redirected or piped run (simlin simulate model.xmile > out.tsv). The MDL export warnings follow the same convention (#856; eprintln! at main.rs:736).

But nothing pins it. During the #919 severity-fix review on branch conveyor-engine, a reviewer mutation that routed warnings to stdout survived the entire test suite:

  • src/simlin-cli/tests/ does not exist
  • there is no process-level CLI harness that spawns the binary and captures the two streams separately

So neither #919's diagnostics nor #856's MDL-export warnings have their output stream verified by any test. The convention only holds as long as every future editor happens to read and honor the comments.

Why it matters

  • Correctness of piped runs: a regression silently corrupts TSV output for anyone redirecting stdout, the primary scripted-use mode of the CLI.
  • Mutation-proven gap: this is not hypothetical -- an actual stdout-routing mutation passed cargo test --workspace cleanly.
  • Documentation without enforcement rots: the rustdoc and CLAUDE.md contract have no teeth.

Component

  • src/simlin-cli (src/main.rs: print_formatted_error, MDL export warning path)

Suggested direction

Add a small process-level integration test harness for simlin-cli (e.g. src/simlin-cli/tests/integration/main.rs, consistent with the one-harness-per-crate convention from #706) that spawns the built binary and asserts on the two streams separately:

  1. Warning-producing simulate run: simulate --ltm test/conveyors/minimal_conveyor.xmile -- assert stdout is exactly the TSV table (no diagnostic lines) and stderr carries the warning diagnostics.
  2. MDL-export-warning case: a convert run that triggers engine: MDL export has no lossiness contract -- callers can't learn an export was degraded #856-style export warnings -- assert the warnings land on stderr and stdout/output file content is clean.

assert_cmd or a plain std::process::Command against CARGO_BIN_EXE_* both work; keep the fixture tiny to respect the test-time budget.

Discovery

Identified during the #919 severity-fix review on branch conveyor-engine: a mutation routing warnings to stdout survived the full test suite.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions