Derive system monitor progress from resolved ActivitySim run plans

[jpn--]

Adding a model component or renaming a multiprocess phase currently requires editing sys_monitor.py as well as the model configuration. Otherwise the monitor can miss steps or stall. This change discovers the ordered models, phases, and expected worker counts from ActivitySim's resolved run_list.txt, then tracks execution using the main log.

Changes

• Replace hard-coded model/phase lists and worker-specific regexes with a validated run-plan adapter and generic execution tracking. Display the earliest step not satisfied by every planned worker, including workers that have not started yet.
• Replay existing log history once and then read appended complete lines incrementally. Support late attachment, partial writes, log replacement/truncation, and plans that appear or change after monitoring starts.
• Use saved resume breadcrumbs conservatively: credit already-simulated phases and workers skipped by a last-checkpoint resume, but do not reuse old worker completion for named-checkpoint reruns. Leave unresolved resume positions explicit until execution evidence arrives.
• Preserve existing CLI options and CSV columns; add --run-list. --tail-bytes now controls read chunk size instead of discarding earlier history. Distinguish unknown/waiting/resuming/running/finalizing/finished/failed states; parent exit alone never implies success.
• Add small synthetic run-plan/log fixtures and usage/compatibility documentation.

Validation

• 40 focused tests passed on Python 3.10.20, covering new steps, renamed phases, staggered workers, late attachment, named and last-checkpoint resumes, single-process execution, failures, partial writes, log replacement, and CLI/CSV behavior.
• Ruff 0.15.6 lint and format checks passed.
• Repository-configured mdformat checks passed (its configured plugins emit a duplicate front-matter-renderer warning).
• git diff --check and CLI help smoke check passed.
• Replayed an existing local multiprocess log/plan and accounted for all 33 model steps; checked the run-list and runner log formats against ActivitySim v1.5.1 source.
• No model simulation was run; the change does not alter model execution or configuration. Tests used the existing Python environment without modifying the dependency lockfile.

Limits

run_list.txt is an informational text format, so parsing is isolated and rejects ambiguous plans. Files must belong to the same invocation; the current artifacts lack a shared run identifier. Some single-process entry points do not emit a run list, so tracking requires an explicitly supplied resolved plan in those cases. Unsupported/missing artifacts leave progress unknown while resource sampling continues. Identical repeated model names are unsupported.

The proposed stable replacement is tracked in ActivitySim/activitysim#1120. This PR does not assume a future manifest schema or change ActivitySim.

Related: #21 (comment). This is a separate implementation PR based on upstream main; it does not include agent-instruction changes.

[Copilot]

Copilot review overview

🔵 Needs a closer look

Fix robust log replacement detection and support colon-formatted durations before approval.

Review effort: Lite
Findings: None

What changed in this PR

Updates ActivitySim system monitoring to derive progress from resolved run plans and execution logs instead of hard-coded model lists.

Changes:

• Adds validated run-plan parsing, resume handling, and generic worker tracking.
• Supports incremental log replay, late attachment, truncation, failures, and single-process runs.
• Adds CLI support, fixtures, tests, and monitoring documentation.

Two moderate issues remain in log replacement detection and parsing durations over 60 seconds.

File	Description
tests/​test_sys_monitor.py	Adds comprehensive monitor behavior tests.
tests/​fixtures/​sys_monitor/​staggered.log.txt	Provides staggered-worker log data.
tests/​fixtures/​sys_monitor/​single_process.txt	Provides a single-process plan fixture.
tests/​fixtures/​sys_monitor/​single_process.log.txt	Provides single-process log data.
tests/​fixtures/​sys_monitor/​run_list.txt	Provides a multiprocess run-plan fixture.
tests/​fixtures/​sys_monitor/​resumed.log.txt	Provides resumed execution log data.
tests/​fixtures/​sys_monitor/​resume_named.txt	Provides a named-checkpoint resume fixture.
tests/​fixtures/​sys_monitor/​resume_latest.txt	Provides a last-checkpoint resume fixture.
src/​lighthouse/​sys_monitor.py	Implements plan parsing, tracking, log reading, and CLI updates.
README.md	Links to monitoring documentation.
docs/​sys-monitor.md	Documents usage, statuses, resumes, and limitations.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.