OpenClaw Two-Host Autonomous Fleet Goal

[mfethe1]

Goal

Make OpenClaw manage Rosie and Lenny as a two-host autonomous fleet. The system should route Telegram work to the best available host, balance Agent Runs across Rosie and Lenny, monitor progress, recover stale hosts through peer recovery, and report status back to Telegram without requiring manual tagging or babysitting.

Measurable outcomes

1. Fast routing: untagged Telegram work becomes Claimable Work and is assigned within 5 seconds when at least one host is healthy.
2. Better host choice: the Fleet Controller chooses Rosie or Lenny based on health, active run count, repo availability, provider support, and recent failures.
3. Run supervision: the Run Orchestrator detects stuck, failed, completed, or quiet Agent Runs and decides whether to wait, reassign, recover, or ask the operator.
4. Peer recovery: if Rosie is stale, Lenny can recover Rosie; if Lenny is stale, Rosie can recover Lenny. Recovery runs in stages, not all at once.
5. Telegram status loop: Telegram gets clear updates for assigned, started, blocked, recovered, reassigned, completed, and failed states.

Child tasks

• Add live host capacity to Rosie/Lenny heartbeats
• Replace first-eligible routing with scored host selection
• Trigger routing immediately from Telegram, heartbeats, run events, and recovery results
• Stage recovery: health-check -> service-restart -> heartbeat wait -> index-update if needed
• Add Telegram summaries for assignment, recovery, reassignment, and completion
• Add fleet dashboard/readout for current host load and active Agent Runs

Current implementation slice

The first slice is to add host capacity to heartbeat payloads and make _select_host score Rosie vs Lenny instead of choosing the first eligible host. This gives the Fleet Controller a real balancing signal before the later routing, recovery, status-loop, and dashboard slices are added.

[mfethe1]

Progress update from local implementation work:

• Added live host capacity to hostd heartbeats: active run count, max parallel runs, and available run slots.
• Replaced first-eligible unpinned host routing with scored Fleet Controller selection using load, active runs, recent failures, slots, provider capability breadth, then host id.
• Added event-triggered task-manager routing from controller NATS heartbeat callbacks, run event/status callbacks, and recovery-result callbacks. Telegram webhook/poll ingestion already uses the auto-assignment path.

Verification:

• pytest tests\openclaw_controller tests\openclaw_hostd -q -> 241 passed
• python -m code_index update --files code_index/openclaw_controller/app.py tests/openclaw_controller/test_api.py --json -> no errors

[mfethe1]

Progress update from the staged recovery slice:

• Automatic peer recovery now authorizes only the candidate's next staged command instead of running every allowlisted command in one tick.
• Recovery results are recorded by the controller-side recovery policy.
• A completed health-check result advances the automatic stage to service-restart through the recovery-result NATS trigger.
• A completed service-restart result does not immediately run index-update; the controller waits for the heartbeat recovery gate.

Verification:

• pytest tests\openclaw_controller\test_api.py tests\openclaw_controller\test_ssh_recovery.py -q -> 66 passed
• pytest tests\openclaw_controller tests\openclaw_hostd -q -> 243 passed
• python -m compileall -q code_index\openclaw_controller -> passed

[mfethe1]

Progress update from the heartbeat-gated index update recovery follow-up:

• Recovery candidates now continue through the final index-update stage only after service-restart has completed and the target sends a fresh healthy heartbeat.
• The final stage is gated by a repo-root heartbeat hint: index_update_needed: true.
• Heartbeat-triggered routing now runs the routing-agent path, so the heartbeat recovery gate can advance stages while still assigning pending Claimable Work.
• SSH recovery docs now describe the implemented stage order.

Verification:

• pytest tests\openclaw_controller\test_api.py tests\openclaw_controller\test_scheduler.py tests\openclaw_controller\test_ssh_recovery.py -q -> 89 passed
• pytest tests\openclaw_controller tests\openclaw_hostd -q -> 244 passed
• python -m compileall -q code_index\openclaw_controller -> passed

[mfethe1]

Progress update: the Telegram status-loop slice is now covered in the controller.

Landed behavior:

• Blocked run events create Telegram replies back to the original task thread.
• No-progress task-manager reassignments now notify Telegram when work is moved to another host.
• Completed recovery results for service-restart and index-update create fleet-room recovery summaries.
• Controller NATS callbacks for heartbeats, run events, task acknowledgements, run messages, and recovery results drain outbound Telegram work after recording and routing.

Verification:

• pytest tests\openclaw_controller tests\openclaw_hostd -q -> 247 passed.
• python -m compileall -q code_index\openclaw_controller passed.
• python -m code_index update --files ... --json completed with no failures.

Remaining before this goal should be closed: dashboard/readout work and empirical validation of the 5-second routing SLO.

[mfethe1]

Progress update: fleet readout and fast-routing SLO coverage landed.

Landed behavior:

• Added GET /fleet/readout for the operator/dashboard projection: host health, aliases, reported capacity, effective available Agent Run slots, controller-observed active Agent Runs, stale active runs, and active fleet leases.
• Documented the readout in the messaging service and UI command-center docs.
• Tightened the trusted Telegram webhook auto-assignment test with an empirical < 5s guard for the webhook-to-Task-Claim path when a healthy eligible host exists.

Verification:

• pytest tests\openclaw_controller\test_api.py -q -> 41 passed.
• pytest tests\openclaw_controller tests\openclaw_hostd -q -> 248 passed.
• pytest tests\test_agent_activity.py::test_run_orchestrator_classifies_and_moves_dead_claimless_run_to_review tests\test_agent_activity.py::test_active_runs_filters_stale_working_runs tests\test_run_lifecycle.py::test_run_orchestrator_apply_uses_process_liveness_for_dead_claimless_run -q -> 3 passed.
• python -m compileall -q code_index\openclaw_controller code_index\openclaw_hostd code_index\run_orchestrator.py passed.
• python -m code_index update --files ... --json completed with no failures.

Completion audit: all child tasks in the Two-Host Autonomous Fleet goal now have concrete implementation and test evidence in this branch.