Skip to content

feat(cluster): rolling "update all workers" orchestration (graceful, one at a time) #1876

Description

@jaylfc

@hognek - a clean, self-contained follow-on to your worker auto-update work (#1690 / #890), in your wheelhouse and orthogonal to everything your workers are currently on (desktop/CI). No rush.

Goal

One admin action to update the whole worker fleet without taking the cluster down: drain + update workers one at a time (never more than one draining concurrently), so there is always capacity serving. Umbrel-style "update all".

Context / what already exists on dev

  • Single-worker update already lands via update_worker in tinyagentos/routes/cluster.py (POST /api/cluster/workers/{name}/update) and drain_worker / cancel_drain in tinyagentos/cluster/manager.py.
  • I just fix-forwarded that single-worker path in fix(cluster): restore local-worker guard + correct worker-update restart handling (#1690) #1872: the restart-disconnect is now treated as success (keep draining, 200 status=updating), connect-refused cancels the drain (502), and the monitor loop skips the local worker. So the primitive your loop calls is correct now - please build on it rather than re-implementing.

Acceptance criteria

  1. New admin-gated endpoint POST /api/cluster/workers/update-all (use the same _require_admin(request) gate as the other mutating cluster routes).
  2. Updates every online worker sequentially - only one worker draining/updating at any moment, so the fleet never fully drains.
  3. Skips the local worker (the controller itself is not a remote worker).
  4. Reuses the existing per-worker update_worker semantics (restart-disconnect = accepted/keep-draining; connect-refused = cancel/fail). Do not duplicate that logic - factor a shared helper if needed.
  5. Fault-tolerant: if one worker's update fails, log it and continue the roll; do not abort the remaining workers. Return an aggregate {updated: [...], failed: [{name, error}], skipped: [...]}.
  6. Tests in tests/test_routes_cluster.py: happy path across 2+ workers, one-fails-others-continue, local-skipped. Mock httpx.AsyncClient per the pattern in the new test_update_worker_* tests from fix(cluster): restore local-worker guard + correct worker-update restart handling (#1690) #1872.

Out of scope (keep it tight)

  • No UI (the Cluster app button is a separate follow-up).
  • No parallel/batched updates - strictly sequential for v1.
  • No version pinning / rollback - just the rolling drain+update.

TDD, single focused branch/PR to dev, draft-then-ready per the taos-development-skill you just wrote. Shout if any of the scope is unclear.

Metadata

Metadata

Assignees

No one assigned

    Labels

    clusterDistributed cluster and workersenhancementNew feature or requestfeatureNew featurekilo-auto-fixAuto-generated label by Kilokilo-triagedAuto-generated label by Kilopython:uvPull requests that update python:uv codetestingTesting and QA

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions