fix(test): give each spawned worker its own metrics port

[stephane-segning]

Summary

main went red on the hooks two-worker gate. Both spawned workers died at startup:

Error: binding metrics listener 127.0.0.1:9091
Caused by: Address already in use (os error 98)

Nothing claimed the seeded attempts, and the wait timed out after 30s.

The red build is the less interesting half

sms-worker binds its metrics listener unconditionally, before any role runs, and main propagates the error — so a worker that cannot bind never claims anything, it exits. --metrics-listen defaults to a fixed 127.0.0.1:9091, all three worker-spawning suites took that default, and cargo test runs test binaries in parallel.

Every assertion in hooks_two_workers_live is satisfied by one worker draining everything: all rows succeeded, attempts == 1 on each, exactly one HTTP request per row. The two workers in that test have always contended for the same port, so one has always died — and the test passed anyway.

So the gate whose entire purpose is "two workers never double-deliver" has been running with one worker. Its green history is not evidence of the property. It only went red now because a concurrently running binary took 9091 first and killed both.

Two changes, because the port alone is not enough

Fixing only the port would leave the silent-degradation mode intact — the next thing to kill a worker would quietly halve the test again.

1. All three suites (hooks_two_workers_live, hooks_node_receiver_live, kill9_reclaim_live) pass --metrics-listen 127.0.0.1:0. The kernel assigns a free port; no probe-then-bind race, no shared global.
2. The two-worker gate asserts both children are still running before it asserts anything else. Deterministic, unlike inspecting leaseOwner — that is cleared the moment an attempt goes terminal, so observing both owners depends on catching them mid-flight.

This is the same class of defect sms-test-support already solves for the database (one per test binary). The metrics port was the global nobody had noticed.

Verification — against a real Postgres, both directions

	result
with the fix	both workers start, both bind, 1 passed in 3.92s
guard-failure proof	port pinned back to 9091: fails in 3.38s naming the dead worker — not a 30s timeout, and not a false pass

The guard-failure output, verbatim:

hooks-two-workers-test-1 exited before the run finished (exit status: 1). Every
assertion in this test is satisfied by one worker draining everything, so a dead
sibling makes it pass while proving nothing about two.

Worth stating plainly: the exactly-once property does hold with two real workers. That had never actually been exercised, so this could have surfaced a genuine bug. It did not.

cargo fmt --check and cargo clippy -p sms-worker-bin --all-targets -- -D warnings clean.

Note on how this was run

The harness's own container could not start on this machine — it binds a fixed host port 127.0.0.1:55432, which a local vaam-postgres-1 already held. I used VSMS_TEST_DATABASE_URL against a scratch Postgres on another port rather than stopping anyone's stack. That fixed harness port is a third instance of the same shared-global-port pattern; not fixed here.

🤖 Generated with Claude Code