Skip to content

fix(sdk-broker): stop per-poll heartbeat checkpoint + full index replay in readiness wait - #6278

Merged
Yeachan-Heo merged 3 commits into
devfrom
gjc-ready-poll-ckpt
Oct 3, 2026
Merged

Yeachan-Heo merged 3 commits into
devfrom
gjc-ready-poll-ckpt

Conversation

@probepark

@probepark probepark commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Why

gjc ACP spawn failures (hermes-ops G1: SPAWN_FAILED 3/56 = 5.4% over 2h, target <= 2%). All samples were on one host (studio), did not become ready ... child=alive (terminal_uncertain): the child host registered ~4.5s after spawn and stayed alive, but the broker gave up on readiness.

Root cause: waitForReady polls every 50ms and each poll calls currentReadyAuthority, which ran broker.heartbeatSessions() (machine-global session-index lock + live-heartbeat checkpoint) and an unconditional broker.index.refresh() (full replay). On a host with a large index (studio: 6779 sessions, checkpoint 150-168ms, index-lock occupancy 15% vs 1.2% on a smaller host) plus 3 concurrent launches, the per-poll lock/replay cost alone pushes host_registered -> ready past the readiness budget (studio p90 7.2s, failures 18.8-38.7s; other hosts p90 0.7s). 2-day readiness failures: studio 73 / work 7 / home2 1.

What

  • currentReadyAuthority now calls broker.index.refreshIfChanged() (change-stamp fast path, locked re-classification only when the index actually changed) instead of heartbeatSessions() + index.refresh() on every poll.
  • Admission authority is unchanged: owned readiness evidence + native-alive process observation + indexed record match (terminal / terminalUncertain / pid / incarnation / endpoint-file checks are all still enforced).
  • No timeout, POLL_MS, readiness budget, or retry changes.
  • Changelog fragment packages/coding-agent/changelog.d/broker-readiness-poll-index-cost.md.

Local tests (command + result)

  • bun test test/sdk-broker-lifecycle-e2e.test.ts -t "readiness polling avoids" -> 1 pass (on tank, linux)
  • bun run --workspaces --if-present check:types -> rc 0
  • bun scripts/telegram-daemon-generation-guard.ts $(git merge-base origin/dev HEAD) HEAD -> "v52 no protected changes"
  • Agent run also reported: focused lifecycle suites exit 0, bun --cwd=packages/coding-agent run check and run lint pass (2 pre-existing warnings in test/sdk-diagnostics-entry.test.ts).

Local CI gate (prepush)

  • bun run --workspaces --if-present check:types rc=0
  • bun scripts/telegram-daemon-generation-guard.ts ... rc=0

Needs e2e

  • Re-measure G1 after this lands on the mac fleet: DRY=1 python3 /opt/data/scripts/gjc-acp-guard.py (SPAWN_FAILED <= 2% over 2h, gjc provider).

Acceptance

AC Where Local test
AC-1 root cause narrowed to a layer SDK broker readiness poll (lifecycle.ts::currentReadyAuthority) measurements above
AC-2 change that fixes the cause (no timeout/retry bump) lifecycle.ts l.~5395 refreshIfChanged() check:types rc 0
AC-3 regression test that fails before the fix sdk-broker-lifecycle-e2e.test.ts "readiness polling avoids ..." (fixture fixed in 7486cb1: valid endpoint + owned lifecycle marker + injected incarnation, so the poll reaches the index lookup) target test 1 pass; RED proof against origin/dev lifecycle.ts exit 1; full file 151 pass / 1 skip / 0 fail (agent run on tank, not re-run by coordinator)
AC-4 G1 re-measured after fix gjc-acp-guard DRY=1 NOT DONE: needs the fix installed on the macs (via gjc-stale-binary-cron after merge)

Risk

  • regression-risk
  • low-risk
  • high-risk

Agent

  • gjc on tank: prompt_failed (-32603, "Agent run failed after execution started", phase post_start), no commits.
  • omo (cliproxy/gpt-6.1-sol) on tank: implemented the change; the runner died twice from pod restarts and was reattached; the agent left changes uncommitted, and the coordinator committed and pushed them.

Open questions

Merge approval is expected to stay red on this agent-authored PR until a maintainer approves the exact head (human-review path, #6037); the contract check itself should be green.

@snowykr snowykr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

APPROVED

Summary

The PR changes readiness polling to use the existing change-aware session-index refresh instead of checkpointing heartbeats and replaying the full index on each poll. Independent reviews found no verified actionable defects; the readiness identity, endpoint, terminal-state, and native-process checks remain intact.

Findings / Required Changes

No blocking or actionable findings.

Non-blocking Observations

The external G1 spawn-failure-rate measurement remains pending until deployment to the target fleet. The new operation-count regression test is Linux-only; exact-head CI ran it on Ubuntu, but that is not cross-platform runtime evidence.

CI / Verification

GitHub Actions Dev CI run 37130902395 is bound to reviewed head 6db191d5344d8e69cef6e9a05f56547463301fa6 and succeeded. The targeted sdk-broker-lifecycle-e2e.test.ts shard, affected-path evidence/aggregate validation, and virtual integration validation passed. The changed test verifies that readiness polling does not repeatedly checkpoint heartbeats or invoke full index refreshes. No local tests or project gates were run during this review; the post-deploy fleet metric is not established by CI.

Axis Coverage

Axis Verdict Coverage
A1 — Intent / Policy / Contract APPROVED The change matches the scoped polling-cost claim; readiness predicates and persisted/API contracts are unchanged. No .gjc intent projection was available.
A2 — Architecture / Correctness / Failure APPROVED Change-aware refresh retains locked reload on detected index changes; missing authority remains fail-closed and the wait retries within its existing deadline.
A3 — Security / Privacy / Trust APPROVED No trust-boundary or authorization checks were weakened; endpoint, process-incarnation, and readiness-event verification remain.
A4 — Verification / Tests / CI APPROVED The regression test is selected on the exact head and the required affected-test aggregate and virtual integration checks passed. The fleet metric remains a post-deploy measurement.
A5 — Context / Compatibility / Platform APPROVED The implementation reuses SessionIndex.refreshIfChanged(); persisted formats and public interfaces are unchanged. The added test's platform evidence is Linux-only.

Limitations

The review did not execute local tests or gates. CI provides exact-head Linux test evidence, but not Windows/macOS execution or the pending post-deploy fleet measurement.

@Yeachan-Heo
Yeachan-Heo merged commit 2d49709 into dev Oct 3, 2026
45 checks passed
@Yeachan-Heo

Copy link
Copy Markdown
Owner

Merged into dev as 2d49709c.

  • snowykr approved the exact head 6db191d.
  • CI on that head: 21 passed, 0 failed.
  • Base 100502f8 is 14 commits behind dev; none of them touch this PR's 3 files.

—
[repo owner's gaebal-gajae (clawdbot) 🦞]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants