Skip to content

m9-admin-surface.md

Latest commit

 

History

History
1203 lines (1092 loc) · 78.1 KB

m9-admin-surface.md

File metadata and controls

1203 lines (1092 loc) · 78.1 KB
Raw

M9 — Opt-in admin web surface

M9 adds an opt-in, admin-authenticated web surface to shunt so an operator can provision upstream Claude and Codex/ChatGPT accounts from a browser and observe account-pool health without shell access. It builds directly on M8 (m8-anthropic-multi-account.md): the store, the per-request account resolution, and the AccountPool quota/cooldown state all already exist — M9 only adds an HTTP surface over them.

The surface is deliberately co-designed to share its foundations (session auth, server-rendered page + CSRF convention, a single-use pending-login store, one [server.admin] opt-in) with the planned Claude-apps gateway-login milestone, which is inbound (client → shunt authorization server) where this feature is outbound (operator provisions shunt → Anthropic upstream accounts). M9 lands first and stands alone; gateway login builds on the same session/page layer rather than growing a second stack.

Motivation

After M8, adding an account to a deployed shunt required SSH (or docker exec) plus an interactive shunt login claude --name <n> flow. Depending on --mode, that flow creates a refreshable full-OAuth login, imports an existing Claude Code login, or creates a one-year setup token. The router exposed only /, /health, /protocol, /v1/models, /routes, and the two /v1/messages* endpoints. Pool health (per-account quota utilization, cooldowns) was observable only through the x-shunt-account response header and logs. M9 relocates directly provisionable OAuth/setup-token flows from a TTY to a browser form and surfaces the pool state that already lives in memory.

Scope

  • Claude and Codex provisioning. The Claude browser form offers a refreshable full-scope OAuth flow and the inference-only, one-year PKCE setup token. The Codex form always uses ChatGPT OAuth and stores a refreshable credential. Claude uses its fixed hosted manual redirect. ChatGPT redirects to http://localhost:1455/auth/callback; that localhost page is expected to fail in the operator's browser, which copies the full address-bar URL (or its <code>#<state> values) back to shunt. Importing an existing Claude Code or Codex credential file stays CLI-only because the source file lives on the host.
  • Usage observation and credential ownership are separate. The primary Accounts and usage view automatically discovers supported provider logins on the host: Claude Code, Codex CLI, Gemini CLI, Kimi Code, Grok CLI, and Cursor.app. It reads current access/session material only in memory and never refreshes, copies, or writes the source; Cursor.app's SQLite state is opened read-only. An expired source is reported as unavailable. Claude usage is cached for 60 seconds; Codex remains response-derived; the other integrations call their first-party read-only quota surfaces.
  • Refresh rotation ownership. A managed full-OAuth account is refreshed and written back by ClaudeAuthStore. Because the provider can rotate the refresh token, its store file must have one active process owner; copying or sharing it across running hosts can invalidate another process. Setup-token accounts remain static and avoid this hazard.
  • Full CRUD. Add (provision), list (metadata only), remove (delete the store file), and replace (re-run the flow with the same name, which the store already supports).
  • Read-only pool dashboard. A JSON endpoint plus a table, over the state AccountPool already tracks. No new state collection.

Configuration

A new [server.admin] block under [server]. Absent ⇒ no admin routes are registered at all — the default HTTP surface is unchanged. Present ⇒ the routes exist and authenticate every request.

[server.admin]
# header carrying the admin credential for API/curl calls; `x-api-key` is
# accepted alongside it on the admin and spend-limit routers only
header = "x-shunt-admin-token"
# env var holding admin credentials as name:token pairs (SEPARATE from
# [server.auth] client tokens). These are the WRITE tier
tokens_env = "SHUNT_ADMIN_TOKENS"
session_ttl_secs = 3600   # browser session lifetime after login
pending_ttl_secs = 600    # time to open the authorize URL and paste the code back
# hide_observed = false   # (default) read host CLI/app logins for the usage table

# Per-credential keys with an id the audit trail records. The key must come
# from ${VAR} / ${file:...} / a SHUNT_* override — a literal is rejected at load
[[server.admin.write_keys]]
id = "terraform"
key = "${SHUNT_ADMIN_KEY_TERRAFORM}"

[[server.admin.read_keys]]
id = "reporting"
key = "${file:/run/secrets/shunt-reporting-key}"
export SHUNT_ADMIN_TOKENS="ops:3f9c…"   # comma-separated name:token pairs

Admin credentials reuse the inbound-auth token format (m4-inbound-auth.md) and its constant-time compare, but are a separate credential from [server.auth]: client tokens are handed to devices; admin credentials add upstream accounts and administer spend limits. Configuration validation is fail-closed — a present [server.admin] whose three credential sources (tokens_env/tokens_file, write_keys, read_keys) are all unset, empty, or malformed is a startup error, never a silently-open admin surface (identical discipline to [server.auth]). An array-only deployment, with tokens_env unset, boots.

The token source can be a file instead of the environment: tokens_file (a path, ~ expanded) holds the same name:token pairs, one per line or comma-separated. When tokens_env is non-empty it wins; otherwise the file is read, and an unreadable configured file is a startup error. This exists so the token does not have to live in the launch environment. shunt dashboard setup automates the whole enablement: it writes a random admin:<token> to ~/.shunt/admin-token (owner-only, 0600, via the same atomic-write path as credential writeback), records it as [server.admin].tokens_file, adds [server.oauth_usage], and prints the dashboard URL. It is idempotent — re-running reuses the token and appends no duplicate block — and leaves any pre-existing [server.admin] untouched.

Two access tiers

read < write, and write implies read:

  • write — full access. The tokens_env/tokens_file name:token pairs are this tier (retained for compatibility; new deployments should prefer [[server.admin.write_keys]], which carries an id), as is every write_keys entry.

  • read — passes every GET on the admin surface and on the spend-limit API, and is refused with 403 permission_error on every mutation. It signs in to the dashboard as well: POST /admin/login mints a session that records the read tier, and require_write refuses that session's mutations exactly as it refuses the header credential's.

    The tier reaching the session is what makes that safe. While a browser session carried full access unconditionally, minting one from a read key would have silently escalated it, which is why the login form answered 401 until sessions recorded their minting privilege.

A credential's privilege is the maximum over every set it matches, not whichever set happened to be scanned last. Array ids must be non-blank and array keys at least 32 characters; ids and key values must each be unique across all three sets, and a collision names the colliding ids without logging a key value. A legacy tokens_env token shorter than 32 characters warns rather than failing, because those tokens predate the rule. Array keys are Secret values: a key written literally in the config file is rejected at load and must come from ${VAR}, ${file:/abs/path}, or a SHUNT_* override.

Optional OIDC browser login

[server.admin.oidc] adds an allowlisted OIDC/SSO button to the token login page. Admin tokens remain mandatory as the API/curl credential and fallback browser login; OIDC is an additional browser path, not a replacement security boundary.

[server.admin.oidc]
public_url = "https://admin.example.com"
issuer = "https://accounts.example.com"
client_id = "shunt-admin"
client_secret_env = "SHUNT_ADMIN_OIDC_SECRET"
allowed_domains = ["example.com"]
# allowed_emails = ["operator@example.net"]
# scopes = ["openid", "email", "profile"]

The registered redirect URI is {public_url}/admin/oidc/callback. public_url must be a bare HTTPS origin (loopback HTTP is allowed for local development). The issuer and optional endpoint overrides use HTTPS, or HTTP on localhost/127.0.0.1 only (the loopback hosts the login page's CSP form-action can name). Startup also fails closed for an empty issuer/client id, missing or empty client-secret environment variable, or an empty email/domain allowlist.

POST /admin/oidc/start is same-origin guarded and rate-limited, discovers or uses the configured authorization endpoint, creates PKCE plus a short-lived single-use state, and redirects to the provider. The callback consumes that state before any exchange, obtains a verified-email identity through the provider's token and UserInfo endpoints, and re-checks the current hot-reloaded allowlist before minting the ordinary admin session cookie. Provider input and exchange failures produce generic browser pages; secrets, codes, and tokens are never logged or echoed. Success always redirects to the fixed /admin target.

Runtime wiring

The split mirrors how M4/M8 already separate hot-reloadable config from process-lifetime state:

  • RuntimeState.admin_auth: Option<Arc<AdminAuth>> — re-resolved on every reload, so admin credential/header edits (including the key arrays) hot-apply just like [server.auth].
  • AppState.admin_stores: Arc<AdminStores> — the session, pending-login, and rate-limiter stores, created once in build_router (like Arc<AccountPool>) and threaded through the per-request snapshot so a reload never drops a live browser session.
  • Whether the /admin* route tree is registered is decided once at boot from the initial config (a reload cannot add or drop routes, like server.bind). A reload that toggles the block on or off logs a warn! that it needs a restart; disabling it on an already-registered surface makes every admin route reject requests (admin_auth becomes None).

Authentication and hardening

  • Two credentials, never mixed. Admin auth is the [server.admin] credential; it is never the [server.auth] client tokens.

  • Browser: sign in at /admin/login with an admin credential of either tier → an opaque session id in an in-memory SessionStore, set as cookie shunt_admin_session (HttpOnly, SameSite=Strict, Path=/admin). The cookie is marked Secure unless the request host is loopback, so local HTTP dev and tests work while any real deployment host gets a Secure cookie (reusing M8's host_is_loopback loopback carve-out). A session carries the tier of the credential that minted it — a read key's session is refused on every mutation, exactly as its header credential is.

  • API/curl: send the admin credential in the configured header (x-shunt-admin-token) or in x-api-key; both slots are accepted, and the resolved privilege is the maximum over whichever slots matched. When both slots carry a different valid credential of the same tier, the configured header supplies the audit actor, because it is checked first. This merged acceptance is scoped to the admin and spend-limit routers — on inference routes x-api-key is the caller's own Anthropic credential slot and an admin credential never authenticates there. Whatever these routers accept, auth::inbound::consumed_by strips from the same slot before any upstream request, alongside the gateway JWT and the [server.auth] static token, so an admin credential is never relayed to a provider. Header callers carry no ambient cookie and are therefore CSRF-exempt.

  • Tier check on mutations: every account-provisioning mutation calls require_write before the CSRF check; a read credential gets 403 permission_error ("read-only admin credential cannot perform this action"). POST /admin/logout needs no credential at all — it only clears a session — and is guarded by the same-origin check instead.

  • CSRF on every cookie-authenticated JSON mutation: a per-session synchronizer token, presented as x-csrf-token, plus a same-origin check (Sec-Fetch-Site, falling back to comparing Origin's authority to Host). No CORS. POST /admin/logout is a plain navigation form that cannot send the header, so it is guarded by the same-origin check plus the SameSite=Strict cookie instead of the synchronizer token.

  • Same-origin on POST /admin/login, for the same reason and checked before the rate limit, so a cross-site flood cannot spend the operator's login budget. SameSite=Strict decides whether the browser sends an existing session cookie; it does not stop the browser storing the Set-Cookie a cross-site form submission gets back. Without this guard, anyone holding a valid credential could submit that form from their own page and replace a visitor's session with one of their choosing. That became worth guarding when read_keys gained the ability to sign in: a read key is handed to someone deliberately given less privilege, and this was its one lever against a write operator — silently downgrading that dashboard to read-only until the operator signed in again. Scripted logins are unaffected: the check passes when neither Sec-Fetch-Site nor Origin is present.

  • The OIDC callback redeems state before it spends that budget. The callback is a GET, so another site can fire it with an <img> tag, and a same-origin check cannot apply to the identity provider's own redirect. A callback carrying no live state is therefore refused before the rate limit is consulted; only POST /admin/api/oidc/start, which is same-origin checked, mints one. Without [server.admin.oidc] the callback answers 404.

  • Pending-login store is in-memory only, single-use, and TTL-bound; each completion attempt is counted and the entry is discarded after a small cap. The 256-bit OAuth state already makes guessing infeasible.

  • Rate-limit on the completion and login endpoints (a coarse global fixed window each) against code- and admin-token-guessing storms.

  • Secrets never leak: the verifier, authorization code, access token, and refresh token are never logged and never returned to the browser. The OAuth state is intentionally carried in the authorize URL and the opaque session id only in the HttpOnly session cookie — both are protocol values the browser must receive, not bearer secrets. Account add/remove is audit-logged by name and provisioning mode only.

  • Docs recommend binding the admin surface behind HTTPS / a tunnel, same as the shared-gateway guide.

  • Emergency token rotation: browser sessions are validated only against the in-memory session store, and the running process's environment is fixed — a config reload re-reads SHUNT_ADMIN_TOKENS from the same startup environment, so it neither rotates the token nor drops issued sessions (those persist until session_ttl_secs, default 1h). If an admin token is compromised, replace it in the environment source (systemd unit, .env, …) and restart the process: the restart both loads the new token set and drops every session the old token minted. To disable the last admin credential, remove the [server.admin] block before restarting (with no key arrays configured, an empty SHUNT_ADMIN_TOKENS fails closed at startup). Rejecting stale sessions on reload is tracked in #100. A ${file:...}-backed [[server.admin.write_keys]]/read_keys entry is the exception to the first half of this: its value is re-read on every config load, so overwriting the referenced file and triggering a reload does rotate that key without a restart. Sessions already minted still survive until session_ttl_secs.

    That last sentence reaches read keys too, which it could not while a read key was refused a session. A read_keys login now mints a read-tier session, so rotating a compromised read key stops its header credential at the next reload while its cookie goes on reading the admin surface until session_ttl_secs elapses — and that key carries no upper bound (AdminConfig::session_ttl_secs is a bare u64 with a default, and Config::validate does not range-check it), so a deployment that raised it for convenience widens the window by exactly as much. What survives is read-only — require_write refuses that session's mutations, and it is strictly less than the full access a write-tier session already carried across the same window — but a deployment that hands read keys out widely because revocation looked immediate no longer has that property, and should restart rather than reload. #100 covers both tiers; neither is fixed by the session tier alone.

Endpoints (registered only when [server.admin] is set)

Pre-split record. The paths below are the ones M9 shipped. Every JSON and mutation route among them has since moved to /admin/api/*; only /admin, /admin/login, and /admin/oidc/callback still answer on the paths written here. /admin kept its path but not its answer: the server-rendered dashboard described below is deleted, and that path now serves the embedded SPA shell — unauthenticated, so the sign-in redirect happens client-side after GET /admin/api/session answers 401, and in a build without --features ui it answers 404 instead. This table is deliberately left as the pre-split record rather than rewritten — see admin-ui-delivery.md (Decision 3, Decision 4, and Resolution 6) for why the namespace split and the SPA move happened, and docs/reference/endpoints.md for the current paths and the before/after migration table.

Method Path Purpose
GET /admin Dashboard (HTML); redirects to /admin/login when not signed in
GET,POST /admin/login Token or OIDC login form → session cookie
POST /admin/oidc/start Start the optional same-origin OIDC/PKCE browser login
GET /admin/oidc/callback Complete OIDC login, enforce the current allowlist, and mint a session
POST /admin/logout Clear the session
GET /admin/accounts JSON: Claude store metadata (name, kind, expiry, UUID — never the token)
GET /admin/accounts/codex JSON: Codex store metadata (name, expiry, account ID — never the token)
GET /admin/observed JSON: read-only observed Claude, Codex, Gemini, Kimi, Grok, and Cursor identity, state, and provider-native usage — never token material
GET /admin/pool JSON: per-claude_oauth/chatgpt_oauth/antigravity_oauth managed-pool state; account objects may include an optional plan string (antigravity_oauth accounts additionally carry a quota_buckets array containing the Gemini and Claude+GPT shared 5h/weekly windows from the retrieveUserQuotaSummary poll)
POST /admin/accounts/claude {name, mode} → start Claude provisioning (oauth or setup_token); omitted mode defaults to setup_token; returns {authorize_url}
POST /admin/accounts/claude/{name}/complete {code} → finish; stores the Claude account
POST /admin/accounts/claude/{name}/refresh Exercise an imported account's refresh grant now and report whether the login is still alive; returns the new expires_at and never token material. 400 for a setup_token account (no refresh grant exists) or a terminal verdict (invalid_grant, no stored refresh token, or a rotated pair that could not be persisted); 502 for a non-terminal failure
DELETE /admin/accounts/claude/{name} Remove the Claude account's store file
POST /admin/accounts/codex {name} → start ChatGPT OAuth; returns {authorize_url}
POST /admin/accounts/codex/{name}/complete {code} with a full callback URL or <code>#<state> → finish and store the Codex account
DELETE /admin/accounts/codex/{name} Remove the Codex account's store file

Gateway-owned errors keep the Anthropic error shape (ShuntError); page routes render minimal server-side HTML with inline CSS/JS and no external requests.

Every GET above is reachable with a read credential, as is POST /admin/login — it mints a session at the tier of whichever credential signed in. The seven account-provisioning routes (POST/DELETE under /admin/accounts/...) require write. POST /admin/logout and the two OIDC routes are login-flow plumbing and are guarded by the same-origin/state checks rather than by tier.

Phase 1 — provisioning flow

The browser flow reuses the CLI OAuth/setup-token internals in auth/claude/login.rs (generate_pkce, build_authorize_url, exchange_code) and stores through claude_store. The upstream redirect URI is fixed to platform.claude.com/oauth/code/callback for this remote/manual flow — the CLI's full-OAuth mode can use a localhost callback, but that loopback would return to the operator's browser host rather than a remote shunt server. The operator therefore pastes <code>#<state> into the form for both web modes.

  1. POST /admin/accounts/claude {name, mode} validates the name and mode, generates a PKCE verifier/challenge + state, stores a single-use pending login with its authoritative flow kind (TTL pending_ttl_secs), and returns the authorize URL (https://claude.com/cai/oauth/authorize). mode = "oauth" requests the full refreshable Claude scope; mode = "setup_token" requests user:inference. Omitting mode defaults to setup_token for API backward compatibility, while the dashboard explicitly sends oauth by default.
  2. The operator opens the URL, signs in to the target Claude account, approves, and pastes the resulting <code>#<state>.
  3. POST /admin/accounts/claude/{name}/complete {code} verifies state (constant-time), exchanges the code at the token endpoint (honoring SHUNT_CLAUDE_TOKEN_URL for tests), then dispatches by the server-stored pending kind. Setup-token mode requests the one-year expiry, requires an account UUID, and calls store_setup_token. Full OAuth omits that expiry override, requires a non-empty refresh token, accepts an optional account UUID, computes expiresAt, and calls store_oauth_tokens. Both writes are atomic at 0600; the pending entry is consumed. The completion request cannot switch modes.
  4. The completion response reports whether the account is live immediately (a claude_oauth provider with an empty accounts list scans the store each request) or needs a name-only [[providers.<name>.accounts]] entry + reload.

Removal deletes the store file directly, path-guarded so a caller-supplied name can never escape the accounts directory. This is new writeback behavior over an operator-owned store file (issue-sanctioned) and touches no upstream state.

Codex/ChatGPT OAuth

The Codex form reuses the shared PKCE generator but follows the Codex CLI OAuth contract from m2-chatgpt-oauth.md: authorize at https://auth.openai.com/oauth/authorize with the fixed http://localhost:1455/auth/callback redirect, then exchange the code using an application/x-www-form-urlencoded POST. The operator may paste either the full redirect URL from the browser address bar or <code>#<state>. Completion checks the pending state in constant time, requires a refresh token, derives the account ID from the access-token JWT, and writes the verbatim auth.json shape at 0600. SHUNT_CODEX_TOKEN_URL overrides the exchange endpoint for local tests; an invalid override, or one that is neither loopback nor HTTPS to the built-in endpoint's host, is ignored with a warning (mirroring the Claude completion flow) instead of silently, and the exchange POST uses the redirect-hardened client, which follows a 3xx only within the host it started on, so a permitted endpoint cannot hand the single-use code to another host. No access, refresh, or ID token is returned or logged.

Like Claude, an empty-account chatgpt_oauth provider scans the Codex store and makes the new account live on its next request. Explicit-account providers need a name-only account entry and reload.

Phase 2 — observed usage and managed-pool health

The dashboard is usage-first. GET /admin/observed discovers supported local credentials on each request but keeps every token/session in a private, non-serializable model. Set [server.admin].hide_observed = true to skip that discovery: GET /admin/api/observed still authenticates and returns an empty accounts list without reading any of those sources, and GET /admin/api/session reports hide_observed so the SPA skips the read. The Accounts and usage table stays, listing managed pool accounts alone — it is also where their usage is shown, so removing it would hide the pool's numbers along with the host's logins. Claude Code checks its configured/default credential file, then macOS Keychain service Claude Code-credentials; Codex, Gemini, Kimi, and Grok read their CLI credential stores; Cursor opens Cursor.app's state.vscdb with SQLITE_OPEN_READ_ONLY. The endpoint masks account identity, labels ownership as observed, and never invokes a refresh/writeback store. Provider requests have a 15-second timeout. Claude reads /api/oauth/usage, with the token-free snapshot cached process-wide for 60 seconds. The Claude and Codex rows are the exception to the identity masking above: each carries its account uuid — the Claude account uuid, or the ChatGPT account id — so the table can tell an observation and a managed pool account holding the same subscription apart from two genuinely different accounts. Both values are already returned unmasked by GET /admin/api/accounts and GET /admin/api/accounts/codex to the same authenticated caller, so this adds no disclosure the admin surface did not already make. Gemini returns every Code Assist model bucket, Kimi returns weekly and 5-hour windows, Grok returns credit/product usage, and Cursor returns billing-cycle, Auto + Composer, and named-model usage. Codex remains response-derived: both translated Messages traffic and raw inbound Responses attach the default CLI account id to x-codex-* quota capture without importing the credential into the managed store. Before such traffic its row explicitly says it is waiting for traffic.

The Accounts and usage heading carries a refresh bar: when the table last finished loading, the countdown to the next usage-poller pass (from the pool response's usage_poll; hidden while the poller is off), and a Reload button. While the tab is visible the page re-reads shunt's own state every 30 seconds: GET /admin/api/pool and the account stores, with the usage table rebuilt from the observations it already has. It never calls GET /admin/api/observed on that timer, because that endpoint asks each provider's usage API about every login on the host, uncached. On those timed reloads, a row that is both a pool account and a host login keeps the numbers the last full load drew, except where the pool's own window has changed since that load: the pool has then learned something the observation predates. A cached waiting-for-traffic is dropped the same way once the pool has seen the account's traffic. One timed reload runs at a time, so a backend slower than the interval still lands its answers. A hidden tab skips the timer and catches up once when it is shown again. When the last observation failed the timer leaves the error showing instead of repainting the table without the logins. Reload repeats the full page load, observations included. It is not the per-account Refresh, which exercises that account's refresh grant, and neither starts a usage-poller pass early.

The usage table groups by provider: the provider is named once and its accounts nest beneath it, folding managed pool accounts in alongside the read-only observations rather than stranding them in the advanced section. An observation and a managed account are coalesced into one row when their account uuid matches — one subscription is one row, labelled with the managed account name, with the observation's windows preferred because the pool only learns a window from a response header it has actually received. Each identity table is matched only against accounts of its own store's auth kind: Claude store uuids against claude_oauth accounts, Codex store account ids against chatgpt_oauth accounts. The Codex case is the common one on a developer machine, where ~/.codex/auth.json and a shunt login codex copy are the same ChatGPT account and previously rendered as two rows with an identical weekly bar.

Coalescing is deliberately conservative: identity resolves to None whenever CLAUDE_CONFIG_DIR or CLAUDE_CREDENTIALS is set, and an unidentified observation never matches. Both Claude credential sources are profile-agnostic (the fixed Keychain service name above; a hardcoded $HOME/.claude/.credentials.json, unless CLAUDE_CREDENTIALS relocates it) while oauthAccount.accountUuid is read from a config directory that CLAUDE_CONFIG_DIR relocates, so with either variable set the recorded identity can name a different account than the credential that was actually read — observed against a live two-account pool, where it labelled an exhausted account's usage with the other account's uuid. Declining to merge costs a combined row; guessing costs a silently mis-attributed quota bar. Closing that gap needs a token-scoped identity source, which does not exist today: /api/oauth/usage is the only Claude endpoint read here, and shuntAccountUuid is captured only at login/import time.

The known uuid is still attached even when the local token has expired: the account id is read independently of token validity, so an expired observation keeps matching its managed row instead of rendering as an unrelated duplicate.

Which state the coalesced row then displays follows a precedence, not a blanket override: a managed operational state (disabled/cooling/ near-quota) is an actionable gateway-side fact and wins over a stale local observation error, so a cooling account still shows its cooldown remediation even though the client's last local check happened to see an expired token. An observed error (expired/unavailable) still surfaces over a merely idle managed state (available/unseen), which is the "Needs login" case this matching was built for.

Within the managed states both tables run one ladder — disabled, needs_relogin, !has_state, hard cap, used-up window, account-wide cooldown, near_quota, Fable cap, Fable-only cooldown — so the same account cannot read near quota in the managed-pool table while the Accounts table calls it Cooling (issue #512). A used-up window gets its own exhausted state ("Exhausted", noted Quota used up · clears when the window resets), read from the pool snapshot's exhausted flag: the 5-hour or weekly window was rejected or reached 100% utilization. The rejection test is selection's own, so a stale aggregate status counts only when no per-window status has been recorded, and a Fable-only (7d_oi) rejection does not count. near_quota cannot carry that on its own, since the pool also sets it at the switch threshold (98% by default) while the account can still serve. Like a hard cap, it lasts until its window resets, so it sits above the short cooldown. The cooldown is tested before near_quota because it is the fact that all of the account's traffic is gated right now, while near_quota warns about what is coming. When both cooldowns are running the row's state is the account-wide one, and its note names both deadlines (retries in 10m · Fable retries in 30m), the way the managed-pool table's Cooldown column already lists both (issue #511).

Managed provisioning and store metadata remain available under a collapsed Manage pool accounts (advanced) section. AccountPool::snapshot(provider, &[AccountConfig], model) returns a token-free, serializable view per account: 5h/7d/7d_oi utilization + reset, unified status, account-wide cooldown-seconds-remaining, Fable-only cooldown-seconds-remaining, near_quota, a needs_relogin flag, and a derived available flag. The Fable-only cooldown counts toward available only when model is a Fable model, so an account cooling on its 7d_oi bucket still reports available to every other family. Because the admin snapshot is taken with model = None, the dashboard carries the Fable-only cooldown as its own cooling-fable row state ("Cooling (Fable)", with a Fable retries in … note) rather than folding it into available — the account is genuinely live for every non-Fable family, so neither "Live" nor a plain "Cooling" describes it. It reads the same entries map select_order reads, clears only already-past quota buckets (as the next selection would), never mutates the round-robin cursor, and never inserts entries for accounts the pool has not yet seen (reported as has_state: false). Such a row still carries a needs_relogin the admin refresh probe recorded by store name — see the side table below — because that is the one thing that can be known about an account the pool holds no entry for.

Each row also carries the account's request counters since process start: requests_attempted, always the sum of requests_succeeded (a 2xx status), requests_failed (any other status, or no response headers at all) and requests_cancelled (the send was dropped before it resolved: the client went away or a deadline cut it), and mean_latency_ms, the mean time from dispatch to response headers over the attempts that received headers (null until one has). One attempt is one proxied HTTP dispatch, a same-account retry included; control-plane calls on the account's credential (usage polls, model discovery, catalog lookups, token refreshes) are not attempts. The adapters record it where the account is already bound (CountAttempt in src/accounts/request_stats.rs, whose drop guard records a cancelled send), since the failover chain sees one result per provider, after any account rotation inside it. The totals live on the account's health entry under the existing entries lock and are never emitted to the metric sinks, which stay free of account identity. Like the cooldowns they are memory-only: a reload or store rescan keeps them, deleting the account through the admin API drops them, and a restart resets them. Rows that resolve to the same account identity share one set. They are reported on a has_state: false row too, because an attempt that timed out before any headers leaves no other trace. Attempts over the Codex WebSocket transport, served or failed, are not counted, since a WebSocket turn has no per-turn response headers to classify; an HTTP fallback after a failed WebSocket attempt is counted.

Each row also carries bound_sessions, the number of that provider's live session bindings (ADR-0008) on the row's physical identity. A binding idle for more than 24 hours is expired and is not counted. Bindings are keyed by provider and session hash, with the physical account identity as the value, so aliases that resolve to one identity share an account_ref, and each of their rows reports that identity's one count. The count is never split per alias and never summed. A consumer that totals bound sessions dedupes by account_ref, as it would for the request counters. The count is scoped to the provider table: a session bound under another provider on the same physical account counts only in that provider's table, not here. The handler reads the binding table once per request, under the binding table's own lock, never while it holds entries. Only counts are returned, never a session hash or id. A runtime PATCH /admin/api/pool of sort_by_reset leaves every binding in place, because that override only reorders failover and sessionless traffic.

needs_relogin — a dead credential, not a pause

needs_relogin marks an account whose credential no operator-free retry can revive, and both dashboard tables report it as needs re-login — the primary Accounts table ("Needs re-login", with a "Re-add this account to sign in again" note) and the collapsed managed-pool table's State column — ahead of every cooldown state. It exists because the two are otherwise indistinguishable on the dashboard: a cooldown expires after five minutes and the account is selected again, so a permanently dead account shows the same cooling a quota pause does, forever — one 401 (and, for an imported account, one already-rejected refresh POST) every five minutes with nothing durable to see.

The mark also records why it was set (ReloginCause, internal — the JSON field stays a boolean). Only one decision reads it: whether a successful refresh grant is enough to clear the mark. RefreshGrant means the grant itself failed, so a later grant that succeeds disproves it and the admin probe may clear it. ServedRequest means the provider rejected a bearer the account actually presented — the grant was never the broken part, so a working grant proves nothing and only a served response clears it.

It is set from exactly three places, all terminal by construction:

  • a 401 on a credential that carries no refresh grant at all — a token_env token, or a setup token still inside its local expiresAt that the provider has revoked early (adapters/anthropic/mod.rs, the RefreshRetry branch). Resolution succeeds because the stored token still looks valid, so the verdict can only come from the upstream 401,

  • a 401 on the retry after a refresh that itself succeeded. A live grant yielding a bearer the API still rejects means the account is de-authorized upstream, not momentarily unlucky; the adapter already cools it for five minutes and rotates, so without the mark it cycles there forever reported as plain cooling, and

  • a refresh that cannot be retried into success, classified by auth::claude::auth::is_terminal_refresh_failure. Three reasons carry the typed TerminalRefresh marker, mirroring the one the gateway store already uses (auth/gateway/auth.rs) and the code-based classification in the Kimi store (auth/kimi/auth.rs):

    • InvalidGrant — the provider returned the OAuth invalid_grant code.
    • NoRefreshToken — the credential file carries no refresh token, so there is no grant left to send. A setup token past its local expiresAt reaches its dead state through this path rather than the 401 above: resolution fails before any upstream request is sent, so the RefreshRetry branch is never entered. The two setup-token bullets are the same credential at different times — revoked-before-expiry surfaces via the 401, expired surfaces at resolution — and an expired token takes this path whether or not it was also revoked, because nothing reaches the provider.
    • WritebackFailed — the provider rotated the token but the new pair could not be persisted. The grant already consumed the refresh token on disk, so every later attempt replays a spent one. Attached only when the response actually carried a different refresh token: a provider that omits the field leaves the stored one live, and that writeback failure stays non-terminal.

    A transient failure (5xx, network, timeout, an unparseable body) is deliberately not terminal: marking one would report a healthy account as dead after a momentary provider blip. Both the 401 → force-refresh path and credential resolution classify this way: once a dead account's access token expires — its steady state within hours — the refresh is rejected on read, before any upstream POST, and that path marks the account too (auth::resolve_claude_account_classified).

The Codex pool (chatgpt_oauth) marks from the same three places, with its own typed marker (auth::codex::auth::TerminalRefresh, carried on ChatGptAuthError next to the logged detail), since #616: a token_env 401 and a still-401 retry set ServedRequest, and a terminal refresh — the invalid_grant code from auth.openai.com, no refresh token stored, or a rotated pair lost before writeback — sets RefreshGrant from both the resolution path and the 401 → force-refresh path. Before that fix every Codex refresh failure was one undifferentiated five-minute cooldown, so a dead Codex account read as live on the dashboard between attempts, indefinitely.

It is cleared by proof the credential works again — but the proof has to match the cause. A served response (mark_healthy_scoped) clears any mark, and so does a re-login through POST /admin/accounts/claude/{name}/complete, which replaces the credential outright. A successful POST /admin/accounts/claude/{name}/refresh probe clears only a RefreshGrant-caused mark.

A quota 429 clears the mark too, on both the initial and the post-refresh attempt. The provider read the credential and then refused on quota, so the bearer authenticated and any mark from an earlier 401 is stale; without this an exhausted account would tell the operator to re-login until some later non-429 response. Narrowed to the quota-rejected 429 specifically — the one carrying an anthropic-ratelimit-unified-*-status: rejected header, which is what accounts::classify routes to Rotate. Two other responses reach the same code and prove nothing: a 5xx, which can come from an edge before the credential is ever read, and a headerless 429, a generic throttle with no account-scoped quota verdict behind it. The initial arm can test the status alone because it is entered only for Rotate; the post-refresh arm groups Rotate | PauseSame | RefreshRetry, so it checks the classification too. Only the mark is cleared — the cooldown still runs.

A successful refresh on the proxy path clears nothing on its own. The adapter deliberately waits for the retried request: a live grant proves the refresh token works, not that the account can serve inference, and the retry may still come back 401 — in which case the mark is set rather than cleared.

The mark follows the credential, not the provider row that tripped over it. One store account activated by name in two [[providers.*]] tables gets two health entries — resolve_pool_accounts leaves a name-only entry UUID-less, so account_key keys it as UpstreamInline, which carries the upstream name — yet both are backed by one credential file. Marking (and clearing) therefore fans out across every entry for the same store account, exactly as the admin re-login and probe paths already do. The fan-out is narrow on purpose: an account carrying its own credentials path or token_env is a different credential that merely shares a name, so a failure on it condemns nothing else. An entry keyed by uuid (Verified) is reached only from an account that carries that same uuid, because AccountKey keeps no name to match on there.

The fan-out reaches three things it would otherwise miss. It stamps observed on the sibling, because snapshot reports an unobserved entry as unseen and drops every field on it — a row select_order has picked but no response has answered yet would carry the mark and still render clean. It runs on mark_healthy_scoped as well, so a response served through any row clears the mark on all of them rather than leaving a live credential condemned until each row happens to serve traffic of its own. And on the success path it clears the mark alone: cooldowns and storm-control ramps stay per-row, because this is a signal and not a routing policy.

That success-path clear is gated on an any_needs_relogin flag, because it sits on the path every served response takes and scanning the map there cost ~2x on the account_pool_healthy_updates benchmark for a signal that is absent in the steady state. The flag is conservative in one direction only — false means no entry carries a mark, true may outlive the last one and cost a single extra scan — and every mutation of it happens under the entries lock, so the recompute can never store false over a mark that landed beside it.

The flag is deliberately independent of the cooldown and changes nothing about routing, selection, or the cooldown clock — it adds a signal, it does not add a policy. It is memory-only: the opt-in [server.pool] state_path persister carries quota alone, so a restart clears the mark and the account's next terminal failure re-establishes it.

POST /admin/accounts/claude/{name}/refresh is the operator's on-demand probe for the same question. It is write-tier, CSRF-checked, and rate-limited through the same complete_rate limiter as the completion route (it POSTs to the provider's token endpoint). It refuses a setup_token account up front rather than attempting a grant that cannot exist, and it always goes through ClaudeAuthStore::force_refresh, never a hand-rolled grant: that store holds the process-global REFRESH_LOCK across read → POST → atomic writeback, which is what keeps the probe from racing the proxy's own refresh and stranding a rotated refresh token. It does not additionally take the per-account AccountPool::refresh_lock the proxy takes — that lock is keyed by an AccountConfig from a provider table, and a store account may be reachable through zero, one, or several provider entries, so there is no single right one to pick; it is also unnecessary, because both of the proxy's critical sections sit inside the same global lock. Success returns the new expires_at and nothing else — no access token, no refresh token, and no provider body that carried them. It clears a RefreshGrant-caused mark, but deliberately not a ServedRequest-caused one: the probe never sends a request the account has to serve, so its success is no evidence against a bearer the provider already rejected. The response's needs_relogin is read back from the pool after that clear, so a probe that succeeded against a still-dead account says so rather than reporting a recovery /admin/pool would contradict. The dashboard reports only what the pool actually stores. A session's account lives in the binding table (ADR-0008), which the pool view reports as a per-identity bound_sessions count. The pool keeps no last-selected timestamp, so none is shown. GET /admin/pool enumerates each claude_oauth and chatgpt_oauth provider's accounts (its configured list, or the corresponding Claude/Codex store scan for an empty list — the same resolution the adapters use). Codex successful responses now populate the 5h/7d fields from x-codex-* rate-limit headers; unsupported windows are ignored and 7d_oi remains None because Codex has no analog. Since issue #195 this recorded state also feeds Codex account selection (see m10-codex-multi-account.md), in addition to the dashboard display.

The mark above lives on health entries, keyed by AccountKey. The admin routes do not have one: POST /admin/accounts/claude/{name}/refresh knows a store name and, when the credential file carries a shuntAccountUuid, a uuid. For an account some provider table has selected that is enough — the setter matches those against every entry's identity. An account the pool has never selected has no entry at all, so the setter used to update nothing and /admin/pool kept reporting the row unseen even though the operator had just been told the credential is dead (issue #439). Inventing an entry is not an option: it would mean synthesizing an AccountKey the selection path never produced.

So the verdict is also recorded in a second, key-free place: store_relogin, a set of StoreAccountRef — (store_family, name, uuid) — on the pool. It is a set rather than a map of causes because only the admin probe writes there and its verdict is always RefreshGrant — membership is the cause. snapshot consults it in both branches: the unseen row reports it alone, and an observed row reports its entry's mark or the set's. The entry is not authoritative, because the setter cannot always reach it — it matches entries with StoreAccountRef::matches, which is uuid-only on a Verified key, so a verdict recorded with no uuid never stamps a row whose config carries a uuid (AccountConfig::uuid is a config key, and the probe's own uuid read returns None both when the credential file has no shuntAccountUuid and when the read failed). Reporting only the entry there made /admin/pool answer needs_relogin: false while store_account_needs_relogin called the same account dead. ORing is safe because every clear purges the set too, so one served response on any row backed by that store account lifts the display. has_state stays false on the unseen row, which is still true and which both dashboard tables read after needs_relogin, so the row renders needs re-login rather than unseen.

Every path that clears the mark purges the set as well — a re-login (set_needs_relogin_for_store_account(.., false)), a successful probe (clear_grant_relogin_for_store_account), the narrow clear (clear_needs_relogin), a served response (mark_healthy_scoped), and account deletion (forget_identity, reached from DELETE /admin/accounts/claude/{name} through forget_pool_health_if_absent). forget_identity takes the store name alongside the resolved identity for the set purge alone (the entry, refresh-lock and membership matcher stays keyed by identity): the admin call sites resolve identity = account_uuid(name).unwrap_or(name), so a delete whose uuid read succeeds arrives with a uuid that equals neither field of a verdict recorded when that read had failed, and the ref would survive the delete to condemn a later re-add of the same name.

That purge is also gated differently from the rest of the deletion cleanup. forget_pool_health_if_absent reaches forget_identity only once nothing in the store family still references the identity and the store scan actually answered, because the identity-keyed state is legitimately shared: two store names can resolve to one Anthropic account, and the surviving alias's health must outlive the delete. The deleted name's verdict must not, whatever the scan said — the caller has already removed or re-provisioned that name. So the name-keyed half runs first and unconditionally, through forget_store_relogin_name(family, name), which takes the set alone and strips on (family, name): wider than accept, and safe for the same reasons the other strips are, since the name is unique within its family, a sibling alias's verdict is a separate ref keyed by that alias's own name, and the next terminal probe re-marks. forget_identity's own store_name arm stays in place — redundant on that path now, but it is public API that has to remain correct on its own.

The served-response purge reproduces the fan-out in both directions. Its narrowing: an account carrying its own credentials path or token_env is a different credential that merely shares a name, so serving on it clears nothing — and for the same reason neither snapshot branch condemns such a row either, since nothing it could do would ever clear the mark. Its width: a Verified-keyed account reaches its name-keyed siblings through the name, so the purge strips on (family, name-or-uuid) rather than on the key's own identity. Matching only the key would strand a verdict recorded with no uuid — the credential file carried no shuntAccountUuid when the probe ran, or the admin path's uuid read failed into None — once the account is only ever reached through a uuid-keyed row, and every name-keyed row would keep rendering "needs re-login" for an account that is demonstrably serving.

Every read and clear of the set speaks (family, name-or-uuid) rather than ref equality — the two clears that carry the admin routes' own (family, name, uuid), and the probe's needs_relogin read-back — so neither is ever narrower than the set that recorded the verdict — the set matches an entry on the uuid or the name, and an equality-removal would strip less than it accepts. The reachable gap is a re-login under the same store name that signs a different subscription in: the credential file now reports a new uuid, the recorded ref still carries the old one, and the fresh account would stay condemned on every name-keyed row. A store name is unique within its family — it is the credential file's name — so widening to it cannot reach another account's verdict. The uuid arm is guarded on the ref actually having one, or a clear carrying no uuid would match every uuid-less ref in the family whatever its name.

The accept side also matches through the account rather than through the AccountKey — snapshot holds an AccountConfig, and a Verified key carries a uuid and no name, so keying off it would be uuid-only. That matters because the probe records whatever claude_store::account_uuid returned, which is None both when the credential file carries no shuntAccountUuid and when that read failed (unwrap_or(None)), while AccountConfig::uuid is a config key an operator can set on the provider entry directly — so the two need not agree even without a race, and matching through the key alone would silently never show a verdict the probe did record. matches stays the predicate for the three paths that match a pool entry, where the key really is all there is.

But accept is narrower than the strip, not identical to it, because the two fail in opposite directions. Over-stripping only drops a verdict the account's next terminal failure re-establishes; over-accepting renders "needs re-login" against a credential the verdict is not about, and nothing on that row can lift it. So store_relogin_ref_condemns takes the uuid as the stronger identity whenever both sides carry one — same uuid, same credential; different uuids, different credentials that merely share a store name — and falls back to the name only when at most one side has a uuid. The is_some guard on that uuid arm is what stops None == None from reading as agreement between two unrelated uuid-less accounts. Every pair accept admits, the strip also strips, so the clear stays at least as wide as the display.

Two ordering rules make this safe. The entries lock is always acquired first and store_relogin second, never the reverse — most sites that touch the set already hold the entries lock, so nesting is the usual pattern; forget_identity takes the set alone, after releasing entries, and forget_store_relogin_name takes nothing else at all. And any_needs_relogin is raised unconditionally by the setter rather than from inside the entry loop (a never-selected account matches no entry), while mark_healthy_scoped folds !store_relogin.is_empty() into its recomputed value after purging. Without that fold the flag would drop to false while a never-selected account's verdict is still live, and the gated scan — the only thing that ever clears it — would never run again.

The Claude store table in that section reports a derived status rather than the raw claudeAiOauth.expiresAt it used to render. That timestamp is the ~8-hour access-token deadline and means opposite things per credential kind, so showing it verbatim made healthy accounts read as broken. An imported account carries a refresh token and shunt renews it in-band (single-flight writeback in auth/claude/auth.rs, triggered five minutes before expiry), so a past timestamp there is routine and needs no operator action — the row reads "Auto-refreshes", with the raw timestamp preserved on hover. A setup_token account has no refresh token at all (one-year lifetime), so the same timestamp is genuinely actionable: with more than the refresh buffer left, the row reads "Valid until" followed by that date; inside that buffer — or past the date, or with no date at all — it reads "Expired" under the expired danger style with a Setup token cannot refresh · re-login required note. Only the setup-token kind can reach that state.

The buffer is the same five minutes routing itself applies, and the dashboard takes it from that constant rather than copying the number. Tokens::is_valid_at accepts a credential only while expiresAt > now + EXPIRY_BUFFER, where EXPIRY_BUFFER is auth/claude/auth.rs's own five minutes — auth/shared.rs declares a same-named constant that drives the generic JWT helper, not this path. A setup token has no refresh token to recover with, so inside that window a routed request already fails on the no-refresh-token path; comparing against the bare deadline here would have the dashboard call a credential usable for the last five minutes of its life while routing rejects it, the window in which the operator most needs the warning. GET /admin/api/session serves the constant's millisecond value as expiry_buffer_ms, so the two cannot drift apart. (The server-rendered dashboard this replaced substituted it into its own script instead; a served value and a substituted one are equally drift-proof, which is why the move changed nothing here.)

The Codex store table alongside it carries the same status column, but unconditionally: that store has no non-refreshable kind at all. Both writers into it reject a missing or empty refresh token (import_auth and store_chatgpt_tokens in auth/codex/store.rs) and there is no setup-token analog, so shunt owns renewal for every row (single-flight refresh and atomic writeback in auth/codex/auth.rs, five minutes before the access token's JWT exp, via the shared EXPIRY_BUFFER). The expiry this column used to print was therefore never actionable — it simply made every healthy Codex account read as broken within the hour, directly beneath a Claude row saying "Auto-refreshes" for the identical situation. Every Codex row now reads "Auto-refreshes" too, with the raw timestamp on hover.

Each store row in both tables also carries a Re-login action beside Remove. It adds no endpoint: re-provisioning is already the ordinary flow run under an existing name (POST /admin/accounts/{claude,codex} → paste → .../complete). Neither start route carries a duplicate-name guard, and both completions capture the pre-store identity, overwrite the account in place, and hand the old and new identities to cleanup_reprovisioned_pool_health, so a reprovision that changes the upstream identity does not strand the replaced one's health entry. The button therefore only primes the existing add form — it fills in the account name, clears any half-finished flow (including the currentName / currentCodexName handle the completion POST interpolates into its URL, so a code cannot be completed against the wrong account), and scrolls the form into view. The Claude button additionally preselects the login method matching that row's current kind, since re-provisioning under the other mode would silently convert the account between refreshable and inference-only; the Codex form has no such choice, ChatGPT OAuth being the only way into that store.

Clearing the form is not sufficient on its own. A start or completion request already in flight writes its result back when it lands, restoring the flow that was just cleared. The late start is the damaging one: it reopens the previous account's authorize step and restores that account's handle while the name field already reads the newly picked one, so following the reopened link stores the freshly authorized credential under the old account's name — silently overwriting a different pool account. A late completion is milder, blanking the just-primed name and reporting success for the wrong account. Each request therefore captures a per-form flow epoch (claudeFlowEpoch / codexFlowEpoch, bumped by a re-login and by every new start or completion) and discards its own response once superseded. Claude and Codex count separately, so re-priming one form never discards the other's live flow.

Re-priming is not the only way a stale authorization step survives. A second Start on a form whose step is already open used to leave the first link clickable, and its Complete button posts to the name captured for that flow — and because a completion bumps the epoch itself, that click also strands the start now in flight, whose response arrives superseded and is dropped. start therefore clears the authorization state before it bumps the epoch, so the link on screen always belongs to the login the operator last asked for (issue #513). The login-method radios are disabled for the same span: the mode is fixed in the server's pending entry when the request is issued, so a live radio would let the form read one method while the pending login is the other. The lock is keyed on starting || authorizeUrl, not on authorizeUrl alone — start nulls authorizeUrl before it sends, so the narrower key would reopen the radios for exactly the length of the request that had already captured the mode. A superseded start does not release the flag; prime and each newer start set it as they take over, so it always belongs to whichever call owns the current epoch. A disabled attribute states no reason, so the group carries a role="status" note naming the lock — the form's own live region is empty at exactly that moment, because start clears it and a successful start never sets it. Pressing Start again does not release the lock (the replacement start re-arms it before any render), so the note names the two things that do: completing the flow, or reloading the page.

Clearing at issue time costs the operator something when the replacement start then fails: the page has thrown away the only handle to a pending login the server may well still hold. Restoring the flow is not the repair — reinstating authorizeUrl and the name handle puts the page back into exactly the state #513 removed, and making that coherent means restoring the name field too, stomping the edit the operator is about to correct. Each failure message names what was closed instead, so the operator restarts rather than hunting for a link that is gone (issue #531). Both of start's failure paths carry it, because the clear runs before the request: a refusal appends the notice to the server's own reason, and an unanswered start says it in place of the "no authorization step opened" line, which speaks only for the step that failed to open.

What the server still holds differs by path, which is why the notice speaks only for the page. A refused start never reaches PendingStore::start — add_account calls it only after every failure return — so the earlier pending login is untouched and lives out its pending_ttl_secs. An unanswered one may have reached the server all the same, and PendingStore::start is a keyed insert that replaces the entry, so a retry under the same account name has already invalidated the login the closed step pointed at. Neither is reachable from the page, which is what the notice says and all it says.

The notice is said only when a step was in fact closed — a failed start that closed nothing has nothing to report, and saying it regardless would teach an operator to read past the sentence on the one occasion it is true. Reading authorizeUrl alone does not decide that, because the closure and the message that reports it can be separated: a start closes the step synchronously, and its own response can then be dropped by the epoch guard when a newer start supersedes it — while that newer start reads an authorizeUrl the older one already nulled. The Start button is not disabled while a start is in flight, unlike Complete — which is disabled for its own request, and carries the 120-second AbortController bound described below precisely so that being disabled cannot close it for the life of the page. start has no such bound. So a second Start during the first is an ordinary double click, not a race. A closedStepUnreported ref carries the fact across the gap and is released by whichever message reports it. The predicate defers on a step whose code is already submitted (completingNow): a completion leaves authorizeUrl non-null until it succeeds and clears it only after the epoch guard, so a Start clicked mid-completion supersedes that completion — suppressing its confirmation while onStored has already stored the account. Claiming a lost step there would send the operator to re-provision an account already in the table. Deferring is not declining, though: when that superseded completion comes back a definite failure, nothing was stored, the step it was spending is closed, and its own error is suppressed by the epoch the newer start took — so complete reports the closure itself on that branch. Which way it reports depends on what the superseding start has managed to do by then, and both orders are ordinary: the refusal is a local validation while the completion is an upstream round trip, so it usually lands first. If that verdict is already on screen, the completion amends it in place; if the start is still in flight, the ref carries the fact to the message it will write. Only the start's own verdict is amended, tracked by shownStartFailure — #addmsg is shared with the row actions, which report through report, and a closed-step note appended to a failed refresh would name a start the operator did not make. Nothing is reported at all when that start has opened a step of its own (stepOpen, mirroring authorizeUrl for the same reason): the operator is not stranded, and arming the carry there would strand the fact instead, since completing the new step consumes it silently and the notice would surface later on a failure that closed nothing. The two unknown completion outcomes — an unreadable answer, an abandoned request — record nothing, because neither can say the account was not stored and onStored has already re-read the table, which is where that question is answered. Nor does a completion superseded by prime: re-login discards the half-finished flow at the operator's own request, so the next start closes nothing and must not say it did. The wider predicate the radio lock uses (starting || authorizeUrl !== null) would reach the same case, but it fires just as readily on two chained starts that never opened a step at all, and would then name a step the operator never saw.

The epoch orders starts, where the later click is the live one, and must not be extended to order two completions of the same flow: a completion consumes the pending login, so there the first click is the one that stores the credential and a second finds the entry already consumed and fails. Superseding by epoch would silence the successful response and surface the failed one, reporting an error over an account that was in fact stored. Each completion handler therefore refuses re-entry while its own request is in flight (claudeCompleting / codexCompleting), disabling the button for the duration.

That marker is deliberately not released by a newer start, tempting as that is for a page whose Complete button is closed. PendingStore::attempt does not consume the entry and complete_account removes it only after the store, so a start issued during an in-flight exchange replaces the entry and lets a second completion pass its own state check — both exchanges then reach the store in an order nothing constrains, and the older one landing last leaves the account holding the superseded credential.

The page's marker cannot be what keeps that sequence out of reach, because it only binds one page: a second tab, a second operator, or a direct API call is subject to none of it, and the abort bound below releases the marker while the server may still be exchanging. So the ordering is server-side. Each completion holds PendingStore::lock_completion for its whole attempt → exchange → store → remove sequence, keyed by the pending key, so a second completion waits and then finds the entry already consumed and fails closed with "start again" (issue #440). The marker stays as what it always was locally — a refusal that costs no round-trip — rather than the thing that orders the mutations.

The lock is bounded by the exchange it holds. COMPLETION_EXCHANGE_TIMEOUT caps that upstream exchange at 30 seconds, so a hung provider releases the lock with a 502 instead of parking every later completion for that key behind it for as long as the connection stays open. The pending entry survives the timeout — the attempt still counts against MAX_PENDING_ATTEMPTS — but the authorization code may already be spent upstream, so the recovery is a fresh start rather than re-posting the same code.

start deliberately does not take that lock: blocking a start behind an in-flight exchange would stall the operator for up to the completion's own timeout, and two racing starts already fail closed on the state check, which is why ordering them is tracked separately rather than here.

Nothing else may release the marker, so the completion request carries its own 120-second AbortController bound, cleared in a finally: a connection that never settles must not close the button for the life of the page. Whether the account was stored is unknown once a completion is abandoned, so that path refreshes the tables and says so, rather than reporting a failure the server may not agree with.

The marker is a per-page-load convenience, not an enforceable lock: it is a let in the inline script, so reloading the dashboard clears it and permits the same retry the bound does. Being page-local it also cannot see a second tab or a direct API call. Its job is only to keep one page's own two clicks from racing; ordering concurrent completions is the server's, through the per-pending-key lock described above.

Both are deliberately confined to the managed store tables: the observed rows in the top-level Accounts and usage table are unchanged, since those credentials are owned and refreshed by the provider client itself and shunt never invokes a refresh/writeback store for them.

The optional plan field is derived from credential data already held by shunt: Claude reads claudeAiOauth.subscriptionType, and Codex reads the chatgpt_plan_type claim from its stored JWT. Whenever an imported Claude credential's on-disk access token is still valid, the request also makes a bounded GET /api/oauth/profile backfill and caches the result — this runs even for an account whose file already carries a subscription type, since that value alone carries no multiplier detail and the profile lookup can refine it toward a more precise one, while never discarding it if the lookup is coarser or fails; this backfill only ever reads a token already on disk, it never refreshes and never writes back. An account whose on-disk token has already expired keeps whatever plan (or lack of one) it already had until a later view, once normal traffic elsewhere refreshes it; an idle account with no traffic and no usage polling can stay at that same value indefinitely while its on-disk token stays expired. Setup-token and token_env accounts are not backfilled. A missing, failed, or unrecognized lookup leaves the existing plan (or its absence) unaffected.

Budget exhaustion never erases a file-derived plan: the file-read phase is guaranteed its own min_slice floor above the shared deadline, so an earlier provider's stalled backfill can never starve this cheap local read. Even in the pathological case where the file phase does not finish within that floor, any plan already cached from an earlier resolution still appears in the response.

Plans, tokens, and credential identities are tracked positionally against the resolved account list, never keyed by display name. resolve_pool_accounts appends the configured accounts after any scoped store entries without deduplicating names, so one provider can legitimately resolve two distinct accounts that share a label; a name-keyed map collapses them, which would both show one account's subscription on the other and — because the same map supplies the bearer token — probe one account with the other's credential.

The profile cache is keyed by the credential's own identity where one exists. An account whose credential file carries the shuntAccountUuid shunt's import stamps in is keyed by that value: it survives a token refresh and changes when the account is re-provisioned, so a resolved plan is held for 24 hours. The file's uuid outranks a uuid carried in config, because the cached value is the plan of whoever that file's token authenticates as, and the config field never selects the file — the credential path is derived from credentials or the account name. A config uuid, or the process-lifetime inline-identity memo that fills it in when config omits it, can therefore still name the previous occupant of a re-provisioned path; keying off it would file the new account's plan under the old account's identity and serve the old plan back for the full day. A config uuid is used only when the file yielded none. Only when no uuid exists anywhere does the key fall back to the account's name, which is stable but not unique over time — the same name may later belong to a different account. Nothing in production clears this cache, so that fallback caps its entries at 10 minutes instead, which is also the retry interval for a failed lookup.

Two further rules follow from that fallback being a name rather than an identity. A name key shared by two accounts within one resolved list is ambiguous, not merely imprecise: the shared cache cannot separate them, so neither account reads or writes it and both are resolved fresh. And because a uuid-less account's identity lives only in its credential file, the uuid last seen at each credential path is memoized, so a resolution whose file phase timed out can still reconstruct the key and serve an already-cached plan. That memo is keyed by path rather than name for the same reason as everything else on this path: two accounts a name cannot separate still have distinct paths, and when they share a path they are reading one file and so share one identity.

That memo is consulted only when the file phase produced no result at all. A completed read is authoritative even when it finds no uuid — the account holding that path today has no identity, whatever an earlier one had — and such a read also clears the remembered value, so a later timeout cannot resurrect it. Every way a read can fail to produce a uuid clears it, not just the parses-but-carries-none case: a missing, unreadable, or unparsable file drops the entry too. The bias is deliberate. Clearing costs at most a fallback to the name key and its 10-minute ceiling, while keeping a no-longer-evidenced uuid risks serving the path's previous occupant's plan for the full exact-identity day.

More than the path memo goes stale on a failed read, so such an account does not use the shared cache at all for that pass. Every identity still available to it names a previous holder: account.uuid is filled by resolve_pool_accounts from the very file that just failed to read and is memoized for the process lifetime, so it outlives that credential; and the name fallback is stable but not unique over time. There is nothing to write either, since a failed read yields no token. So the plan is omitted for that pass and reappears once the credential reads again — an honest gap rather than another account's subscription.

A file that parses and merely carries no shuntAccountUuid is not a failed read and none of this applies to it: that is an ordinary hand-placed credential, keyed as it always was.

The batched read walks its candidates in order, so it can record one account's failure and then be abandoned mid-list when a later credential stalls past the deadline. The per-account failures are therefore tracked outside the result the timeout discards — a failure the read already observed is knowledge the request keeps, and dropping it would hand that account back the cache access this rule exists to withhold.

A failure also has to outlive the request that saw it, for the same reason one step further out. Per-request failure state says nothing about the next request, so a later one that times out would carry none and fall back to the still-memoized uuid — restoring precisely the identity the failed read refuted. The path memo therefore records a read failure rather than merely dropping what it knew, and an account whose path is remembered that way stays out of the cache on a timed-out pass too. Only a pass that reads the file again can clear it: a timeout is not evidence that the path became readable.

The batched credential read is single-flight, process-wide. A read holds its permit until it genuinely finishes, not until the request waiting on it gives up, because a spawn_blocking task cannot be cancelled once started. A credential file on a hung network or FUSE mount therefore leaks one blocking worker rather than one per /admin/pool request: later requests fail to acquire the permit, skip the file phase, and fall back to their cached plans.

Shared foundations with gateway login

The gateway-login milestone (Claude Code /login against shunt) is inbound and separate, but should reuse rather than duplicate:

  • the browser/admin session-auth layer — the /device approval page needs an authenticated human, the same session mechanism as /admin;
  • the server-rendered page + CSRF convention;
  • the [server.admin] opt-in surface — the gateway-login block can nest beside it;
  • the single-use, TTL-bound pending store — the device-flow "pending authorization" is the same shape (session::PendingStore is written generically for this reuse).

Testing

  • Unit: session/pending TTL + single-use + attempt cap, rate limiter, CSRF accept/reject, constant-time admin auth, cookie Secure loopback carve-out, AccountPool::snapshot, claude_store::list_account_meta/remove_account.
  • Integration (tests/admin_surface.rs): the routes are absent without the block (404); API requires auth (401); setup-token mode keeps the legacy omitted-mode behavior and one-year exchange; full Claude OAuth requests the full scope, omits the expiry override, and persists a refreshable account; ChatGPT OAuth carries the Codex CLI authorize parameters, accepts both callback paste forms, uses a form-encoded exchange, persists verbatim auth.json, and appears in the pool; malformed or unknown modes and invalid account names fail without storing a file; missing refresh tokens fail closed; list/pool/response payloads never expose token material; cookie mutations without a CSRF token are rejected (403); fail-closed startup without the tokens env.