Repository navigation
Epic: fleet consumption observability — live per-lane table, provider dashboards, cron-to-cost mapping #41
Description
Activity
Phase 1 spec correction:
cc_workloadis not a header, and it is not on every requestMeasured on 300 recent
req-*.jsoncaptures (ai-01):It is not in a header.
request-logger.ts:84persists exactly{ ts, src, machine, model, pid, body }— no headers reach a capture file.user-agent,x-forwarded-for,x-real-ipare read and go only to the stdout line. The flag lives inbody.system(the harness block), 300/300.It is not on every request. Present on 98/300 (32.7%), and its only observed value is
cron. It is a presence flag, not a multi-valued field — so "absent" does not mean "interactive".The usable discriminator is
cc_entrypoint(100% coverage), and it yields three bucketscc_entrypointcc_workloadn claude-vscodecron98 claude-vscode(absent) 104 sdk-cli(absent) 25 sdk-ts(absent) 73 SDK traffic (98 req, 33%) never carries
cc_workload: neither declared-cron nor genuinely interactive. A binary cron-vs-interactive split would file all of it as interactive.Revised acceptance for Phase 1: three buckets —
cron(flag present),interactive(claude-vscode, no flag),agent(sdk-*) — read frombody.system, not from headers.⚠ Sample bias, do not propagate the proportions. These captures are exclusively AUTONOMOUS-mode requests (a NOMINAL sidecar writes nothing), so the ratios describe ai-01 during a hub outage, not the fleet. What generalizes is the structure — where the field lives and its flag semantics — because that is a property of the client. Re-derive proportions from hub captures.
Trap that will produce a wrong-but-plausible table
CLAUDE.mdalready records it: local model APIs reportprompt_tokensas the FULL conversation context on every request, not a delta — which is whywriteTokenFileassigns (=) rather than accumulating (+=). Naively summinginput_tokensover local lanes overcounts massively. Encode the guard in the script, not in a comment.Measurement hygiene for Phase 3
A hub event-loop freeze skews any throughput figure computed over the window it lands in. Rate measured from the sidecar relay log: 1.22 freezes ≥30s per hour before the #30 rebuild, 0.15/h after (p=0.003 over 6h38) — an 8x drop, but not zero. Windows containing a freeze should be flagged, not silently averaged.
Correction to my previous comment: those 30s events are not hub freezes
I labelled the
forward failed (connect: AbortError)events "hub event-loop freezes ≥30s" and gave a before/after rate. The freeze reading is wrong, and the control that disproves it was available the whole time.The sidecar's relay heartbeats the hub every 10s with a 4s timeout, and 2 consecutive failures flip it to AUTONOMOUS (
HEARTBEAT_INTERVAL_MS,HEARTBEAT_TIMEOUT_MS,relay.ts:126). A 30-second freeze would fail roughly three heartbeats in a row — comfortably past the threshold.Observed over 30h: 33 aborts, 2 AUTONOMOUS windows, and both windows are separately accounted for (a po-2023 machine reboot at 00:04Z; the pre-#30 capture stall at 10:26Z). Every abort line reads
[1/2 before AUTONOMOUS]— the counter never reached 2.So for 31 of 33 events the hub was answering
/healthin ~4.5ms while a realPOSTtook >30s to produce headers. That is not a frozen hub. It is either upstream-provider time-to-first-token, or hub-side queueing/backpressure — and distinguishing the two needs hub-side data I do not have.Corrected consequences for #41:
- The measurement-hygiene note stands, with a different cause. Windows containing a >30s stalled request still skew throughput figures; flag them. But do not attribute them to hub freezes.
- The "8x reduction after the fix(fork): make request capture non-blocking (fire-and-forget async write) #30 rebuild" is much weaker than I stated. It is a drop in slow-header events (1.22/h → 0.27/h over 7.3h), and provider-side load variation across the day explains it as well as fix(fork): make request capture non-blocking (fire-and-forget async write) #30 does. Not a clean before/after.
- What survives unchanged: the 10:26Z event was a genuine hub stall (AUTONOMOUS and machine up and pre-fix(fork): make request capture non-blocking (fire-and-forget async write) #30 image), corroborated independently by the hub-side burst at 10:20Z.
- New, and relevant to Phase 1: hub load does not drive event-loop latency. Across 2072
/healthsamples the median is flat at ~4.5ms from 1 to 9 concurrent streams (Pearson +0.007), with exactly one sample above 500ms. So per-request latency in the consumption table should not be modelled as a function of fleet concurrency without evidence.
The Phase 1 spec correction in my previous comment (
cc_workloadlocation and semantics, the three-bucket taxonomy, theprompt_tokensfull-context trap) is unaffected — it was measured directly from capture contents.Phase 1: do not add a "freeze marker" — the phenomenon it would mark does not exist
Following up on the correction above, because the retracted claim reached a DM before the retraction did, and it was about to become a rule in the instrument.
Do not implement "mark any window containing a hub freeze". There are no hub freezes. A second, independent instrument now confirms what the heartbeat already said:
/healthprobe, 4268 samples over 179 min (day + night)median 4.5 ms p99 66 ms max, absolute 1.03 s The abort threshold is
FORWARD_HEADERS_TIMEOUT_MS = 30_000— 30x above the worst point ever observed, with no samples anywhere in between. A rule keyed on freezes would mark on nothing, and would inject a silent bias into throughput figures instead of removing one. Remove the rule rather than replace it; if a real signal ever appears, it can be added then.If a window-quality marker is still wanted, the real, already-logged event is the relay abort — 33 in 24h, sidecar-side, one
forward failedline each, timestamped. It is measurable, it is real, and it genuinely shifts attribution for a window: an aborted forward means that request was served locally instead of by the hub, so the hub's own capture set does not contain it.Cause of the aborts remains open — upstream provider TTFT, or per-lane queueing on the hub. That needs hub-side data.
Unrelated but adopted for the Phase 1 spec: the fifth bucket
(none, none)(~9% of hub captures — probes, ingress translations, non-CC traffic) is absent from my earlier three-bucket table because I sampled sidecar-side, where it does not occur. Four buckets read frombody.system(cron / interactive / agent-sdk / other) is the right shape.Gap analysis 2026-09-11 (po-203, hub operator) — what is already built vs. what the new mandate adds
Verified in code today, not from memory.
Already delivered (Phase 1 is essentially done)
scripts/traffic-consumption.py— Phase 1 as specified: rollup per provider-lane × bucket (cron/interactive/agent-sdk/other), tokens in/out, cache read/creation, hit-rate, débit/h. Five traps already encoded:(pid,reqN)is not unique across restarts; GLM emits a zero usage block first (take MAX); wire-openai lanes report no cache (n/a, not 0%); bucket read from thex-anthropic-billing-headercc_workloadflag; giant captures need head+tail.scripts/traffic-mxws.py— machine × workspace × model (workspace parsed from the system prompt'sPrimary working directory:; machine fromx-claudish-machinein the capture metadata).scripts/harness-injection-measure.py— harness block cost, chars injected per token.- MCP
claudish_traffic([HARNESS][TRACES] Fiabiliser et mutualiser la consultation des traces proxy — outil MCP claudish_traffic + corpus captures vers le pipeline commun #72) — reliable temporal reading, gap-declarative.
What the 2026-09-11 mandate adds
User scope, verbatim: qui = Machine:Workspace:Harnais en condition · quoi = tokens lus/écrits, fréquence de réveil, fréquence de condensation, idéalement part des messages / du harnais fixe / des mémoires.
- Harness as a third attribution key — MISSING. Today two lanes sharing machine+workspace are indistinguishable. Proposal: harness fingerprint =
cc version(already inua=) + hash of the position-0 harness block (CLAUDE.md +.claude/rules/+ MEMORY.md) + the model actually resolved. Both are already inreq-*.json— this is aggregation, not new plumbing. - Wake-up frequency — PARTIAL. The cron/interactive split exists, but
cc_workloadis a flag (~33% coverage; absent = "not cron") anduacarries a second, independent marker. Needs (a) consolidation of the two signals, (b) a per-lane fires/h count — which is what "adjust the crons" actually requires, not a share. - Condensation frequency — NOT MEASURED AT ALL. No detector for
/compactanywhere inpackages/cli/src(only thestream:falsebuffering plumbing). The harness block is re-paid in full at every compaction, so this is a cost multiplier, not a curiosity. → opened as a dedicated issue. - Coverage hole on the Sol lane.
createResponseCaptureis wired innative-handler.ts:254,anthropic-sse.ts:63,openai-sse.ts:151— not inopenai-responses-sse.ts. Sol/Codex traffic writes noresp-*.sse, so its tokens are invisible to every capture-based tool. → opened as a dedicated issue. - Capture topology changed with the 2026-09-05 migration. The hub is on po-2025; a NOMINAL relay writes no local capture by construction. Any fleet-wide rollup must therefore run hub-side, and an empty per-machine reading on a relay is expected — worth writing into the Phase 1 acceptance so nobody chases a phantom (this machine's
D:\claudish-captures\upstream-errors.loghas been frozen since 00:31 for exactly that reason). - Phase 2/3 unchanged but re-prioritised: the Anthropic weekly credit is the binding constraint this week, and native Anthropic is precisely the lane where our own tooling is weakest.
Acceptance (in the #3174 sense — "answers Y without a second call")
One command answers, for a window: which Machine:Workspace:Harness consumed how much (in/out/cache), at what fire rate, with how many compactions — and reconciles with a manual capture count.
Delivery proposal (3 days, hub-side first)
- D1 — harness key + wake-up rate in
traffic-consumption.py(both derivable from existing captures). - D2 — compaction detector + the
compactions/dayandtokens re-paidcolumns. - D3 — the live operator table (top consumers by Machine:Workspace:Harness) + the markdown report slice for Epic: Fleet traffic & consumption analytics — recurring markdown report #60.
Prerequisite already open: #1 (machine header on 5 machines) — without it, "qui" is partial fleet-wide.
- added a commit that references this issue
on Sep 12, 2026 Rattachée comme sub-issue de #116 (EPIC parapluie datastudy consommation)
Cette issue est absorbée comme couche P1 (instrumentation) de #116 — elle n'est ni fermée ni remplacée : le parapluie a besoin de son contenu.
Ce qui est repris tel quel, sans réécriture :
- Phase 1 est déjà livrée (
scripts/traffic-consumption.py) — le parapluie la traite comme acquise, pas comme à faire. - La correction de spec du 23/08 (quatre buckets lus dans
body.system—cron/interactive/agent-sdk/other— et non dans un header) reste la spécification de référence. - La décision du 23/08 de ne pas ajouter de marqueur de gel (le phénomène n'existe pas : max
/health1,03 s contre un seuil d'abort à 30 s) reste en vigueur — on ne rajoute pas ce garde-fou. - Le trou de couverture Sol signalé ici le 11/09 a été confirmé et corrigé : fix(capture): wire openai-responses-sse to createResponseCapture - Sol/Codex lane writes no resp capture #90 (fermée 14/09 21:42Z) câble
createResponseCapturedansopenai-responses-sse.ts. Il reste Responses wire drops usage.input_tokens_details.cached_tokens — la lane Sol/Codex est aveugle au cache #115 (le wire Responses perdcached_tokens) — la lane reste aveugle au cache même après fix(capture): wire openai-responses-sse to createResponseCapture - Sol/Codex lane writes no resp capture #90.
Ce que le parapluie ajoute et qui n'était pas dans le périmètre de #41 : la table de prix (blocage n°1 —
pricing-cache.jsonabsent,all-models.jsonsans champpricing), les métriques projet (NanoClaw, Hermes, notebooks), la synthèse dettes/leviers, et un dashboard avec graphiques et interprétations plutôt qu'un tableau.- Phase 1 est déjà livrée (
Live decomposition delivered (2026-10-02, po-2025 hub captures) — Phase 1 granularity extended to machine:workspace:session.
Method:
Get-CaptureRequests(CaptureUtils attribution) joined withresp-*MiniMax*.sseusage blocks (MAX per field), paired on (pid, counter, nearest-ts). Today 00:00→14:30Z: 10,130 paired MiniMax responses — IN 63.9M · cache_read 1.81B · OUT 3.2M (cache hit 96-97%, cross-checked against platform.minimax.io hourly rows: ±5-10%).Top consumers = the 8 interactive-cron worker sessions (CoursIA/CoursIA-2/CoursIA-3 on po-2023/2024/2026/2027, ai-01), e.g. po-2026:CoursIA-2 session
e7dab0a1= 1,269 req / 241.8M cache_read alone. Entrypoint split: 7,059/7,060 haiku requestsclaude-vscode— zero headless footprint on this lane; sub-agent share ≈0.Console cross-check conclusions: hub≈provider (no unattributed volume), no image/video API rows in 30 days (the “10 images” card is not API-key usage — not the H3-rumor shape), no key rotation justified.
Outgrowth of this session: fleet rule from the user — authorized = exactly 1 interactive-session cron per workspace×machine + roo-extensions schtasks; split-brain/headless = leak (Vibe-Worker precedent: $21.86 burned 01/10 before disable). Self-check asked cluster-wide on the global dashboard. 6-day ramp attribution (per-machine, from local archives) lands as
multiday-minimax.json— will follow here.Mandat user 02/10 (outillage consommation, suite du commentaire précédent) : « petit à petit vous affineriez votre outillage pour pouvoir me présenter un tableau de bord précis et en temps réel. Ça ne viendra pas du jour au lendemain, mais à plusieurs avec des relances vous pouvez y arriver. »
Briques livrées aujourd'hui (scratchpad session, à consolider) :
- Attribution 26 h req↔resp par machine:workspace:session (CaptureUtils + decompose_minimax.py) — 10 130 paires, MAX-per-field usage.
- Scan live sessions (fenêtre glissante N min) : table machine×session×workspace avec req/min, profondeur msgs, workload — c'est lui qui a identifié le drop 880→456 req/h après les restarts user.
- Détecteur split-brain (3 signatures hub) : régression de profondeur non-compaction, ping-pong workload cron↔None <12 min, salves parallèles <3 s même session.
- Décomposition multijours depuis archives 7z (rampe par machine depuis 26/09).
Feuille de route proposée (incrémentale) :
- v0 (prochaine session) : consolider 1-3 en un script unique
scripts/live-consumption+ tâche planifiée hub produisant un snapshot horaire (JSON+MD) — le « tableau de bord précis » à défaut du temps réel. - v1 : expose via
claudish_trafficMCP (le contrat existe déjà — lane-series.sqlite) : dimension session au-dessus de la dimension lane existante. - v2 : quasi-temps-réel (scan incrémental sur fichiers récents, pas re-parse complet) + alertes (seuil req/h par session, split-brain signature, >1 session concurrente par workspace×machine).
Verdict du jour sur la question d'origine « d'où vient chaque token » : answered end-to-end pour la lane haiku/MiniMax — burn = workers interactifs-cron, 0 % headless, coût = scie de compaction (rebuild 0→500 msgs toutes les 1,5-2 h re-lu à chaque requête).
Rampe multijours MiniMax (cacheR/jour, pairing req↔resp des archives 7z) — réponse mesurée à « qui consomme » :
Jour cacheR Dominant 26/09 1,66 B 4 workers équilibrés (0,33–0,46 B chacun) 27/09 4,20 B ai-01 seul : 3,30 B — 43 479 req (~30/min sur 24 h), 13,3 M OUT 28/09 1,91 B po-2026 0,85 B, ai-01 0,42 B 29/09 1,16 B po-2026, po-2024 30/09 — archive locale absente (à documenter) 01/10 2,64 B 5 workers 0,31–0,65 B 02/10 ~3,2 B trajectoire, post-restart user → ~490 req/h Décomposition du pic 27/09 : quasi intégralement des subagents ai-01 (buckets 28 975 + 5 742 + 3 697 + 1 069 req, dont un workspace
argumentum.assetconverte…) ≈ 3,0 B des 3,3 B. Forme légitime (whitelist user : subagents de sessions interactives), volume runaway — ~20 req/min soutenues 24 h, sans plafond (caps MaxBudgetUsd non branchés). ASK posée aux lanes ai-01 (dashboard global 02/10 ~20:00Z) : quel workflow a tourné le 27/09.Régime depuis 28/09 : structurel — les 5 workers CoursIA en scie de compaction (rebuild 0→~500 msgs / 1,5–2 h, chaque requête re-lit le contexte accumulé). Le levier de volume n'est pas le nombre de workers mais (taux de requêtes) × (profondeur moyenne).
Instrument :
multiday_7z.py(prefiltre par index resp MiniMax avant parse — 139 k fichiers/jour sinon tuent le run ; sauvegarde incrémentale par jour ; reprise automatique). Données :multiday-minimax.json.- added 5 commits that reference this issue
on Oct 4, 2026 1 remaining item
- added 15 commits that reference this issue
on Oct 5, 2026
Why
User mandate 2026-08-23: "un tableau nominal des consommations en temps réel, sur toutes les lignes" + regular verification against provider dashboards + mapping spend and cache misses to workspace crons and their frequencies, eventually peak/off-peak (heures creuses/pleines). Goal: consume every subscription fully with only slight PAYG overflow (DeepSeek), and tune the grid cadences on data instead of intuition.
The hub already sees 100% of fleet traffic. Provider dashboards hold the authoritative pool/wall state. Neither is joined today.
Phase 1 — live consumption table (hub-side, no auth)
scripts/traffic-consumption.ps1(or extendtraffic-summary.ps1): from hub captures + logs, a rollup table per provider-lane (gc@ GLM, mmc@ MiniMax, qwen-token-plan, ds@ PAYG, cx@ Sol, native Anthropic):cc_workloadis already in the request's billing headerAcceptance: one command prints the fleet table for a window; cron-vs-interactive split works; numbers reconcile with a manual capture count.
Phase 2 — provider dashboards via Playwright (user authenticates)
User offered to authenticate Playwright sessions. Per provider: z.ai coding plan, MiniMax coding plan, Qwen/DashScope token plan, DeepSeek PAYG (balance), Mistral subscription, OpenAI Pro credits, Anthropic usage (ai-01). Fetch pool/wall/reset state on a schedule, store JSON snapshots (never secrets), trend them.
Verify (provider-versioning discipline): each provider's actual dashboard structure + any off-peak pricing (DeepSeek PAYG documents discounted hours — capture the real hours from the dashboard, don't assume).
Phase 3 — join: crons → cost
Join Phase 1 + Phase 2 + the lane/cron registry (workspace, model, cadence): cost per lane, cache-miss rate per cron (fresh-context crons should show high cache_creation), heures creuses/pleines analysis (fleet self-congestion + wall timing + any provider off-peak pricing) → cadence recommendations.
Coordination
Epic coordinated by ai-01 (roo-extensions); hub-side phases live here. Phase 1 is unblocked today.