Skip to content

fix(langfuse): review fixes 3 - branch-quiet notify, 1h cache-write price, lean scorer, trigger label, memory byte budget - #1232

Merged
aaronjmars merged 3 commits into
mainfrom
fix/langfuse-review-3
Oct 16, 2026
Merged

aaronjmars merged 3 commits into
mainfrom
fix/langfuse-review-3

Conversation

@aaronjmars

@aaronjmars aaronjmars commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes from the 2026-10-16 Langfuse review of aeon-agent and miroshark-aeon (follow-up to #1230).

  1. No real notifications from test branches. "Send pending notifications" now needs github.ref_name == github.event.repository.default_branch, like the health-issue step fix(langfuse): branch-gated health issues, traced feed render, prices, usage-row timing #1230 gated. workflow_dispatch runs of holdings on verify/langfuse-models / verify/langfuse-slicing delivered 4 real Telegram messages ("Delivered 1/1" in aeon-agent runs 38778704213, 38789847709 and miroshark-aeon runs 38779134169, 38789854033). Repo variable NOTIFY_ON_BRANCH=1 opts a test branch back in. Skipped payloads live in $RUNNER_TEMP, so nothing is redelivered later. Documented in docs/CONFIGURATION.md.
  2. 1-hour cache writes priced at the 1-hour rate. Each usage <model> row now also sends langfuse.observation.usage_details with the Langfuse price keys (input, output, input_cache_read, input_cache_creation_1h, input_cache_creation_5m). Langfuse takes that attribute over gen_ai.usage.*, which has one cache-write count it priced at the 5-minute rate (1.25x input). Claude Code on a subscription writes 1h caches (2x), so cost read ~35% low on every Sonnet run (the reported aeon.cost_usd_reported fit $4/M writes on 7 of 7 runs). The split comes from the envelope's usage.cache_creation.ephemeral_1h_input_tokens / ephemeral_5m_input_tokens (checked in a real Claude Code 2.1.301 --output-format json result; modelUsage has no split, so with more than one model each gets the run's 1h share). No split in the envelope = one input_cache_creation count, as before. gen_ai.usage.* stays for other OTLP backends. fleet-scorecard.mjs prices cache writes at 2x input too (token-usage.csv has no TTL column).
  3. Lean scorer. New run-harness --no-tools (claude: --tools ""; other harnesses ignore it). The claude scorer uses it and runs from an empty scratch dir, the way fix(langfuse): branch-gated health issues, traced feed render, prices, usage-row timing #1230 runs the feed render, so a 1-turn text grade no longer loads tool schemas and the repo CLAUDE.md (~18.7k read + ~18k write per call on both instances). Other harnesses keep their current call.
  4. Trigger label tells cron from manual. Every skill starts as workflow_dispatch, so trigger was workflow_dispatch for cron, chain and manual runs alike. New optional aeon.yml input trigger: the Scheduler passes cron, chain-runner chain; langfuse-otel.sh maps them to cron / chain / manual (only those values, the input is free text). Both dispatchers retry once without the field when an older aeon.yml answers "Unexpected inputs", so a fork that syncs the Scheduler before its aeon.yml keeps running.
  5. MEMORY.md byte budget. aeon-agent's MEMORY.md is 88 lines but 27.7 KB and loads into every run; the cap was lines only. memory-flush now holds it to 8 KB with no line over 300 chars (step 3e), and memory_prep.py window prints the measured size plus the longest lines so the model trims by numbers. CLAUDE.md (and AGENTS.md) say the same.

Verified:

  • A real usage_details row sent to the aeon-agent Langfuse project (environment fix-test): 100k 1h writes priced $0.40 (input_cache_creation_1h), total $0.422 for the row.
  • claude -p --output-format json envelope fields checked locally; run-harness claude --no-tools from a scratch dir returns a valid envelope (48.5k -> 19.6k context tokens on a laptop with a large global CLAUDE.md; on a runner the drop is to the bare system prompt).
  • Local: test_otel_span.sh (1h/5m split per model + no-split fallback), test_langfuse_otel.sh (cron / chain / manual / free-text / non-dispatch), test_memory_prep.py (size report), test_fleet_scorecard.mjs, test_chain_runner*.sh, the full ci-tests list (only test_chain_runner.sh fails on macOS, same on main: BSD grep has no -P), validate-config.js, gen-agents-md.js --check, actionlint, shellcheck error gate.

Instances: aeon-agent and miroshark-aeon get the aeon.yml hunks for 1 and the trigger input now (instance PRs), so the Monday aeon-update brings the Scheduler / chain-runner / script changes onto a workflow that already has the input. The scorer hunk (3) needs the new run-harness flag, so it arrives with that sync.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…rice, lean scorer, trigger label, memory byte budget

From the 2026-10-16 Langfuse review of aeon-agent and miroshark-aeon.

1. Send pending notifications runs on the default branch only (repo var
   NOTIFY_ON_BRANCH=1 opts a test branch in). verify/* test dispatches of
   holdings sent 4 real Telegram messages.
2. Usage rows send langfuse.observation.usage_details with the 1h/5m cache-write
   split from usage.cache_creation, so 1-hour writes get the 1h price (2x input).
   Langfuse priced them at the 5m rate, ~35% under per Sonnet run.
   fleet-scorecard prices cache writes at the 1h rate too.
3. The claude scorer runs with run-harness --no-tools (new, claude only) from an
   empty scratch dir, like the feed render: no tool schemas, no repo CLAUDE.md.
4. trigger metadata: the Scheduler passes trigger=cron and chain-runner
   trigger=chain (new aeon.yml input; retried without it on an older aeon.yml),
   langfuse-otel.sh maps them to cron / chain / manual.
5. memory-flush holds MEMORY.md to a byte budget (8 KB, no line over 300 chars);
   memory_prep.py window prints the measured size and the long lines.
@blacksmith-sh

This comment has been minimized.

@aaronjmars
aaronjmars merged commit 3dc55bb into main Oct 16, 2026
16 checks passed
aaronjmars added a commit to aeonfun/aeon-agent that referenced this pull request Oct 16, 2026
…dings windows in code (#267)

* fix(notify): deliver only from the default branch; add the trigger input

Instance copy of the aeon.yml hunks in aeonfun/aeon#1232. Send pending
notifications now needs the default branch (repo var NOTIFY_ON_BRANCH=1 opts a
test branch in); verify/* test dispatches of holdings sent real Telegram
messages. The optional trigger input and AEON_DISPATCH_TRIGGER env are inert
until the Scheduler / chain-runner / langfuse-otel.sh changes arrive with
aeon-update.

* fix(holdings): pick the 7d/30d windows in holdings.py

The model did the nearest-snapshot math by hand and broke its own +/-2 day
rule on an off-schedule run (sent 7d vs a 10-day-old snapshot). holdings.py now
reads the HOLDINGS_STATE lines (memory/logs and logs/archive) and prints
growth.<symbol>.{last,7d,30d} or null; step 2 only renders it. --no-history for
a one-off var wallet. The notify card gains a one-line sources footer, which the
scorer docked the bare stat line for.
aaronjmars added a commit to MiroShark/miroshark-aeon that referenced this pull request Oct 16, 2026
…dings windows in code (#214)

* fix(notify): deliver only from the default branch; add the trigger input

Instance copy of the aeon.yml hunks in aeonfun/aeon#1232. Send pending
notifications now needs the default branch (repo var NOTIFY_ON_BRANCH=1 opts a
test branch in); verify/* test dispatches of holdings sent real Telegram
messages. The optional trigger input and AEON_DISPATCH_TRIGGER env are inert
until the Scheduler / chain-runner / langfuse-otel.sh changes arrive with
aeon-update.

* fix(holdings): pick the 7d/30d windows in holdings.py

The model did the nearest-snapshot math by hand and broke its own +/-2 day
rule on an off-schedule run (sent 7d vs a 10-day-old snapshot). holdings.py now
reads the HOLDINGS_STATE lines (memory/logs and logs/archive) and prints
growth.<symbol>.{last,7d,30d} or null; step 2 only renders it. --no-history for
a one-off var wallet. The notify card gains a one-line sources footer, which the
scorer docked the bare stat line for.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant