Conversation
OpenAI-compatible streams omit the usage frame unless stream_options include_usage is sent, so every successful streamed attempt recorded usage: null — blinding cost observability for the production pro rewrite stage and making PAY-B-COST billing evidence unassemblable (found assembling the first G002 bundle). The SSE parser already captures a usage chunk when present; this sends the request flag, verified against the Anthropic compat endpoint. The collector's billing classification also corrects to usage-presence (an errored attempt with usage was still metered).
…designed Three probes measured on the real pipeline (uncached claude-sonnet-5, standard $3/$15 pricing): upper COGS $878/1M chars, dominated by the fixed ~17k-token catalog prompt per call — the char-scaled worst case collapses at short inputs. The 60% gate refuses the current shape ($9.99, 1M chars/mo) at -10,243% margin. Raw bundle persisted so cap, price, and model scenarios recompute offline with zero further spend. derivePayBCostFinancial exported for refusal reporting and scenario analysis; the collector persists the bundle before issuing.
The OpenAI-compat endpoint ignores prompt caching in every form
(verified empirically: content-block and top-level cache_control both
bill full price on repeat). With ~34k static catalog tokens per paid
call, caching is the difference between $0.164 and ~$0.07 per pro
rewrite — so the paid path gets a first-party /v1/messages branch:
- Opt-in via PATINA_ANTHROPIC_NATIVE_CACHE + first-party host check;
zero caller-surface change (api.js / streaming-api.js branch
internally), compat behavior byte-identical when off.
- Zero prompt-semantics change: the prompt stays one user message,
split at the first input fence into a cache_control prefix block
(the static catalog) and the dynamic tail. Refine prompts fall under
the cache minimum and skip caching — no write-only cache churn.
- Native SSE parser maps message_start/content_block_delta/
message_delta into the existing delta/attempt plumbing; usage keeps
Anthropic field names (cache-token extractor and the G002 usage
adapter already accept them).
- sonnet-5 rejects temperature natively too ('deprecated for this
model'): the native body consults the shared rejection memo and the
temperature_schema retry now covers both paths.
Live verification: first call cache_creation 34,254 tokens; second call
cache_read 34,254 at the 0.1x rate — 90% input-cost cut confirmed.
Suite: 8 new adapter tests, full gate green.
…ched bundle - Thinking stays at the provider default: the A/B showed thinking-off rewrites amputate content (fidelity 50, judged by a thinking-on scorer), and quality is what pro sells. PATINA_ANTHROPIC_THINKING=0 remains as an experiment-only opt-out. Measured context: thinking is 76% of pro-path output tokens. - G002 usage adapter v2 accepts the first-party 2026 usage shape (cache_creation breakdown, thinking details, routing metadata) and fail-closes on 1-hour cache writes that would break single-rate pricing. - Cached measurement bundle checked in: warm-cache per-request cost $0.0892 (output tokens 80% of it), vs $0.164 uncached.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Self-judging (candidate model grades its own rewrite) made cross-model
comparisons incomparable and noisy run-to-run — measured on the Token Plan
screening (2026-07-25) where glm-5.2 verdicts flipped between identical runs.
- PATINA_LIVE_JUDGE_{MODEL,PROVIDER,API_BASE,API_KEY,TIMEOUT_MS} env vars
and --judge-* flags pin scoreText/scoreMPS/scoreFidelity to one judge
- judge on a different host never inherits the primary credential (fails
closed with a clear error instead)
- report records settings.judge; markdown header prints judge: <model>
- default behavior unchanged (self-judging) when no judge is configured
…ccounting Judge selection needs measured speed and quota cost, not estimates, and cost is easily distorted by hidden reasoning tokens, cache reads, and schema-retry doubling. Every live call now records wall ms, paid attempt count, and per-attempt normalized usage (OpenAI-compat + native Anthropic shapes): prompt/completion/reasoning/cached-read/cache-write tokens. Results carry usage.candidate / usage.judge; the report sums summary.usage. Failed paid retries stay billed via per-attempt usage.
gpt-5.5 — the best measured judge (AUC 1.00, 2026-07-13 calibration) — and claude are reachable through logged-in subscription CLI seats without any API key. PATINA_LIVE_JUDGE_BACKEND / --judge-backend routes the scoring calls (scoreText/scoreMPS/scoreFidelity) through the local backend chain (codex-cli, claude-cli, gemini-cli, kimi-cli); the key fail-closed gate is skipped for backend judges since the seat is the credential. CLI backends report no token usage, so cost accounting records calls and wall time only. Verified live: claude-cli judge scored a Token Plan candidate end-to-end (zero marginal cost); codex-cli routing works but the seat's refresh token is revoked upstream (re-login is an operator action).
The migrated Kimi Code CLI (0.28.x) removed --print/--input-format/ --final-message-only/--no-thinking/--max-steps-per-turn, which left the kimi-cli backend fully broken (every invoke exited 1 with 'unknown option'). - modern invocation: --prompt <text> (argv; the CLI no longer reads stdin) with --output-format stream-json, recovering the final assistant message from NDJSON events (same semantics as the retired --final-message-only) - legacy CLIs keep working via an 'unknown option' fallback to the old stdin --print invocation - security stance unchanged: never --yolo/--auto, tools stay unapprovable in non-interactive prompt mode - isAuthenticated now also checks ~/.kimi-code (kimi-code data dir) so auth detection survives a legacy ~/.kimi cleanup after 'kimi migrate'
Local CLI backends serialize on a concurrency cap of 1, so the parallel Promise.all scoring pass only burned each call's budget inside the slot queue (observed: 4 x 300s all-timeout with a kimi judge), and the single shared absolute deadline expired before later calls could start. Backend judges now score sequentially with a per-call timeout and no shared deadline; the HTTP judge path is unchanged. Verified live with codex-cli (gpt-5.5, 4 calls / 55.6s) and kimi-cli (stream-json verdicts returned).
…rovider-key reference .env.example led with 35 lines of the retired ops harness (DISCORD_*/ MARKETING_*/PATINA_RUNTIME_CLI) that nothing in this snapshot reads — AGENTS.md already declares that harness absent. Meanwhile the keys the CLI actually resolves (src/auth.js HTTP_KEY_ENV_VARS, src/providers.js presets) were undocumented. Lead with the provider-key reference and a pointer to the subscription-CLI and fixed-judge (PATINA_LIVE_JUDGE_*) docs; the hosted-service section is unchanged.
Provider docs review + live probes confirmed every reasoning-default
provider exposes a body-level off-switch the harness could not send:
DeepSeek v4 thinking:{type:disabled} (193s -> 1.6s single-call probe),
Gemini OpenAI-compat reasoning_effort (accepted on 3.6-flash), Alibaba
enable_thinking:false (4.5x). Reasoning is the dominant judge cost
distortion (measured 93-95% of output tokens).
- src/api.js callLLM accepts extraBody, spread into the OpenAI-compat
body before protocol fields so model/messages can never be clobbered;
ignored on the native Anthropic path
- live-quality: PATINA_LIVE_EXTRA_BODY / PATINA_LIVE_JUDGE_EXTRA_BODY
(+ --extra-body / --judge-extra-body) parse a JSON object and ride
every candidate/judge call; junk input fails fast
…soning judges fail Admitted three HTTP challengers to the 2026-07-13 calibration under the same pre-registered criteria (44-doc KO corpus, same prompt, PASS = AUC >= 0.75 & median repeat SD <= 12). 192/192 challenger calls parsed, 0 lost. - gemini-3.6-flash: accuracy 0.91, AUC 0.96 [0.91, 1.00], repeat SD 2.2, 4.6s/call on a near-free tier -> PASS (second seat; gpt-5.5 keeps the 1.00) - grok-4.20-non-reasoning: AUC 0.71, called 23 of 24 AI docs "human" -> WATCH - deepseek-v4-flash thinking-off: AUC 0.70, 20 of 24 missed, SD 12.5 -> WATCH Speed bought nothing for the non-reasoning judges: their human/AI mean gap is 4-9 points versus 53-65 for every PASS judge, and their low repeat SD is the consistency of always answering ~20, not stability. Harness: HTTP judge transport (records per-call latency + usage next to the verdict), --judges filter for bounded resumable passes, kimi invocation moved to the Kimi Code >= 0.28 argv/NDJSON shape. The pre-registered pooled panel stays defined as the original 3-judge 2-of-3 mean.
v6.2 made the persona the sole voice owner and both shipping surfaces moved with it — src/cli/run.js and src/web-rewrite.js each call resolvePersonaForRun before building the prompt. The live-quality harness never did, so it measured a third prompt that nothing serves. For Korean the missing piece is the persona directive itself: '원문의 주장·사실·수치·인용·논지 순서를 100% 보존하고 어투·리듬·어휘·문장구조만 페르소나에 맞춘다'. Every meaning-preservation number this harness produced was taken from a rewrite that was never told to preserve meaning, so MPS failures here were biased against the product — including the five fixtures recorded this week as an open meaning-loss problem. With the fix the harness prompt is byte-identical to the hosted rewrite prompt (41,775 chars, diff 0), pinned by a parity test so the two cannot drift again. Every measurement in docs/operations from 2026-07-25/26/27 predates this and should be re-run before it is cited.
Rerun with the harness prompt now byte-identical to the hosted one, on subscription seats (gemini-3.6-flash rewriting, gpt-5.5 judging, zero cost): | | broken rubric | fidelity fixed | + prompt parity | |---|---|---|---| | pass | 9/22 (41%) | 17/22 (77%) | 20/22 (91%) | | MPS mean | 76.3 | 80.5 | 90.4 | | ko | 2/11 | 5/11 | 11/11 | Every register recorded this week as systematically broken now passes: ko-instructional MPS 20 -> 100, ko-marketing 50 -> 100, ko-social 40 -> 100, ko-blog 45 -> 70, ko-product 50 -> 80. Neither defect was in the rewriting engine; both were in the apparatus measuring it. Two English fixtures remain and fail in opposite directions: en-marketing-01 strips hype thoroughly (AI 35.6 -> 5.7) but drops anchors (MPS 60), while en-public-docs-01 holds meaning and barely moves the AI score (15.6 -> 16.5). Both are single fixtures; the handoff now says to add a second fixture per register before reading either as a pattern.
The comparison that moved both tiers to gemini-3.6-flash was taken under a broken fidelity rubric and a harness prompt missing the persona, so it was void. Rerun on the same 22 fixtures with both fixed, on subscription seats (candidates via their own seats, judge gpt-5.5 via codex, zero API spend): | | gemini-3.6-flash | claude-sonnet-5 | |---|---|---| | pass | 20/22 (ko 11/11, en 9/11) | 20/22 (ko 11/11, en 9/11) | | MPS mean | 90.4 | 91.5 | | fidelity mean | 92.4 | 94.7 | | $/rewrite | $0.030 | $0.156 | | s/rewrite | 8.3 | 27.7 | The decision holds but its justification does not: the two engines are level on quality, and sonnet-5's 1.1-point MPS edge is inside one fixture of noise at n=22. The case for gemini is cost and latency, not rewrite quality. Docs and .env.example now say that instead of the void 'AI improvement 13.0 vs 11.8' figures; the .env.example freeze hash is revised in both ledgers. Still void and not rerun: deepseek-v4-flash, grok-4.3, gpt-5.3-chat-latest, gpt-5.4-mini, gpt-5.6-luna, and the gpt-4.1-mini result that motivated moving the free tier. en-marketing-01 fails on both engines, which points at that fixture or the prompt rather than at either model.
Reran the cheap candidates on the fixed apparatus to see whether the shipped engine could be undercut. It cannot, and the two failure modes are mirror images of each other: | engine | pass | MPS worst | not-improved | $/rewrite | |---|---|---|---|---| | gemini-3.6-flash | 20/22 | 60 | 1 | $0.030 | | claude-sonnet-5 | 20/22 | 50 | 0 | $0.156 | | deepseek-v4-flash | 18/22 | 24 | 1 | $0.003 | | gemini-3.5-flash-lite | 10/22 | 40 | 8 | $0.007 | deepseek-v4-flash is 10x cheaper and still rewrites hard, but gutted ko-news-01 to MPS 24. gemini-3.5-flash-lite fails the opposite way: four Korean fixtures score MPS 100 while the AI score barely moves, i.e. it returns the input nearly unchanged — the same evasion that made gpt-4.1-mini look like a leader under the broken apparatus. Holding both ends at once is the real difficulty here, so $0.030 per rewrite is recorded as the floor rather than a number to shave. The OpenAI-hosted candidates stay unmeasured on an exhausted balance; none is in production and, given both cheaper models failed in opposite directions, none looks promising.
patina rewrote marketing copy and added 'No hidden commitments—cancel anytime'
plus 'see how much time you save every day'. Neither appears in the original.
Inserting contractual promises into a user's published copy is a liability, not
a style choice, and nothing in the default prompt forbade it: the only 'never
invent' rule was about passive-voice actors, and the fact/number prohibition
lived exclusively inside the --transform block that most runs never emit.
Meanwhile the marketing profile actively demands specificity ('replace a vague
future-promise ending with a concrete CTA', 'who, where, in what context?'). On
hype-only source text there is no specific fact to supply, so the model
invented one.
Added, unconditionally: Phase 2 now forbids adding any claim, fact, number,
guarantee, or commitment absent from the source, and instructs cutting the
vague sentence instead of inventing a replacement; Phase 3 self-audit gains a
check that every claim, number, and promise traces back to the input. Also
corrected the length rule, which still cited the pre-2026-07-27 70-130% band
while lengthRatioPoints now scores 50-130% — the model was self-censoring
against a threshold that no longer existed.
Verified qualitatively on en-marketing-01: both fabricated commitments are gone
and the three real anchors (7-day trial, $9/month, all features) survive.
Honest about the aggregate: 22 fixtures scored 20/22 before and after, with MPS
mean 90.4 -> 88.2. Per-fixture scores swing +-20 MPS between identical runs, so
this change is not measurable at one run per fixture. It is kept on correctness
grounds, and the variance is itself the finding — single-run sweeps cannot
validate anything at this effect size.
Every conclusion drawn this week rested on one sample per fixture, and that is not enough: identical configurations swing +-20 MPS per fixture. ko-blog-01 scored 45 in one sweep and 100 in three consecutive reruns, and the no-fabrication prompt change measured as a 2-point regression in aggregate while its intended effect was plainly visible in the rewrite text. Three separate conclusions this week were revised after resampling. --repeat N samples each fixture N times. Reported scores are medians, the status is the worst sample so repeating can only expose instability rather than average it away, and result.repeat carries every value with its spread for inspection. Paired with the subscription seats the extra samples cost nothing, so there is no reason to run a decision-grade comparison at N=1 again.
Production returned provider 429s to free-tier users three times on 2026-07-27 and nothing alerted. The monitor exists and runs every 15 minutes, but it evaluates tier 'pro' — which has no users while checkout is disabled — so its aggregate is permanently zero, its one live signal (monitor_blind) fires constantly as background noise, and its synthetic probe is pro-only, hourly, and needs three consecutive failures. The free counters were being written the whole time under patina:mon:v1:<channel>:free:*; only the reader was missing. evaluateFreeTierHealth reads them, in two layers because either alone has a blind spot: - aggregate ratio: alerts when >50% of at least 5 requests in the window did not complete. Costs nothing and runs every tick, but is silent at zero traffic. quota_denied is excluded — that is the product working. - HTTP canary: a real free rewrite, for when traffic is zero and the aggregate cannot tell idle from down. It consumes the free IP quota (20/day), so a lease budgets it to one probe per two hours; probing every tick would exhaust the quota and manufacture its own alerts. Wired into the existing cron, so no new endpoint and no vercel.json change (that file is frozen). It runs after the paid evaluation and can never fail the cron: a canary problem must not mask the pro evidence run, and the status code still speaks for the paid path alone. Detection goes from never to at most two hours at zero traffic, and to one cron tick once anyone is actually using the service.
…g pastes Pasting any text containing HH:MM clock times (chat logs, meeting notes, timelines like '16:47 - 16:50') always failed the web rewrite with a generic '리라이트 실패' — digit:digit hit NUMERIC_OPERATOR_RE, so addClaims fail-closed on the ORIGINAL text alone and every rewrite, including an identity rewrite, emitted number_safety_failed before scoring. Retry could never succeed and the 'check the mode/key' copy was wrong. - meaning-proxy: claim HH:MM(:SS) as exact time: claims before the operator check. Two-digit minutes keep ratios/scores (1:2, 3:1) and invalid times (25:30, 16:75) fail-closed exactly as before; time drift or dropped times fail as numeric_claim_changed. Safety is not weakened: reject-always becomes must-match-exactly. - playground: number_safety_failed now classifies to a dedicated NUMBER_SAFETY kind with honest localized copy (en/ko/zh/ja) instead of the generic mode/key failNote. Verified: unit suite (1481 pass), full test run, lint, plus a live byok e2e of the originally failing paste through runWebRewriteStream (gemini-3.6-flash): start -> 16 deltas -> done, MPS 100 / fidelity 100, all timestamps preserved verbatim.
…screen readers The .output-status line under every patina message showed 'Unapproved — checks have not passed' from the moment streaming started, which reads as an alarming warning during a perfectly normal in-flight rewrite (10-60s with a real model), then flipped to 'Approved' on done. Sighted users already get the flagged border, floorWarn, and error notes for real failures, so the visible line is redundant noise. Clip the element out visually (standard sr-only pattern) while keeping the role=status aria-live region, the localized copy, and the unapproved/approved dataset markers untouched — the a11y contract pinned by playground-pro.test.js is unchanged and all 9 assertions still pass.
Bump 6.3.1 -> 6.3.2 across every version-bearing surface (package.json, lockfile, README x4 badges + config examples, SKILL.md, .patina.default.yaml, playground ver-badges, patina-humanizer alias, Claude plugin manifests, release-metadata test pin) and add the 6.3.2 CHANGELOG entry. Re-freeze the v6.4 preflight hold SHA-256 manifest (script + JSON, 12 files) for the version-bump mutations only — every blocker, decision, prohibition, and provider semantic is byte-identical, matching the precedent of 8ffb920. The 6.4 tag/publish prohibitions remain in force; this release is authorized because the guard scopes them to 6.4.x. Refresh the checked-in benchmark reports (content unchanged; timestamp and node-version metadata only). Gates: release:check OK for 6.3.2, npm test 1620 pass (hold validator green on the new freeze), lint clean, benchmark 100%, dogfood under 30.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The paid-path cost work, measured end to end: native /v1/messages adapter with prompt caching (opt-in flag, 90% input-cost cut verified live), thinking control defaulting to provider-on (quality A/B showed thinking-off amputates content), G002 usage adapter v2 for the 2026 first-party usage shape, cached measurement bundle checked in (warm per-request $0.0892 vs $0.164 uncached). Production behavior unchanged until PATINA_ANTHROPIC_NATIVE_CACHE is set in Vercel. Gate: full suite + 10 adapter tests green.