Repository navigation
fix: record concurrency as a run condition, not a plan field - #3
Merged
Merged
Conversation
contract.generation declared a concurrency it could not enforce: prepare wrote a hard-coded 1,
while run defaults to 16, and run emitted the same deviation string on every run, naming neither
value. A frozen plan cannot decide how many requests a client keeps in flight, so the field is
removed from the plan and the run records the condition it actually ran under.
- plan: drop concurrency from contract.generation and remove `prepare --concurrency`; contract_sha256
is again a statement about the evaluation alone (prompts, gold answers, limits, scorer)
- run: record `concurrency` plus an optional `expected_concurrency` (default: not checked); deviations
become structured {field, expected, actual, source} and appear only when a target was requested or a
resumed invocation changes concurrency
- run: name meta files run.<ts>.<seq>.json so two invocations in the same second no longer overwrite
each other (this used to silently drop the resumed invocation's record)
- summary: merge deviations across all invocations and report `invocations`
- docs and tests follow; 47 tests pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
contract.generation.concurrencydeclared something a plan cannot enforce:preparewrote a hard-coded1(there was no flag), whilerundefaults to16, so the declared valuewas false for the standard flow documented in
README.md.runappended the same deviation string on every run --"concurrency differs from contract.generation.concurrency"-- without comparing anything and withoutnaming either value. That is noise, not a record.
contract.json,contract_sha256changed when only the client's schedulingchanged, which weakens "same contract hash <=> same evaluation definition".
Change
Concurrency is a property of a run, not of the evaluation, so it moves to the run record.
plan: dropconcurrencyfromcontract.generation;prepare --concurrencyis removed.contract.jsonagain describes only what is evaluated (prompts, gold answers, limits, scorer) and what a request must carry.
run: recordconcurrencyplus an optionalexpected_concurrency.--expected-concurrency Nis opt-in(default: not checked); a differing
--concurrencyis written todeviationsas a structured{field, expected, actual, source}entry instead of free text.run: meta files are nowrun.<ts>.<seq>.json. Previously two invocations inside the same secondoverwrote each other, which could silently drop a resumed invocation's record.
run: a resumed invocation that changes concurrency is always recorded, whether or not a target was given --one run directory must not silently mix two run conditions.
run:concurrency < 1is rejected instead of failing insideThreadPoolExecutor.summary: read everyrun.*.json, merge deviations and reportinvocations; previously only the firstmeta was read, which would have hidden the resumed drift.
Verification
deviations == []; mismatch -> exact structured entry; match -> empty; resume drift-> both invocations keep their own meta and
summaryreports both;concurrency=0rejected.runreads onlytemperatureandseedfromgeneration, so the stale key is inert.Note
This replaces the earlier approach of writing the target into the contract. It keeps the useful part -- the run
records the condition it ran under, and drift stays detectable -- without making
contract_sha256depend on howmany requests a client had in flight.