The loop was missing its transition, and two demos were flattering themselves - #2
Open
di-omics wants to merge 1 commit into
Open
The loop was missing its transition, and two demos were flattering themselves#2di-omics wants to merge 1 commit into
di-omics wants to merge 1 commit into
Conversation
…e it This repo already had every stage of a perception-action loop lifted into a named, swappable module except one. labcv.synth.tip_view and column_view are observation models: they take a scalar physical state and render the frame a camera would see. verify reads a frame back. decide chooses an action. What was missing was the function that says what the action did, and it lived as two magic constants at demos/pipette_cam/run.py:47 applied inline nine lines later. That function is a world model, and it now has an interface. labcv/dynamics.py names it: an AnalyticTransition carrying the constants the demo used, a PlantedTransition drawing per-episode coefficients, and an InjectedTransition for a learned model. labcv/bridge.py promotes the action and verdict types out of the demo first, because a library must not import from a demo. labcv/episodes.py generates action-conditioned rollouts, which is new: this repo produced passive observations, and these are sequences where the scene changed because of a commanded action. Unknowable episodes are constructed by independence rather than asserted, and the build refuses rather than warns: an occluder must cover 100 percent of the target region of interest in every frame, checked pixel-wise against the boxes synth.microplate returns and again by byte-identity of that region against a render at the other level. export writes a versioned manifest, which is the entire interface to the consumer: no package imports either direction. A scoring pack withholds the coefficients and the future frames but still carries knowability and the hidden latent, because the consumer's separation gate reads that latent to decide which grid a label is checked over, and a pack without it carries labels nothing can ever check. That leaves a real gaming channel, so the manifest now names it in a disclosed_to_harness field: a submitter reading the file directly can abstain on exactly the right episodes, and a scoring split is an adversarial holdout only where the harness mediates access. Better in the file than in a README beside the data. Two demos were asserting things they had not measured: run_qc.py set the corrected volume to the target exactly, which is a transition model claiming every correction lands with zero error. It now commands against the camera reading. Every published number is unchanged because the residual stays inside tolerance, and the line saying corrected wells land 0.24 uL from target rather than on it is the point. vocab_vlm computed an escalation mask and then sent every box to the VLM anyway, so "52 percent of calls saved" described a branch nothing took and no test could have noticed: the accuracy was identical either way. The gate now gates. At the published configuration the numbers hold and are finally true, 13 calls made instead of 27. At seed 11 accuracy falls from 1.000 to 0.889 and the demo fails, because three specks take the cheap path and that path has no word for a bubble. Raising the threshold would fix the number by fitting it to the data it judges, so the demo prints the cost of the saved calls instead and fails honestly. Adds test and lint CI. This repo had house-rules CI before it had test CI, and a red suite sat in it unnoticed, because the unittest runner these modules use aborts without printing a total and tailing its output looks exactly like a pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The observation
This repo already had every stage of a perception-action loop lifted into a named, swappable module except one.
labcv.synth.tip_viewandcolumn_vieware observation models: they take a scalar physical state and render the frame a camera would see.verifyreads a frame back.decidechooses an action. What was missing was the function that says what the action did, and it lived as two magic constants atdemos/pipette_cam/run.py:47applied inline nine lines later:That function is a world model. It now has an interface.
What is added
labcv/bridge.py-- promotes the action and verdict types out of the demo, because a library must not import from a demo. Load-bearing:dynamics.pycannot be written without it.labcv/dynamics.py--Transitionprotocol,AnalyticTransition(the constants, finally named),PlantedTransition,InjectedTransition.labcv/episodes.py-- action-conditioned rollouts. This is the new capability: the repo produced passive observations, and these are sequences where the scene changed because of a commanded action.eval/rollout_metrics.py-- terminal error, in-tolerance, trajectory RMSE, first divergence step.Unknowability is constructed, not asserted, and the build refuses rather than warns: an occluder must cover 100% of the target ROI in every frame, checked pixel-wise against the boxes
synth.microplatereturns and again by byte-identity of that region against a render at the other level.A scoring pack withholds coefficients and future frames but still carries
knowabilityand the hidden latent, because the consumer's separation gate needs that latent to check the labels at all. That leaves a real gaming channel, so the manifest now names it indisclosed_to_harness: a submitter reading the file directly can abstain on exactly the right episodes, so a scoring split is an adversarial holdout only where the harness mediates access. Better stated in the file than in a README beside the data.Two demos were asserting things they had not measured
run_qc.pyset the corrected volume to the target exactly, which is a transition model claiming every correction lands with zero error. It now commands against the camera reading. Every published number is unchanged (the residual stays inside tolerance) and the demo now says corrected wells land 0.24 uL from target rather than on it.vocab_vlmcomputed an escalation mask and then sent every box to the VLM anyway. The published "52% of VLM calls saved" described a branch nothing took, and no test could have caught it because the accuracy was identical either way.The gate now gates. At the published config the numbers hold and are finally true: 13 calls made instead of 27, accuracy still 1.000, 52% saved.
At other configs, enforcing it exposed what the un-enforced version was hiding:
--seed 11--distractors 8Three specks score above the threshold, take the cheap path, and that path has no word for "bubble". Raising the threshold would fix the number by fitting it to the data it judges, so the demo prints the cost of the saved calls and fails honestly instead.
CI
This repo had house-rules CI before it had test CI, and a red suite sat in it unnoticed, because the unittest runner these modules use aborts without printing a total and tailing its output looks exactly like a pass. Adds test and lint jobs, including a step that counts scored verdicts so a demo cannot stop scoring silently.
Verification
83 tests pass,
make allexits 0 with all 8 demos passing, ruff clean under the same rule set the sibling repo enforces, house-rules clean. Thepipette_camblock ofmake allis byte-identical to before this change.