fix: always reduce the early stopping set to available features in train_final_model - #462
Open
jfrog64 wants to merge 3 commits into
Open
Conversation
train_final_model() only reduced the early stopping dataset to the cell
lines and drugs that actually have features when a response
transformation was configured, because the reduce_to() call was nested
inside the `if response_transformation:` branch. Without a
transformation the early stopping set kept samples whose features are
missing, and training the final model aborted with
AssertionError: 4 of 21 ids are not in the FeatureDataset.
Missing ids: {'COLO 205', 'BT 271', 'OVCAR-5', 'DLD-1'}
raised by FeatureDataset.get_feature_matrix() as soon as the model
looked up the early stopping features. This affects every model with
early_stopping = True (DIPK, SimpleNeuralNetwork, MOLIR, SuperFELTR,
PharmaFormer, ...) when --final_model_on_full_data is used without
--response_transformation, i.e. the plain default setup: the whole
cross-validation completes and only the final model fails.
The reduction is now performed unconditionally, mirroring what
train_and_predict() already does, and the transformation branch only
transforms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The test drives train_final_model with a minimal early stopping model whose features cover only half of the cell lines and no response transformation. It asserts the precondition (the raw early stopping split does contain cell lines without features) so it cannot go vacuous, and then that train() only sees cell lines that have a feature row. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The parameter was annotated as TransformerMixin, but None is a valid and in fact the default value: run_suite declares it as TransformerMixin | None, get_response_transformation returns None for "None", and the body guards with `if response_transformation:`. It is the only one of the twelve response_transformation parameters in experiment.py without | None. mypy does not catch this because sklearn ships no type stubs, so TransformerMixin is Any; the typeguard session does, as soon as a test calls train_final_model without a transformation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## development #462 +/- ##
===============================================
+ Coverage 80.34% 82.12% +1.78%
===============================================
Files 101 121 +20
Lines 8171 9831 +1660
===============================================
+ Hits 6565 8074 +1509
- Misses 1606 1757 +151 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Checklist for all PRs
docsis updated — no user-facing API change, so nothing to updateChanges
Bug fixes
In
train_final_model, the training set is reduced to the cell lines and drugs that actually havefeatures, and a matching reduction exists for the early stopping set. That second reduction, however,
sits inside
if response_transformation::So whenever a final model is trained with early stopping but without a response transformation
(the default), the early stopping set is never reduced. The model then receives early stopping cell
lines / drugs that have no feature row, while the training set has already been filtered. Depending
on the model this surfaces as a
KeyErroron the missing identifier, or as a silently wrong earlystopping signal.
The reduction is not conditional on the transformation — it depends only on which features are
available — so this moves it out of the
ifblock. The transformation call stays where it was.Tests:
tests/test_final_model_early_stopping.pydrivestrain_final_modelwith a minimal earlystopping model whose features cover only half of the cell lines and with
response_transformation=None.It first asserts the precondition (the raw early stopping split really does contain cell lines without
features) so the test cannot go vacuous, then asserts that
train()only ever sees cell lines thathave a feature row. Against the current
developmentthe test fails; with the fix it passes.The full suite is green locally (219 passed).
New features
Maintenance
Follow-up commit:
response_transformationannotationThe
typeguardsession flagged the regression test, becausetrain_final_modelwas annotatedresponse_transformation: TransformerMixin— without| None. The test has to passNone, since thebug only occurs on the path without a response transformation.
The annotation is simply wrong, and it is the only one of the twelve
response_transformationparameters in
experiment.pythat lacks| None:experiment.py:463, forwardsrun_suite'sresponse_transformation: TransformerMixin | None = Noneget_response_transformationis declared-> TransformerMixin | Noneand returnsNonefor"None"train_final_modelitself guards withif response_transformation:mypycannot catch this: sklearn ships no type stubs, soTransformerMixinisAnyand passingNoneis unremarkable. Only the runtime check in the
typeguardsession sees it. The second commit widens theannotation to
TransformerMixin | Noneand updates the docstring; no behaviour changes.