schema-firewall

Three checks that catch the leakage and schema bugs that slip past peer review.

pip install schema-firewall

Production usage. Extracted from the firewall layer of nyc-real-estate-predictor — the flagship pins schema-firewall==0.1.0 in requirements.txt and re-validates the firewall integration in its External Benchmark CI job on every push. Directional coupling signal (pinned dep + consuming CI), not a semantic contract invariant.

---

The problem

In the last five years, published and competition-grade ML systems have repeatedly shipped with one of these three bugs:

Bug	Real example	Impact
Feature statistically mirrors the target	COVID-19 chest X-ray classifiers learned hospital-ID confounders, not pulmonary features	Internal AUC 0.99, external-hospital AUC near-chance
Forbidden / post-outcome feature in the input	JAMA Network Open 2024: 40.2% of MIMIC same-admission prediction studies fed in ICD codes finalised at discharge	AUROC 0.97 from leaky codes alone
Transform that reads across the whole dataset	Kaggle Santander 2019 "magic" leak: frequency features computed on (train ∪ real-test)	Public AUC jumped 0.90 → 0.92

Each one escaped peer review, code review, or competition scrutiny — because the bug isn't a type error. It's a statistical / semantic contract violation.

schema-firewall provides three drop-in checks, one per bug class.

---

Usage

import pandas as pd
from schema_firewall import (
    check_leakage,
    check_schema,
    check_stateless,
    SchemaContract,
    LeakageError,
)

X: pd.DataFrame  # your feature frame
y: pd.Series     # your target

# 1. Statistical leakage — Pearson + Spearman + adjusted mutual information.
#    Pearson catches linear copies, Spearman monotonic transforms, and the
#    chance-corrected MI catches NON-monotone and discrete deterministic leakage
#    (y=x**2, |x|, low-order oscillations, binary/k-class target encodings) that
#    both correlations miss — while leaving honest noisy predictors alone.
#    (High-frequency oscillatory encodings are a documented non-goal.) Needs
#    >=100 rows. Raises LeakageError on fail.
check_leakage(X, y)

# 2. Schema contract — forbidden columns, required columns, dtypes.
#    Catches ICD-code-style post-outcome features and schema drift.
contract = SchemaContract(
    forbidden_columns=frozenset({"SALE PRICE", "PRICE_PER_SQFT"}),
    required_columns=frozenset({"sqft", "year_built"}),
)
check_schema(X, contract)

# 3. Statelessness — runs your feature pipeline on the full frame vs a
#    single-row subset. Flags any transform whose per-row output depends
#    on other rows: mean encoders, frequency encoders, target encoders
#    applied outside CV, ComBat/global normalisation, etc.
check_stateless(my_pipeline_fn, raw_frame)

Each function raises on failure and returns None on pass. No silent degradation.

---

The demo notebook

examples/leakage_demo.ipynb — 60 seconds, California housing dataset, one deliberate leak, one library call.

Open it. It reproduces the target-encoding bug that sits in real production pipelines, shows an R² that looks impressive, then one call to check_stateless catches the leak before the model ships.

If you've ever applied .mean(), .value_counts(), TargetEncoder, or ComBat/fit_transform to your full dataset before cross-validation, the notebook is pointed at you.

---

Verified invariants under execution

The library is in production use today as a pinned dep of nyc-real-estate-predictor. The flagship's External Benchmark CI job re-checks these invariants against the published wheel on every push to main:

• Statistical leakage detection triggers on the bundled California housing demo. Build a target-mean-encoded feature on rounded lat/lon buckets — Ridge regression returns R² = 0.9495 (leaky). Apply the same target encoding per train fold only — R² collapses to 0.4384 (honest). Both check_leakage and check_stateless raise on the leaky pipeline. Reproducible in 60 seconds via examples/leakage_demo.ipynb.

• Statelessness holds under subset perturbation. check_stateless runs the user pipeline on the full frame, then on a one-row subset. Any transform whose per-row output depends on other rows (frequency encoders, target-mean encoders, ComBat-style global normalisation) fails this invariant by construction. The default spot-check deliberately targets the rows a global transform is most likely to edit — each numeric column's min/max row (winsorise/clip/quantile filters touch the tails), NaN-bearing rows (the first 10 by default — fillna(df.mean()) edits every NaN row identically, so a capped sample still catches it), and a fixed-stride spread across the rest — rather than being fooled by a plain stride sample that misses a tail- or NaN-only edit. Pass an explicit sample_indices to check more rows; checking every row is the strongest guarantee.

• Forbidden-column gate raises on the documented set. nyc-real-estate-predictor configures SchemaContract(forbidden_columns=frozenset({"SALE PRICE", "SALE DATE", "PRICE_PER_SQFT", "TARGET", "log_price"})). Verifiable from this repo: the parametrized tests/test_checks.py::test_schema_rejects_forbidden_column asserts check_schema raises on each of those names. The flagship additionally re-validates the integration in its own CI (see the production-usage note above); its internal test suite is that repo's claim, not verified here.

• Determinism check catches non-deterministic transforms. Two consecutive pipeline_fn(raw) calls must produce identical frames. Unseeded random initialisation, dict-order dependency, and side-effecting transforms all fail. Internal pd.testing.assert_frame_equal.

These hold across the test matrix; numbers (test counts, coverage %) age — the invariants don't.

---

What this is NOT

• Not a replacement for train/test splitting, cross-validation, or sklearn Pipeline.
• Not a feature-importance tool.
• Not a drift-monitoring service.
• Not a validation framework with its own DSL.

Three checks. One contract class. Four exceptions. That's the whole library.

---

Design constraints (locked)

• ≤ 500 LoC of core implementation across src/schema_firewall/ — currently 372 lines of code (515 including blanks and comments). Verify: find src/schema_firewall -name '*.py' -exec grep -vhE '^\s*(#|$)' {} + | wc -l.
• 3 public check functions — check_leakage, check_schema, check_stateless. No more.
• An adversarial test for every documented failure mode (and a regression test for each fixed bug).
• Three dependencies: numpy, pandas, scikit-learn. Nothing else.

If schema-firewall v0.1 is missing a check you need, the library is wrong for your use case. Build the check in-line. v0.1 will not grow to absorb it.

---

When to use each check

You did this	Run this
Built any feature-engineering function that reads the full frame	check_stateless(pipeline_fn, raw)
Joined multiple datasets with different origins / schemas / timestamps	check_schema(X, SchemaContract(forbidden_columns=…))
Want a fast sanity gate before training	check_leakage(X, y) on the final feature frame

---

What it caught in production (dogfood)

The schema-firewall checks are the same ones used by the NYC Real Estate Predictor external benchmark against NYC.gov 2024 Rolling Sales data. The flagship benchmark uses schema-firewall as a dependency, not a vendored copy. When the library breaks, the benchmark breaks. This is by design.

---

Attribution

Extracted from the firewall layer of the NYC Real Estate Predictor's external benchmark. The scoring-determinism pattern comes from the Protocol-based core of the Job Decision Engine project. Credit for the underlying problem classes goes to:

• DeGrave et al. (Nature Machine Intelligence, 2021) — COVID X-ray shortcut learning
• Rosenblatt et al. (Nature Communications, 2024) — connectome leakage
• Ramadan et al. (JAMIA, 2024) — clinical label-leakage framework
• YaG320 — Santander "magic" competition kernel

---

License

MIT. See LICENSE.