m12-pr12: provider-failover guarantee + TD-2 flip#30
Open
BalaShankar9 wants to merge 1 commit into
Open
Conversation
Closes the Must-Never-Happen scenario "Single Gemini outage takes down platform" with an explicit scenario test, and flips the already-implemented TD-2 row. backend/tests/ai/test_provider_failover.py (15 tests, all green) ================================================================ The model_router cascade contract was already pinned at the unit level (test_model_router.py), but the platform-level guarantee -- "if every Gemini model is unhealthy AND ff_anthropic_provider is ON, the cascade returns claude-* so traffic still serves" -- had no explicit scenario test. A unit refactor that broke the healthy-filter / Anthropic-strip interaction would not have tripped any guard. Coverage: - Anthropic OFF (ship state) for every Tier-1 task: cascade has zero claude-* entries (ADR-0031 / m7-pr28 contract). - Anthropic ON: claude-* is present in every Tier-1 cascade. - Anthropic ON + entire Gemini family (Pro, 2.5 Flash, 2.0 Flash) unhealthy: cascade returns claude-* and is non-empty -- the Must-Never-Happen guarantee. - Partial outage (Pro down): Flash comes before claude (cost + data-residency preference for same-provider failover). - Anthropic OFF + full Gemini outage: cascade still non-empty (last-resort "return all" path), Gemini-only. - Recovery probe brings Gemini back to head of cascade after RECOVERY_TIMEOUT. - get_model_health() exposes unhealthy state for ops alerting. TD-2 flip ========= MAX_SCRUB_DEPTH was already bumped from 8 -> 16 in backend/app/core/observability.py (covered by test_observability_redaction.py::test_redact_scrubs_at_depth_15). The blueprint row was stale; flipped to SHIPPED with file + test references. Blueprint ========= - TD-2: Trigger -> SHIPPED. - Section 21 row "Single Gemini outage" Tested-by: future tests/ai/test_provider_failover.py -> backend/tests/ai/test_provider_failover.py (m12-pr12). Stack ===== #27 (m12-pr09) -> #28 (m12-pr10) -> #29 (m12-pr11) -> THIS Files: 2 changed, +186 / -2.
There was a problem hiding this comment.
Pull request overview
Adds a scenario-level test to guarantee multi-provider failover continues serving traffic during a full Gemini-family outage (when Anthropic is enabled), and updates the World Class Architecture Blueprint to reflect shipped TD-2 redaction depth and the new §21 guarantee test.
Changes:
- Add
backend/tests/ai/test_provider_failover.pyscenario tests covering Anthropic flag gating, full/partial Gemini outage behavior, recovery probing, and health surface visibility. - Update blueprint rows to mark TD-2 as shipped and to reference the new provider-failover scenario test for the “Single Gemini outage” must-never-happen guarantee.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| docs/architecture/WORLD_CLASS_ARCHITECTURE_BLUEPRINT.md | Marks TD-2 as shipped and links §21 “Single Gemini outage” scenario to the new test file. |
| backend/tests/ai/test_provider_failover.py | Introduces scenario tests that pin provider failover behavior across flag states and outage/recovery conditions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| |---|---|---|---| | ||
| | TD-1 | `generate/jobs.py` is 1500+ lines | Merge conflicts at scale | Next major change in that file | | ||
| | TD-2 | Sentry redaction depth bound = 8 | Deep AI objects can leak at depth 9+ | Increase to 16 or convert to iterative | | ||
| | TD-2 | Sentry redaction depth bound = 8 | Deep AI objects can leak at depth 9+ | SHIPPED — `MAX_SCRUB_DEPTH = 16` in `backend/app/core/observability.py`; covered by `tests/test_observability_redaction.py::test_redact_scrubs_at_depth_15` | |
Comment on lines
+57
to
+64
| def _enable_anthropic(monkeypatch) -> None: | ||
| """Force ``ff_anthropic_provider`` ON via settings + env fallback.""" | ||
| monkeypatch.setenv("FF_ANTHROPIC_PROVIDER", "1") | ||
| try: | ||
| from app.core.config import settings # type: ignore | ||
| monkeypatch.setattr(settings, "ff_anthropic_provider", True, raising=False) | ||
| except Exception: | ||
| pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
§21 "Must Never Happen" guarantee + TD-2 blueprint flip.
Closes the "Single Gemini outage takes down platform" scenario with an explicit scenario test, and flips the already-implemented TD-2 (
MAX_SCRUB_DEPTH = 16) blueprint row.Why a scenario test, not just unit tests
The model_router cascade contract was already pinned at the unit level (
backend/tests/unit/test_model_router.py) — tier-1 ordering, healthy-filter, env overrides, recovery timeout, etc. But the platform-level guarantee — "if every Gemini model is unhealthy ANDff_anthropic_provideris ON, the cascade returnsclaude-*so traffic still serves" — had no explicit test. A unit refactor that broke the healthy-filter / Anthropic-strip interaction would not have tripped any guard.This PR adds the scenario test that holds the line.
backend/tests/ai/test_provider_failover.py— 15 tests, 15 greenclaude-*entries (ADR-0031 / m7-pr28 contract).claude-*is present in every Tier-1 cascade.claude-*and is non-empty. The Must-Never-Happen guarantee.RECOVERY_TIMEOUT.get_model_health()exposes unhealthy state for ops alerting.TD-2 flip
MAX_SCRUB_DEPTHwas already bumped from8 → 16inbackend/app/core/observability.py(covered bytests/test_observability_redaction.py::test_redact_scrubs_at_depth_15). The blueprint row was stale — flipped to SHIPPED with file + test references.Blueprint
future tests/ai/test_provider_failover.py→backend/tests/ai/test_provider_failover.py(m12-pr12).Stack
Files: 2 changed, +192 / -2.