Skip to content

Add adversarial replay validation suite#72

Merged
ProfRandom92 merged 1 commit into
mainfrom
codex/implement-adversarial-replay-evaluation-pipeline
May 13, 2026
Merged

Add adversarial replay validation suite#72
ProfRandom92 merged 1 commit into
mainfrom
codex/implement-adversarial-replay-evaluation-pipeline

Conversation

@ProfRandom92

Copy link
Copy Markdown
Owner

Motivation

  • Stress-test replay continuity under hostile long-horizon conditions and expose evaluator weaknesses (leakage, over-anchoring, self-grading, collapse points) rather than producing suspiciously perfect scores.
  • Force separation of replay generation and independent evaluation so failure modes (hidden constraints, temporal/order drift, architecture mutation, contradictions, ambiguity, fragmentation) are detected aggressively.

Description

  • Introduce a strict adversarial evaluator StrictReplayEvaluator and compatibility wrapper evaluate_state, and decouple generator adapters from independent scoring in src/validation/replay_continuity.py.
  • Add seven hostile scenarios (hidden constraint traps, temporal order confusion, architecture mutation, contradictory instructions, dependency inversion, semantic ambiguity, context fragmentation) and expose hidden truth/temporal channels in build_adversarial_scenarios.
  • Expand the V7ReplayAdapter and comparative adapters to model honest long-horizon degradation and failure probes, and add many new continuity metrics (collapse iteration, continuity half-life, drift acceleration, contradiction growth/rate, hidden constraint survival, temporal/ordered scoring, architecture mutation resistance, semantic ambiguity resilience, adversarial evaluator score, embedding divergence, etc.).
  • Generate new adversarial visual artifacts (replay collapse curves, drift acceleration graphs, contradiction accumulation heatmaps, constraint survival curves, replay longevity comparisons, failure point timelines, semantic stability heatmaps) and update the benchmark runner default to 100 iterations (benchmarks/run_replay_continuity.py).
  • Update tests in tests/test_replay_continuity.py to assert realistic degradation visibility, evaluator independence, multi-iteration ladders (10/25/50/100) and the presence of the new adversarial visualizations.

Testing

  • Ran pytest -q and the modified test suite passed (25 passed).
  • Compiled the updated modules with python -m py_compile src/validation/replay_continuity.py benchmarks/run_replay_continuity.py which succeeded.
  • Executed the benchmark runner python benchmarks/run_replay_continuity.py --iterations 100 --output-dir reports/replay_continuity which produced deterministic artifacts under reports/replay_continuity including comparison_summary.json (sample summary shows comptext_v7 mean_final_continuity ~ 0.789024) and the SVG visualizations listed in the summary.

Codex Task

@vercel

vercel Bot commented May 13, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
comptextv7 Ready Ready Preview, Comment May 13, 2026 5:16pm

@netlify

netlify Bot commented May 13, 2026

Copy link
Copy Markdown

Deploy Preview for comptext-v7 canceled.

Name Link
🔨 Latest commit b2c04d3
🔍 Latest deploy log https://app.netlify.com/projects/comptext-v7/deploys/6a04b1cd34157d00082d7d7e

@ProfRandom92 ProfRandom92 merged commit 4a08690 into main May 13, 2026
10 checks passed

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly enhances the adversarial replay continuity evaluation suite for CompText V7. It introduces a new StrictReplayEvaluator to independently score replay states against hidden truth sets, adds robust failure flags for tracking continuity loss, and expands the benchmark to include 100-iteration stress tests. Additionally, the visualization layer has been updated to provide more detailed insights into drift acceleration, constraint survival, and failure timelines. I have provided feedback on improving the robustness of the metric aggregation, optimizing file I/O for artifact generation, and removing brittle hardcoded limits in the SVG rendering logic.

@@ -706,44 +861,65 @@ def _heatmap_svg(chains: list[object]) -> str:
value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This line will raise a ZeroDivisionError if the input chains does not contain any entries for a specific mode in the modes tuple. While the current benchmark runner ensures all modes are present, this helper function should be more robust to handle arbitrary chain sets.

Suggested change
value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index]
finals = [chain for chain in chains if isinstance(chain, dict) and chain["mode"] == mode]
if not finals:
continue
value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index]

Comment on lines +797 to +808
paths["replay_collapse_curves"].write_text(_line_svg(chains, "overall_continuity", "Replay collapse curves"), encoding="utf-8")
paths["drift_acceleration_graph"].write_text(_line_svg(chains, "semantic_drift_growth", "Drift acceleration graph"), encoding="utf-8")
paths["contradiction_accumulation_heatmap"].write_text(_heatmap_svg(chains, ("contradiction_accumulation", "contradiction_growth_rate", "embedding_divergence"), "Contradiction accumulation heatmap"), encoding="utf-8")
paths["constraint_survival_curves"].write_text(_line_svg(chains, "hidden_constraint_survival", "Constraint survival curves"), encoding="utf-8")
paths["replay_longevity_comparisons"].write_text(_longevity_svg(comparison["summary"]), encoding="utf-8") # type: ignore[index]
paths["failure_point_timelines"].write_text(_failure_timeline_svg(chains), encoding="utf-8")
paths["semantic_stability_heatmaps"].write_text(_heatmap_svg(chains, ("semantic_ambiguity_resilience", "temporal_consistency_score", "architecture_mutation_resistance", "hidden_truth_verification", "overall_continuity"), "Semantic stability heatmap"), encoding="utf-8")
paths["replay_degradation_curves"].write_text(paths["replay_collapse_curves"].read_text(encoding="utf-8"), encoding="utf-8")
paths["continuity_heatmap"].write_text(paths["semantic_stability_heatmaps"].read_text(encoding="utf-8"), encoding="utf-8")
paths["semantic_drift_graph"].write_text(paths["drift_acceleration_graph"].read_text(encoding="utf-8"), encoding="utf-8")
paths["replay_longevity_chart"].write_text(paths["replay_longevity_comparisons"].read_text(encoding="utf-8"), encoding="utf-8")
paths["contradiction_accumulation_graph"].write_text(paths["contradiction_accumulation_heatmap"].read_text(encoding="utf-8"), encoding="utf-8")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current implementation performs redundant disk I/O by writing files and then immediately reading them back to create aliases. It is more efficient to store the SVG content in variables and write them to multiple paths directly.

    collapse_svg = _line_svg(chains, "overall_continuity", "Replay collapse curves")
    paths["replay_collapse_curves"].write_text(collapse_svg, encoding="utf-8")
    paths["replay_degradation_curves"].write_text(collapse_svg, encoding="utf-8")

    drift_svg = _line_svg(chains, "semantic_drift_growth", "Drift acceleration graph")
    paths["drift_acceleration_graph"].write_text(drift_svg, encoding="utf-8")
    paths["semantic_drift_graph"].write_text(drift_svg, encoding="utf-8")

    stability_heatmap = _heatmap_svg(chains, ("semantic_ambiguity_resilience", "temporal_consistency_score", "architecture_mutation_resistance", "hidden_truth_verification", "overall_continuity"), "Semantic stability heatmap")
    paths["semantic_stability_heatmaps"].write_text(stability_heatmap, encoding="utf-8")
    paths["continuity_heatmap"].write_text(stability_heatmap, encoding="utf-8")

    longevity_svg = _longevity_svg(comparison["summary"])
    paths["replay_longevity_comparisons"].write_text(longevity_svg, encoding="utf-8")
    paths["replay_longevity_chart"].write_text(longevity_svg, encoding="utf-8")

    contra_heatmap = _heatmap_svg(chains, ("contradiction_accumulation", "contradiction_growth_rate", "embedding_divergence"), "Contradiction accumulation heatmap")
    paths["contradiction_accumulation_heatmap"].write_text(contra_heatmap, encoding="utf-8")
    paths["contradiction_accumulation_graph"].write_text(contra_heatmap, encoding="utf-8")

    paths["constraint_survival_curves"].write_text(_line_svg(chains, "hidden_constraint_survival", "Constraint survival curves"), encoding="utf-8")
    paths["failure_point_timelines"].write_text(_failure_timeline_svg(chains), encoding="utf-8")

f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" viewBox="0 0 {width} {height}">',
'<rect width="100%" height="100%" fill="#0f172a" />',
'<text x="18" y="28" fill="#e2e8f0" font-size="22" font-family="sans-serif">Failure point timelines</text>',
*rows[:120],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The hardcoded slice [:120] is brittle. Since each chain appends 3 elements to rows, this limit effectively caps the visualization at 40 chains. Furthermore, the SVG height (420px) only fits approximately 30 chains before elements overflow the viewBox. It is better to remove the arbitrary slice and instead break the loop if the vertical space is exhausted.

Suggested change
*rows[:120],
*rows,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant