Add adversarial replay validation suite#72
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
✅ Deploy Preview for comptext-v7 canceled.
|
There was a problem hiding this comment.
Code Review
This pull request significantly enhances the adversarial replay continuity evaluation suite for CompText V7. It introduces a new StrictReplayEvaluator to independently score replay states against hidden truth sets, adds robust failure flags for tracking continuity loss, and expands the benchmark to include 100-iteration stress tests. Additionally, the visualization layer has been updated to provide more detailed insights into drift acceleration, constraint survival, and failure timelines. I have provided feedback on improving the robustness of the metric aggregation, optimizing file I/O for artifact generation, and removing brittle hardcoded limits in the SVG rendering logic.
| @@ -706,44 +861,65 @@ def _heatmap_svg(chains: list[object]) -> str: | |||
| value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index] | |||
There was a problem hiding this comment.
This line will raise a ZeroDivisionError if the input chains does not contain any entries for a specific mode in the modes tuple. While the current benchmark runner ensures all modes are present, this helper function should be more robust to handle arbitrary chain sets.
| value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index] | |
| finals = [chain for chain in chains if isinstance(chain, dict) and chain["mode"] == mode] | |
| if not finals: | |
| continue | |
| value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index] |
| paths["replay_collapse_curves"].write_text(_line_svg(chains, "overall_continuity", "Replay collapse curves"), encoding="utf-8") | ||
| paths["drift_acceleration_graph"].write_text(_line_svg(chains, "semantic_drift_growth", "Drift acceleration graph"), encoding="utf-8") | ||
| paths["contradiction_accumulation_heatmap"].write_text(_heatmap_svg(chains, ("contradiction_accumulation", "contradiction_growth_rate", "embedding_divergence"), "Contradiction accumulation heatmap"), encoding="utf-8") | ||
| paths["constraint_survival_curves"].write_text(_line_svg(chains, "hidden_constraint_survival", "Constraint survival curves"), encoding="utf-8") | ||
| paths["replay_longevity_comparisons"].write_text(_longevity_svg(comparison["summary"]), encoding="utf-8") # type: ignore[index] | ||
| paths["failure_point_timelines"].write_text(_failure_timeline_svg(chains), encoding="utf-8") | ||
| paths["semantic_stability_heatmaps"].write_text(_heatmap_svg(chains, ("semantic_ambiguity_resilience", "temporal_consistency_score", "architecture_mutation_resistance", "hidden_truth_verification", "overall_continuity"), "Semantic stability heatmap"), encoding="utf-8") | ||
| paths["replay_degradation_curves"].write_text(paths["replay_collapse_curves"].read_text(encoding="utf-8"), encoding="utf-8") | ||
| paths["continuity_heatmap"].write_text(paths["semantic_stability_heatmaps"].read_text(encoding="utf-8"), encoding="utf-8") | ||
| paths["semantic_drift_graph"].write_text(paths["drift_acceleration_graph"].read_text(encoding="utf-8"), encoding="utf-8") | ||
| paths["replay_longevity_chart"].write_text(paths["replay_longevity_comparisons"].read_text(encoding="utf-8"), encoding="utf-8") | ||
| paths["contradiction_accumulation_graph"].write_text(paths["contradiction_accumulation_heatmap"].read_text(encoding="utf-8"), encoding="utf-8") |
There was a problem hiding this comment.
The current implementation performs redundant disk I/O by writing files and then immediately reading them back to create aliases. It is more efficient to store the SVG content in variables and write them to multiple paths directly.
collapse_svg = _line_svg(chains, "overall_continuity", "Replay collapse curves")
paths["replay_collapse_curves"].write_text(collapse_svg, encoding="utf-8")
paths["replay_degradation_curves"].write_text(collapse_svg, encoding="utf-8")
drift_svg = _line_svg(chains, "semantic_drift_growth", "Drift acceleration graph")
paths["drift_acceleration_graph"].write_text(drift_svg, encoding="utf-8")
paths["semantic_drift_graph"].write_text(drift_svg, encoding="utf-8")
stability_heatmap = _heatmap_svg(chains, ("semantic_ambiguity_resilience", "temporal_consistency_score", "architecture_mutation_resistance", "hidden_truth_verification", "overall_continuity"), "Semantic stability heatmap")
paths["semantic_stability_heatmaps"].write_text(stability_heatmap, encoding="utf-8")
paths["continuity_heatmap"].write_text(stability_heatmap, encoding="utf-8")
longevity_svg = _longevity_svg(comparison["summary"])
paths["replay_longevity_comparisons"].write_text(longevity_svg, encoding="utf-8")
paths["replay_longevity_chart"].write_text(longevity_svg, encoding="utf-8")
contra_heatmap = _heatmap_svg(chains, ("contradiction_accumulation", "contradiction_growth_rate", "embedding_divergence"), "Contradiction accumulation heatmap")
paths["contradiction_accumulation_heatmap"].write_text(contra_heatmap, encoding="utf-8")
paths["contradiction_accumulation_graph"].write_text(contra_heatmap, encoding="utf-8")
paths["constraint_survival_curves"].write_text(_line_svg(chains, "hidden_constraint_survival", "Constraint survival curves"), encoding="utf-8")
paths["failure_point_timelines"].write_text(_failure_timeline_svg(chains), encoding="utf-8")| f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" viewBox="0 0 {width} {height}">', | ||
| '<rect width="100%" height="100%" fill="#0f172a" />', | ||
| '<text x="18" y="28" fill="#e2e8f0" font-size="22" font-family="sans-serif">Failure point timelines</text>', | ||
| *rows[:120], |
There was a problem hiding this comment.
The hardcoded slice [:120] is brittle. Since each chain appends 3 elements to rows, this limit effectively caps the visualization at 40 chains. Furthermore, the SVG height (420px) only fits approximately 30 chains before elements overflow the viewBox. It is better to remove the arbitrary slice and instead break the loop if the vertical space is exhausted.
| *rows[:120], | |
| *rows, |
Motivation
Description
StrictReplayEvaluatorand compatibility wrapperevaluate_state, and decouple generator adapters from independent scoring insrc/validation/replay_continuity.py.build_adversarial_scenarios.V7ReplayAdapterand comparative adapters to model honest long-horizon degradation and failure probes, and add many new continuity metrics (collapse iteration, continuity half-life, drift acceleration, contradiction growth/rate, hidden constraint survival, temporal/ordered scoring, architecture mutation resistance, semantic ambiguity resilience, adversarial evaluator score, embedding divergence, etc.).benchmarks/run_replay_continuity.py).tests/test_replay_continuity.pyto assert realistic degradation visibility, evaluator independence, multi-iteration ladders (10/25/50/100) and the presence of the new adversarial visualizations.Testing
pytest -qand the modified test suite passed (25 passed).python -m py_compile src/validation/replay_continuity.py benchmarks/run_replay_continuity.pywhich succeeded.python benchmarks/run_replay_continuity.py --iterations 100 --output-dir reports/replay_continuitywhich produced deterministic artifacts underreports/replay_continuityincludingcomparison_summary.json(sample summary showscomptext_v7mean_final_continuity ~0.789024) and the SVG visualizations listed in the summary.Codex Task