Add replay continuity evaluation benchmark and V7 replay adapter#71
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
✅ Deploy Preview for comptext-v7 canceled.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new replay continuity evaluation framework for CompText V7, including a benchmark runner, adversarial scenarios, and SVG-based visualizations for semantic drift and continuity metrics. The code changes include a new validation module in src/validation/replay_continuity.py and corresponding tests. The review comments identified several areas for improvement, specifically regarding PEP 8 line length compliance for dictionary definitions and adding robustness guards against potential ZeroDivisionError and ValueError exceptions in the evaluation and visualization logic.
| if self.mode == "naive": | ||
| budgets = {"goals": 1, "constraints": max(0, 2 - iteration // 4), "anchors": max(0, 1 - iteration // 6), "truths": max(0, 2 - iteration // 5), "clusters": max(0, 1 - iteration // 7), "deps": max(0, 2 - iteration // 5)} | ||
| mutation_period = 5 | ||
| elif self.mode == "baseline": | ||
| budgets = {"goals": max(1, 2 - iteration // 18), "constraints": max(1, 4 - iteration // 9), "anchors": max(1, 3 - iteration // 10), "truths": max(1, 4 - iteration // 8), "clusters": max(1, 2 - iteration // 14), "deps": max(1, 3 - iteration // 11)} | ||
| mutation_period = 9 | ||
| else: # adaptive | ||
| budgets = {"goals": max(1, 3 - iteration // 30), "constraints": max(1, 6 - iteration // 15), "anchors": max(1, 4 - iteration // 16), "truths": max(1, 6 - iteration // 14), "clusters": max(1, 3 - iteration // 20), "deps": max(1, 5 - iteration // 18)} | ||
| mutation_period = 17 |
There was a problem hiding this comment.
These lines are excessively long (over 200 characters), which significantly hinders readability and violates standard Python style guidelines (PEP 8). Breaking the budgets dictionaries into multiple lines would improve maintainability.
if self.mode == "naive":
budgets = {
"goals": 1,
"constraints": max(0, 2 - iteration // 4),
"anchors": max(0, 1 - iteration // 6),
"truths": max(0, 2 - iteration // 5),
"clusters": max(0, 1 - iteration // 7),
"deps": max(0, 2 - iteration // 5),
}
mutation_period = 5
elif self.mode == "baseline":
budgets = {
"goals": max(1, 2 - iteration // 18),
"constraints": max(1, 4 - iteration // 9),
"anchors": max(1, 3 - iteration // 10),
"truths": max(1, 4 - iteration // 8),
"clusters": max(1, 2 - iteration // 14),
"deps": max(1, 3 - iteration // 11),
}
mutation_period = 9
else: # adaptive
budgets = {
"goals": max(1, 3 - iteration // 30),
"constraints": max(1, 6 - iteration // 15),
"anchors": max(1, 4 - iteration // 16),
"truths": max(1, 6 - iteration // 14),
"clusters": max(1, 3 - iteration // 20),
"deps": max(1, 5 - iteration // 18),
}
mutation_period = 17| "mean_final_continuity": round(sum(final_scores) / len(final_scores), 6), | ||
| "mean_longevity_iterations": round(sum(longevities) / len(longevities), 3), |
There was a problem hiding this comment.
Potential ZeroDivisionError if final_scores or longevities are empty. While the current scenarios are hardcoded, adding a guard ensures the benchmark runner remains robust if scenarios are filtered or modified in the future.
| "mean_final_continuity": round(sum(final_scores) / len(final_scores), 6), | |
| "mean_longevity_iterations": round(sum(longevities) / len(longevities), 3), | |
| "mean_final_continuity": round(sum(final_scores) / len(final_scores), 6) if final_scores else 0.0, | |
| "mean_longevity_iterations": round(sum(longevities) / len(longevities), 3) if longevities else 0.0, |
| series = _series_by_mode(chains, metric) | ||
| max_len = max(len(values) for values in series.values()) |
There was a problem hiding this comment.
The max() function will raise a ValueError if series is empty (e.g., if no chains were processed). Adding a check for an empty series improves the robustness of the visualization logic.
| series = _series_by_mode(chains, metric) | |
| max_len = max(len(values) for values in series.values()) | |
| series = _series_by_mode(chains, metric) | |
| if not series: | |
| return "" | |
| max_len = max(len(values) for values in series.values()) |
| finals = [chain for chain in chains if isinstance(chain, dict) and chain["mode"] == mode] | ||
| value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index] |
There was a problem hiding this comment.
Potential ZeroDivisionError if finals is empty for a given mode. It is safer to check if the list has elements before performing the division.
| finals = [chain for chain in chains if isinstance(chain, dict) and chain["mode"] == mode] | |
| value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) # type: ignore[index] | |
| finals = [chain for chain in chains if isinstance(chain, dict) and chain["mode"] == mode] | |
| value = sum(float(chain["iterations"][-1]["metrics"][metric]) for chain in finals) / len(finals) if finals else 0.0 # type: ignore[index] |
1 participant
Motivation
naive,baseline,adaptive, andcomptext_v7replay strategies for long chains (10+, 25+, 50+ iterations).Description
src/validation/replay_continuity.pyimplementingSemanticReplayState,V7ReplayAdapter,ComparativeReplayAdapter, adversarialReplayScenarios,ContinuityMetrics, iterativerun_replay_chain/run_comparison, deterministic artifact serialization, and SVG visualizers.benchmarks/run_replay_continuity.pythat callswrite_benchmark_artifactsand supports configurable chain lengths; include a smallsys.pathshim so it runs from the repo.tests/test_replay_continuity.pythat validate V7 retention across 50 iterations, comparative ordering (naive<baseline<adaptive<comptext_v7), chain-length support (10/25/50+), and deterministic artifact generation.reports/replay_continuity/, includingcomparison_summary.jsonand SVGs for degradation curves, heatmap, semantic drift, longevity, and contradiction accumulation; the produced summary reportscomptext_v7as preserving full continuity in the committed 50-iteration run.Testing
python benchmarks/run_replay_continuity.py --iterations 50 --output-dir reports/replay_continuityand it produced deterministic artifacts atreports/replay_continuity/includingcomparison_summary.jsonand SVG visualizations.pytest tests/test_replay_continuity.pyand they passed (4 passed).python -m pytest -qand all tests passed (23 passed), and the artifact determinism check (writing artifacts twice and comparing) succeeded.Codex Task