Progress-stall watchdog + Kimi context-window ablation run#36
Closed
seanrivera wants to merge 26 commits into
Closed
Progress-stall watchdog + Kimi context-window ablation run#36seanrivera wants to merge 26 commits into
seanrivera wants to merge 26 commits into
Conversation
Brings in the combined context_window condition that emits the activity summary before the last3 image blocks. Conflicts resolved to keep this branch's fair-default config (image_text + description + last3) and the stateless-consistent condition-set variants, while adding PR #33's new `text_summary_and_last3` variant, `leading_summary_blocks()`, the runner wiring, and the image_only last3 `FINAL_OUTPUT:` label. prompts.txt regenerated against the merged defaults; new PR tests retained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a self-contained Kimi-only run family for the final context-window comparison: last3 vs text_summary vs text_summary_and_last3 over the 15-maze conditional set, thinking OFF, on a 3-Kimi-worker fleet (+coordinator). Gated behind SWEEP_TOPO=kimictx so the existing 10-config sweep (BATCHES) is untouched. - run_config.smoke_kimi3.json / run_config.conditional_context_window_kimi.json: Kimi-only (no Claude/Qwen), worker_count 3, temp 0.6, thinking OFF. - sweep_state: _KIMI_CTX topo selects a 3-batch family (last3/text_summary/ text_summary_and_last3); default family preserved via _FULL_BATCHES. - sweep_run.sh: kimictx provision config = smoke_kimi3. - Fix d3 data blocker: the ogbench D3 maze had a stale maze.goal=(8,1) (behind the dead-end door) vs the authoritative goal.target=(8,8), which the single-goal validator rejects. Vendored the corrected maze into mazes/conditional/ (git-archive excludes submodules, so this keeps the fix reachable on the coordinator) and repointed the manifest; ogbench submodule left pristine. - Tests for the topo family and the Kimi-only config resolution. Combined-job prep yields 45 clean units (15 per variant). 771 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
max_in_flight is a PER-GROUP concurrency cap (_below_max_in_flight counts active units across the whole model_group), carried over unchanged from the single-Kimi-worker sweep config. At 1 it serialized the 3-worker fleet: the kimictx smoke ran one unit at a time while 2 of 3 workers sat idle (~40 min for 3 episodes). Set it to worker_count=3 so all three workers run in parallel. Smoke otherwise passed end-to-end (3/3 episodes success). Test now asserts max_in_flight == worker_count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The provisioned fleet's coordinator has an uninitialized ogbench submodule (git submodule status: -cc7426...), and code-sync uses `git archive`, which excludes submodules — so the three ogbench-sourced conditional mazes (S5, M1, D1) were absent on the coordinator and prepare_job failed with FileNotFound. Vendored them into mazes/conditional/ (same fix as d3, precedent: the B-probe) and repointed the manifest, so all 15 conditional_eval sources now live in the git-archive'd main-repo tree. ogbench submodule untouched; goal-consistent and static-scorable; disjointness from the 50-run preserved (vendored paths aren't ogbench paths). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…add terminated_failure
Adds two runner-level tests for the branches introduced in 201a3fb: a switch-goal spec that terminates from TOGGLE (event_type "TOGGLED", not DONE) to cover the OR-success path, and a lava-hazard spec that terminates without reaching the goal (event_type "MOVED") to cover end_reason="terminated_failure". Both verified against the real MiniGridBackend before being written into tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ay < condition variant)
…workers use it verbatim Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…error Regenerate the plan with coordinator-prepare, don't reconstruct it worker-side.
…ct on survive_steps
…er-counting omissions Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…et-v2.0 into feature/early_terminate # Conflicts: # interface/config.py # interface/runner.py # prompting_experiments/condition_set_3_context_window.py # prompting_experiments/prompts.txt # tests/test_analysis_episodes.py # tests/test_prompt_observation_text.py
…ext_summary_and_last3 - observation.py: `_history_record_action` shows `cardinal_action` (the model's emission) for cardinal runs instead of the expanded primitive, since history renders under `FINAL_OUTPUT:` and must show an action the model can output; egocentric runs (cardinal_action=None) fall back to `action`, unchanged. - preview_prompts.py: extend the one-shot text_summary preview to the text_summary_and_last3 variant too. - Regenerate prompts.txt. (The repeated no-op-PICKUP history lines are a pre-existing preview-rollout artifact, unchanged by this commit.) - Tests: +history-action coverage in test_history_dedup / test_prompt_observation_text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…endored copies The ogbench submodule D3 goal mismatch is now fixed upstream (maze.goal := goal.target=(8,8); ManifoldRG/ogbench @ 319b902), so the four conditional mazes (S5/M1/D1/D3) no longer need vendoring into mazes/conditional/. Repoint the manifest at the ogbench sources, bump the submodule pointer, and delete the vendored copies. The B blind-probe stays in mazes/conditional/ (it has no ogbench source). All 15 sources validate; 844 tests pass. Note: git-archive excludes submodules, so distributed runs must sync the submodule separately (recurse-submodules or on-VM init) — tracked follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hem) The on-VM code sync used `git archive`, which excludes submodule trees, and the coordinator VM image leaves ogbench uninitialized (its remote needs creds the VM lacks) — so ogbench-sourced conditional_eval mazes were missing at prepare time (FileNotFound), the exact gap the now-reverted maze vendoring worked around. sync_code_to_vm now enumerates submodules (git ls-tree commit entries) and ships each one's tracked tree at the commit the superproject pins at $sha. verify_code_on_vm fail-closes if any submodule tree didn't land on the VM. Tests cover both the ship+verify path and the missing-submodule abort. 846 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oal fix)
R1 preflight artifacts for the next major benchmark run:
- manifest.r1_{balanced_03,long_tail_02,mechanism_rich_01,pairwise_01}.json —
4 candidate 50-maze panels, ordered ascending by BFS optimal_steps, all sources
resolve via the pipeline loader; expected_mechanisms = on-path door/gate ids.
- run_config.r1.json — single fixed cell (no conditions): minimal / image_only /
egocentric / zero_shot / text_summary_and_last3 (chat_history=stateless) /
progress_stall_k=30. Uniform max_tokens=16384 across all 3 models (Claude
effort=high + adaptive thinking sets the ceiling; Kimi/Qwen thinking off).
Qwen max_model_len raised to 32768 to fit prompt+16384 generation.
- ogbench bumped 319b902->31a0549: S4/10x10_dense_1 maze.goal aligned to
goal.target (last corpus goal-invariant defect).
Every maze in all 4 panels re-validated (goal invariant + BFS beatability +
optimal_steps == packaged claim). Model token cap 16384 pending pre-launch
smoke confirmation (not an Anthropic-published high-effort figure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Ej5sVfJmQZUGij3PmwsdJ
…p->0.6) Thinking ON for Claude/Kimi/Qwen at the uniform 16384 cap — makes the cross-model comparison symmetric (all reason). Kimi temperature set to 1.0 (Moonshot mode-forces it when thinking on; agent pins it regardless). Qwen temperature set to 0.6 (greedy temp=0.0 risks repetition/degeneration in Qwen thinking mode). Pre-launch smoke should confirm neither Opus (high effort) nor Kimi (large-budget) thinking truncates before FINAL_OUTPUT at 16384. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Ej5sVfJmQZUGij3PmwsdJ
pranavguru
requested changes
Jul 16, 2026
| FINAL_OUTPUT: TURN_LEFT | ||
|
|
||
| Recent history (last 3 steps, oldest first): | ||
| Position after: (1, 1), facing EAST |
Member
There was a problem hiding this comment.
This file was regenerated with a multi-line recent history step format, but the corresponding template update in prompting_experiments/prompt_templates/observation.py was omitted from the commit. Result: this file + the new test_prompt_observation_text.py assertions all expect multi-line output, but the actual template still emits " ({row}, {col}) facing {facing} -> {action} -> {feedback}" (single-line). 4 tests fail as-is.
…k/xhigh thinking - All 4 R1 manifests regenerated from updated candidate sets: the 9/11-step S1 empty_room anchors were dropped corpus-wide (all panels now 23..106 steps); balanced_03 specifically swaps the 2 empty_rooms for S5/14x14_corridor_0 (77) and _1 (89). Every maze re-validated (goal invariant + BFS + optimal_steps). - run_config.r1.json: uniform max_tokens=64000 (Anthropic xhigh starting budget), thinking ON all 3, Claude effort=xhigh, Qwen max_model_len=96000, timeouts 600/900s for deep-thinking generations. Budget (1 seed, Claude+Kimi API, Qwen local): ~$435 central, $200-1025 range, driven by unknown image_only+thinking queries/episode. A 2-3 maze micro-smoke would collapse the range. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Ej5sVfJmQZUGij3PmwsdJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch bundles three related pieces of work: (1) the progress-stall early-termination watchdog (the headline feature), (2) the Kimi-only context-window ablation run infrastructure and its data-plumbing fixes, and (3) the merged
text_summary_and_last3context mode (PR #33).All work is on
feature/early_terminate; 813 tests pass at the pre-merge HEAD, and the watchdog feature carried a per-task + final whole-branch review (0 Critical/Important findings).1. Progress-stall early-termination watchdog (new feature)
Implements proposal 3 of
docs/design-proposals-2026-07.md. Ends an episode withend_reason="stalled"after K consecutive executed steps that revisit an already-seen runtime-state signature — recovering the wasted step/token cost of agents that grind to the step cap without progress.interface/runner._progress_signature):(agent_position, agent_carrying, collected_keys, open_doors, active_switches, open_gates, block_positions, explored_cells-if-partial-obs). Facing is excluded (spin ≠ progress); mechanism/key/door/switch/gate/block changes make every cell novel again for one traversal (the free backtrack budget keyed mazes need);explored_cellscounts only under partial observation.ExperimentConfig.progress_stall_k: int | None = None(OFF by default → byte-identical forreach_positiontasks). Validated (rejects bool/float/zero/negative). Enabled per-run via a new top-levelexperiment_configrun-config overlay (defaults < overlay < condition variant), which is serialized into distributed units so workers use the exact resolved config.goal_reached, not onlyevent_type == "DONE"— socollect_all/push_block_to/ switch-completion goals that terminate from a non-DONE action are correctly scored; non-goal terminations get a newterminated_failureend_reason. Backend terminals always take precedence over the watchdog on the same step.stalledandterminated_failureare failures (zero success credit);episode_metricsmaps them to the rightterminated/truncatedbooleans; the analysis loader prefers the aggregateend_reason.analysis/stall_replay.pyreplays candidate K over a frozen corpus (win/failure/step-savings counts) as a decision aid — no single K is baked in; the run owner picks (spec documents the tradeoff).survive_stepsgoals reject the watchdog (repeated states are the intended behavior there).2. Kimi context-window ablation run (infrastructure + data fixes)
SWEEP_TOPO=kimictx: a gated, Kimi-only 3-worker run family (scripts/sweep_state.py) comparinglast3vstext_summaryvstext_summary_and_last3over the 15-maze conditional set, thinking-off. The default 10-config sweep is untouched.max_in_flightfix: it is a per-group concurrency cap; carried over as1it serialized the 3-worker fleet. Set toworker_count(3) so all workers run concurrently. Test assertsmax_in_flight == worker_count.git archive(the on-VM code sync) excludes submodules, and the coordinator VM's ogbench submodule is uninitialized — so the 4 ogbench-sourced conditional mazes (S5, M1, D1, d3) were absent on the coordinator. Vendored them intomazes/conditional/(git-archive-reachable) and repointed the manifest; the ogbench submodule is left pristine. Also fixes the d3 goal mismatch (maze.goalcorrected to the authoritativegoal.target = (8,8)).3.
text_summary_and_last3context mode (PR #33)The combined context condition (activity summary + last-3 image blocks), merged in and reconciled with this branch's fair-default config and movement-trail summary logic.
Test / review
a740f67).🤖 Generated with Claude Code