Generated-approach anti-cheat scans only approach.py: bypassable via siblings, and rejects legitimate private-env planning

[yixuanhuang98]

Summary

_reject_state_mutation (src/robocode/utils/episode.py:68) regex-scans the source of
approach.py for .set_state / .sample_next_state and refuses to load on a hit. It
is wrong in both directions: it misses the call when it lives in a sibling module, and
it rejects programs that plan in a private environment of their own and cannot affect
scoring.

This cost a replicate of a real sweep. Filing rather than fixing, since it is shared
infrastructure and was out of scope for #168.

It misses the real channel

load_generated_approach puts the sandbox dir on sys.path (episode.py:93) so that
approach.py can import siblings the agent wrote, but only approach.py is read
(:95-97). A program reaches the same call by moving it one file over.

Not hypothetical. In the pr2packed_generalized whitebox sweep of 2026-08-21,
replicate 24 passed the check while doing exactly this:

approach.py:18   from planner import PackedPlanner
planner.py:357   self.env.set_state(state0)
planner.py:360   self.env.set_state(state0)
planner.py:386   self.env.set_state(mid)

_reject_planner_references (:42) has the same single-file scope, so the
bilevel_models anti-cheat is bypassable the same way.

It rejects programs that do nothing wrong

Planning in a private environment the program constructs itself is the ordinary way to
write a TAMP policy, and it cannot influence scoring — but it calls set_state, so it
is refused.

In the 2026-08-24 sweep, replicate 42 was killed by this after the agent had run to
completion (362 turns, $19.78, error=False):

ValueError: Generated approach references set_state; approach.py is scored
through reset()/get_action() only and must reach the goal via the actions it
returns, not by mutating the environment's state

Its constructor never receives the environment:

approach.py:98   def __init__(self, action_space, observation_space, primitives):

and every set_state targets a clone it builds itself:

approach.py:133  env = PR2PackedEnv(num_blocks=count)

Loaded with the check bypassed, that same program solves 3 and 5 blocks in 40 and 56
steps. It was penalised for inlining its planner instead of importing it. The four
replicates that survived did so only by not using set_state at all.

The check is crude enough to reject a string literal that merely mentions the name;
tests/utils/test_episode.py::test_anti_cheat_rejects_set_state asserts precisely
that, passing extra="_ = 'env.set_state(goal)'".

Why this biases results

A sweep silently selects against approaches that keep their planner in one file. That
is not a property anyone intends to measure, and the failure is a hard error partway
through evaluation, so the replicate is lost rather than scored.

Suggested direction

The invariant is that the scored environment is not mutated, and there is exactly
one path to it from generated code: env-bound primitives close over the live env
(check_action_collision is a partial over it), so a program granted one can pull it
out of fn.args. With primitive_level=none there is no path at all, and the check
has no job.

Guarding that object rather than scanning source is exact: aliasing, getattr and
sibling modules all reach the same guarded instance, and a private environment is
unaffected. A worked implementation (read-only view bound in build_primitives,
ScoredEnvMutationError on the mutators, isinstance preserved so primitive dispatch
still works, tests) is in commit b492d5a on pr2-packed-tamp-env, reverted in 4bef58d
to keep #168 scoped. git cherry-pick b492d5a recovers it.

Evidence on disk

• multirun/pr2packed_generalized__agentic__none__whitebox__claude_opus5__timeout_60s__31d77034/2026-08-24_20-51-26/replicate_42.rejected_by_old_anticheat/
• multirun/pr2packed_generalized__agentic__none__whitebox__claude_opus5__timeout_60s__31d77034/2026-08-21_20-36-31/replicate_24/

🤖 Generated with Claude Code

https://claude.ai/code/session_01ENLdUL2iw3ut92VjdsH8ds