Repository navigation
Conversation
added 4 commits
October 15, 2026 14:47
gaia2-eventd logs one line per scenario event at startup (id, type AGENT, deps/succs) plus "Oracle data: N turns, M total oracle events", and the judge appends every judged turn's oracle events (tool, args) to judgments.jsonl. All three destinations were readable by the agent user: - /tmp/gaia2-eventd.log (eventd's console log) is created by touch as root with no chmod since a9e6435, so 0644 under podman and 0664 + gaia2 group under the FEH apptainer wrapper. - {state_dir}/eventd.log and {state_dir}/judgments.jsonl (a9e6435) were protected only by /var/gaia2/state being gaia2 0700. The FEH apptainer wrapper opens the state dir to the agent through the gaia2 group, because the setuid gaia2-exec is inert there and the agent CLIs must write events.jsonl. In adaptability scenarios with 2-3 oracle turns, judgments.jsonl gains a record (oracle tool calls and args) after each judged turn, while the agent is still working on the next one. gaia2-eventd gets --log-dir for eventd.log and judgments.jsonl (default --state-dir, so other callers are unchanged). The OpenClaw entrypoint points it at /var/gaia2/logs, created gaia2:gaia2 0700 via install -d, and makes /tmp/gaia2-eventd.log gaia2 0600 before eventd starts appending. The hermes, mini-swe and oracle entrypoints get the same /tmp log fix. events.jsonl, daemon_status.json and the other agent-facing files stay in the state dir. The runner reads judgments from /var/gaia2/logs, falling back to the state dir for older images, and still extracts both logs as root.
gaia2-exec runs as setuid-gaia2 so the agent user can invoke the GAIA2 CLIs without direct access to gaia2-owned state. It set GAIA2_STATE_DIR and re-injected libfaketime, but otherwise passed the agent's environment straight through. The agent controls that environment, and several CLIs choose a path from it: - cloud-drive (files app) and the emails --attachment-paths resolver take their sandbox root from GAIA2_FS_ROOT (apps/files.py, shared.resolve_sandbox_path). So `GAIA2_FS_ROOT=/var/gaia2 cloud-drive cat --path custom_scenario.json` ran as gaia2 and printed the oracle scenario (every oracle event and its args); `GAIA2_FS_ROOT=/var/gaia2/state` reached judgments.jsonl and the other state. - PYTHONPATH / PYTHONSTARTUP would let the agent import its own code into the gaia2 CLI process, i.e. arbitrary code execution as gaia2; LD_PRELOAD / LD_LIBRARY_PATH the same through the loader. - GAIA2_EVENT_ID is stamped into events.jsonl by log_action, so an agent-set value forges event ids in the action log. This matters on the native podman/docker path, where the setuid bit is honoured (no no-new-privileges in the launcher). Under the FEH apptainer wrapper the setuid bit is inert and the tool runs as the agent anyway, so our 0600 custom_scenario.json still protects it there; this closes the native path. gaia2-exec now snapshots a small locale/timezone allowlist (TZ, LANG, LC_*), drops the gaia2 privileges, clearenv()s, and rebuilds the environment from scratch: a fixed PATH, gaia2's HOME, GAIA2_STATE_DIR=/var/gaia2/state, the restored locale vars, and the libfaketime variables when faketime is configured (now including DONT_FAKE_MONOTONIC, matching the agent runtime). GAIA2_FS_ROOT, PYTHON*, LD_* and everything else the caller set are gone. The tool directory is a compile-time constant (GAIA2_EXEC_TOOL_DIR, default /usr/local/bin) so a non-setuid unit test can point it at a stub; it is never read from the environment.
Under the FEH apptainer wrapper the state dir is group-writable by the agent (setgid, no sticky bit), so the agent could replace daemon_status.json and the runner would record a forged success. eventd now writes daemon_status.json to --log-dir instead of the state dir (one copy whose location moves, not an extra copy), and the adapter /status and the runner read the verdict from there. log_dir defaults to state_dir, so callers that pass no --log-dir (hermes, mini-swe, oracle) still write and read it in the state dir, unchanged; the OpenClaw entrypoint points --log-dir at the gaia2-only /var/gaia2/logs (0700) and sets GAIA2_STATUS_DIR to match, moving the verdict out of the agent's reach. For older images that wrote the status file to the state dir, the runner falls back to reading it there.
The runner reads the scenario verdict (daemon_status.json) from the gaia2-only log dir and fell back to the agent-writable state copy whenever the log-dir copy was merely missing. A new image always has the log dir, so that let a missing log-dir copy hand the grade to a file the agent can forge under the FEH apptainer wrapper. Gate both readers that feed the verdict -- _read_daemon_status_from_container and the _extract_daemon_logs artifact fallback -- on the log dir being absent: fall back to the state copy only on genuinely old images that never had /var/gaia2/logs. The dir is probed with copy_from, which (unlike exec) works on a stopped/exited container. Behaviour on old images is unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change class: Evaluation-integrity — closes a PROVEN agent-exploitable hole.
Summary
On the pre-fix harness a solver agent could read the answer key and/or forge a passing grade. This closes both paths: it keeps
gaia2-eventd's logs/judgments and the scenario verdict out of the agent's reach, wipes the agent environment before a tool runs, and takes the verdict only from the private grader log dir (gating the legacy state-copy fallback on that dir being absent).Proof it was exploitable (our adversarial probes, with negative controls)
neg_iso, pre-fix imageare-1db32e3): a real OpenClaw exec-tool call read the privatescenario.jsonoracle (expected tool-call sequence) andjudge_context.json(original text/answer) —A.e2e_openclaw_exec_tool_cannot_read_private_files = FAIL; the live grader logs (/tmp/gaia2-eventd.log,/var/gaia2/logs) were agent-readable —C = FAIL. On the fixed harness: PASS.GET /statusserved the forged "pass" (w1 = FAIL/possible); on the fix, blocked (w1 = PASS). Two pre-fix negative controls confirm the detector is live.Impact
Without this, an agent could trivially "solve" any scenario by replaying the oracle events, or skip the task and forge a pass — the benchmark score would be meaningless.
Changes (4 commits)
Keep eventd logs/judgments out of agent reach;
gaia2-execwipes the agent env before running a tool; take the scenario verdict from the gaia2-only log dir; gate the legacy state-verdict fallback on a missing log dir. Touchesrunner/gaia2_runner/runner.py,shared/gaia2-exec.c,shared/gaia2_adapter_base.py, the openclaw/oracle entrypoints.Testing
runner/tests/test_runner.py(+154) covers the read-blocking and verdict-source behavior.Honesty caveat: a full audit of 1,361 real model trajectories (including on the vulnerable pre-fix images) found 0 oracle reads and 0 forged verdicts — these holes were exploitable but never actually exploited by the models we ran. This is preventative integrity-hardening backed by adversarial proof-of-possibility, not incident response.
Overlaps
runner.py/gaia2-exec.cwith the hardening and judge-text branches.Part of a set of isolated
gaia2-clifixes developed while building an audio benchmark on top of the GAIA2 harness. Draft; no reviewers assigned yet.Known residuals & integration notes
w2(accepted, by design): the agent can still append to/var/gaia2/state/events.jsonl— inherent to the FEH shared-group apptainer design (agent CLIs author their own event log). It is inert for grading because the judge trace-matches an oracle the agent can no longer read or reconstruct (closed by the reads above); a future hardening would make a trusted recorder the sole writer of the judged stream.r4(accepted, by design):GET /statusstays readable by the agent (turn/oracle-event count pre-action; failure reason post-judgment). The forge-via-/statuspath is closed here; the residual read is low-value (post-judgment is too late to act) and loopback-binding wouldn't help (shared net namespace). Tracked against the adapter-bind PR.gaia2-execenv-wipe (closing theGAIA2_FS_ROOT/PYTHONPATHsetuid vectors) is inert under apptainer (setuid disabled); on native podman/docker the image must be rebuilt with this change for those vectors to close.gaia2-exec.cwith the private-judge-context PR — merge this one first (clearenv()+allowlist is the broader base; it also stripsLD_*), then layer the other'sPYTHONNOUSERSITE=1.