Repository navigation
Conversation
added 4 commits
October 13, 2026 13:59
When gaia2-eventd died during setup (e.g. creating the judge), it never
wrote daemon_status.json, nothing sent the task, the adapter kept
answering {"status": "waiting"}, and the runner only gave up after its
600 s activity timeout with "Agent produced no response and no events".
eventd now records why it stopped: a setup failure, a crash of the main
loop, or an adapter that never connected all write status "error" with
a one-line "error" field (exception type, message and innermost frame);
a failed setup also exits 1. The adapter already serves the file as is.
The runner fails such a scenario as soon as it reads that status, with
"Daemon error: <reason>", never as a model FAIL; and it stops waiting
when the adapter has answered "waiting" for 120 s after becoming ready
(healthy daemons report "running" before the adapter reports ready).
Healthy runs and the existing error messages are unchanged.
Addresses: H3 (eventd fatal-error fast fail)
COPY keeps the build context's modes. A checkout made under umask 0007 (the cluster default) has 0660 files, so every file copied into the images was unreadable to the in-image gaia2 and agent users; the OpenClaw build failed at "RUN PROVIDER=anthropic bash /etc/profile.d/openclaw-setup.sh" (Permission denied). Every COPY from the build context in cli/ and all four runtime Dockerfiles now sets a numeric --chmod (0644 for data, 0755 for scripts and copied directories), replacing the separate chmod +x steps. buildah 1.23 (podman 3.4.4 on fair-sc) accepts numeric modes only. COPY --from (node, chatbot, gaia2-exec), the setuid gaia2-exec, /var/gaia2 (0700) and /opt/gaia2_filesystem (0700) are unchanged. A new static test parses every Dockerfile and fails on a context COPY without a readable, non-writable numeric mode. Verified by building cli/ and containers/openclaw/ from a context whose files were all forced to 0660: the OpenClaw build passes the openclaw-setup.sh step and every /opt file is 0644/0755, whereas the same COPY layers from a9fd2ed leave them 0660 and agent gets exit 126. Addresses: H4 (image file modes)
Launchers that pass the host environment through (FEH's apptainer launcher) handed the container the fair-sc login node's SSL_CERT_FILE=/var/facebook/x509_identities/tls.crt, which exists on neither compute nodes nor in the image. litellm 1.105.0 builds an httpx SSL context while it is imported, so gaia2-eventd died with FileNotFoundError creating the judge; requests and curl fail the same way on REQUESTS_CA_BUNDLE / CURL_CA_BUNDLE. shared/gaia2-tls-env.sh (installed as /opt/gaia2-tls-env.sh) provides sanitize_tls_env, which every gaia2-init-entrypoint.sh calls as root before it starts any process: SSL_CERT_FILE, REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE that do not name a PEM bundle readable by both the gaia2 and agent users (checked with su, since root reads anything) are pointed at the image bundle /etc/ssl/certs/ca-certificates.crt (or unset if that is missing too); a missing SSL_CERT_DIR and an unusable NODE_EXTRA_CA_CERTS are unset. A world-readable bundle mounted by the runner (GAIA2_CA_BUNDLE -> /etc/ssl/certs/gaia2-host-ca-bundle.crt) is kept. Each replacement is logged. gaia2-eventd is also started with LITELLM_LOCAL_MODEL_COST_MAP=True, so importing litellm for the judge never fetches the remote price map. Verified in a built OpenClaw image: with the Stage 0 login-node env, SSL_CERT_FILE becomes the image bundle and gaia2 imports litellm; a readable mounted bundle is kept; a 0660 mounted bundle is replaced. Addresses: H2 (entrypoint env sanitisation)
The judge extra declares litellm>=1.0, so every fresh image build resolved the newest litellm and dependencies: the a9fd2ed image got litellm 1.105.0 where an earlier build of the same tree had 1.98.0, and 1.105.0 changed import-time behaviour (it builds an httpx SSL context). cli/image-constraints.txt pins all 58 third-party packages the base image installs, at the versions of the a9fd2ed image that passed the Stage 0 judged runs (litellm 1.105.0, httpx 0.28.1, openai 2.54.0, ...), read from that image's layers. cli/Dockerfile passes it with -c to all three pip installs (gaia2-core, gaia2-cli[judge], huggingface_hub) and deletes it afterwards. A constraints file rather than a tighter pyproject pin: it freezes the whole closure, not just litellm, and it leaves cli/uv.lock and runner/uv.lock (host-side, checked with uv --locked in CI) untouched. Keeping 1.105.0 rather than going back to 1.98.0 keeps the validated image's stack; its import-time SSL issue is handled by the entrypoint (previous commit). Tests check that every pip install uses the file, that it is copied first, that every pin is exact and satisfies the declared requirements. A real build of cli/Dockerfile with the constraints installed exactly the pinned set (pip dry-run resolution: 60 packages, none unpinned). Addresses: H1 (litellm pin)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change class: Reliability / reproducibility — not an agent-exploit vector.
Summary
Container + runner robustness for the
gaia2-cliimage; each commit fixes an observed operational failure.Consequence addressed (4 commits)
gaia2-eventderrors instead of idling out — if the grader dies, the runner errors immediately instead of hanging until wall-time (eliminates silent idle-timeouts that waste GPU and mask problems).shared/gaia2-tls-env.sh) — strips inheritedSSL_CERT_FILE/CA vars that are invalid inside the container and break TLS clients.None of these change the agent/grader trust boundary; they are reliability + reproducibility. Touches
containers/oracle/Dockerfile, the oracle entrypoint,runner/gaia2_runner/runner.py,shared/gaia2-tls-env.sh.Testing
runner/tests/test_runner.py+containers/openclaw/test_gaia2_adapter.py(+847 test lines) cover fatal-error surfacing and env cleanup.Overlaps
runner.pywith the oracle-exposure and judge-text branches.Part of a set of isolated
gaia2-clifixes developed while building an audio benchmark on top of the GAIA2 harness. Draft; no reviewers assigned yet.