Skip to content

fix(e2e-exe-dev): never score a failure notice as a ping reply - #28

Merged
glifocat merged 2 commits into
mainfrom
fix/ping-failure-notice
Oct 1, 2026
Merged

glifocat merged 2 commits into
mainfrom
fix/ping-failure-notice

Conversation

@glifocat

@glifocat glifocat commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

What changes?

The headless installer counted any non-empty ping reply as a pass. With a bad API key, NanoClaw answers the ping with its failure notice, and the run ended in:

PING: ok
REPLY: The agent run failed. Check the logs for details.

Found during the nanocoai/nanoclaw#3901 r6 e2e on 2026-10-01. The cause there was an unfilled placeholder in the credential file.

Ping classification (e2e-install.sh, shared by e2e-exe-dev headless, e2e-proxmox and e2e-macos):

Credential check:

  • exe-run.sh and proxmox-run.py refuse an empty or placeholder Anthropic credential file, or one without an sk-ant- value, before creating a VM or guest. The value is checked exactly as it is sent (only CR/LF removed) and is never printed.
  • With --base or a remote OneCLI vault, the secret may already exist, so the host leaves the check to the installer. The installer checks the file only right before handing it to the gateway, so a reused vault with an unused placeholder file still passes.

Plugin 0.12.2; CHANGELOG, catalog and the e2e-exe-dev SKILL.md are updated.

The same gap in nanoclaw's own setup ping is fixed separately in nanocoai/nanoclaw#3980.

Validation

  • Offline (at cd9580d; exit-4 commit 5772a59 re-ran test_e2e + contracts, 80 OK): python -m unittest discover -s tests (with skills/e2e-wizard/requirements.txt installed): 354 tests, one error. That error is test_payload_transport.test_alternate_owning_remote_transports_exact_selected_commit, which also fails on main.
  • The new installer tests fail against main's e2e-install.sh: notice text, late notice, the pre-#3746 Error: reply, the marker with arbitrary text, and placeholder/foreign/embedded-space credentials. They pass with this change. Tests also cover: earlier and other-channel notices ignored, the marker trusted over quoted text, fallback when no DB is readable, hardened-image fallback (case-insensitive, .env and environment), a reused vault ignoring a placeholder file, --base skipping the host check, and no VM or guest allocation on a bad credential.
  • tests/test_repository_contracts.py passes (version 0.12.2 in plugin.json, catalog and CHANGELOG).
  • Codex adversarial review, 3 rounds: 9 findings, 8 fixed, 1 refuted. Round 3 asked to scope notices to the e2e agent's group; that contradicts round 1, and notices on cli/local are what chat.ts prints, so scoping is by channel.
  • Live, exe.dev headless, onecli, nanoclaw main 962d527c, installer from this branch at 5772a59 (2026-10-01):
    • Placeholder credential (the real unfilled anthropic_api_key file): exit 66 before any VM was created; value not printed.
    • Good OAuth token: STATUS: pass, PING: ok, real agent reply, VM removed by --rm (removal verified).
    • Fake but well-formed API key: exit 2, ping: no_reply, not a pass. The SDK retried the 401s, so the runner's marked notice (cli/local, failureNotice: true) landed about 3 minutes after the ping, past chat.ts's 120 s stop. Running this branch's failure_notice_ids against that VM's real outbound.db returned the notice's id (exit 0). So the marker read works on a real install; the in-window path is covered by the offline tests. VM deleted after inspection.

A bad credential ended in PING: ok with REPLY "The agent run failed.
Check the logs for details." because any non-empty reply counted.

The headless installer (exe.dev, Proxmox, macOS) now records ping
agent_failure (exit 2) when a session gets a new cli/local
messages_out row marked failureNotice (nanocoai/nanoclaw#3908). When
the runner may predate the marker (checkout without it, a pulled
hardened image) or a DB cannot be read, the notice text and a reply
starting "Error: " (pre-#3746) also count.

exe-run.sh and proxmox-run.py refuse an empty, placeholder or
non-sk-ant Anthropic credential before creating a VM or guest, except
where a reused vault may not need it (--base, remote OneCLI); the
installer checks the file right before handing it to the gateway.
Values are never printed. Plugin 0.12.2.
nanocoai/nanoclaw#3980 makes chat.ts forward the failureNotice flag and
exit 4 when a reply carried it. Without this case the installer recorded
that as no_reply.
@glifocat
glifocat marked this pull request as ready for review October 1, 2026 16:00
@glifocat
glifocat merged commit 357a355 into main Oct 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant