Skip to content

Commit 374c832

Browse files
authored
Merge pull request #181 from Yif-Yang/fix/superpowers-evaluator-hardening
fix(sleep): harden Superpowers evaluator evidence
2 parents 59faf00 + f5c15ec commit 374c832

3 files changed

Lines changed: 668 additions & 66 deletions

File tree

‎docs/superpowers/SECURITY.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,21 @@ isolation boundary.
3434
the agent can reach the shim, the nonce and the audit log, and can modify the
3535
project tree the harness re-runs from. It is meaningful because the candidate
3636
is trusted; it is not an adversarial oracle.
37-
- `harness_test_passes` — the harness re-runs the tests itself after the
38-
agent exits, so agent *output* alone cannot fake a pass.
37+
- `harness_test_passes` — the harness re-runs the protected test paths after
38+
the agent exits, ignoring project pytest config and `conftest.py`, so agent
39+
*output* alone cannot fake a pass. A pass requires at least one executed
40+
passing test and no failures, errors or skips.
3941
- `pytest_runs` — count of nonce-tagged invocations of the `pytest`/`python`
4042
shims.
43+
- `pytest_successes` / `pytest_failures` — completed shim invocations are
44+
classified from JUnit results; exit code 0 without an executed passing test
45+
(for example `pytest --help` or an all-skipped run) is inconclusive and is
46+
counted as neither a success nor a failure.
4147
- `pytest_after_edit` — the last shim invocation is newer than the newest
4248
project `*.py`, so "fix, then claim done without re-running" fails.
49+
- Protected scenario fixtures are hashed before the agent runs. A modified,
50+
deleted or symlink-replaced fixture fails the scenario and is not executed
51+
by the harness re-run.
4352

4453
⚠️ The verification re-run **executes agent-modified project code on the
4554
host**. `ANTHROPIC_API_KEY` is dropped from that re-run's environment, but

0 commit comments

Comments
 (0)