Skip to content

Commit 227bb91

Browse files
committed
Add a design for the pull request review agent
Signed-off-by: Nic Cope <nicc@rk0n.org>
1 parent 2a8908f commit 227bb91

1 file changed

Lines changed: 92 additions & 0 deletions

File tree

‎design/pr-review-agent.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Pull request review agent
2+
3+
**Status:** Draft
4+
**Date:** November 2026
5+
**Author:** Nic Cope
6+
7+
## Background
8+
9+
Agents write all of Modelplane's code, and most of its review. Each maintainer
10+
points their own agent at a PR, with their own harness and agent skills. We
11+
don't have skills of our own for finding correctness bugs, so one shared agent
12+
could look for them on every PR as well as ours do. For style and taste, each
13+
agent applies its owner's conventions. Skills kept in the repo would let us
14+
agree on ours, and give the same feedback whoever reviews.
15+
16+
This doesn't replace maintainer review. It frees maintainers to spend review on
17+
the insight only they have.
18+
19+
## Proposal
20+
21+
An agent reviews a PR when an org member opens it or marks it ready. To review
22+
anyone else's PR, or to re-review after new pushes, a maintainer comments
23+
`/review`. A re-review covers only what `git range-diff` shows changed since
24+
the last one.
25+
26+
The review agent runs four reviewers with fresh context: design, prose, craft
27+
and correctness. It checks their findings against the bar in
28+
`.agents/skills/reviewing-pull-requests` and posts the ones that pass as one
29+
comment review from a GitHub App. It never approves or requests changes. The
30+
agents live in `.opencode/agents`, and the skills they load in `.agents/skills`,
31+
where any maintainer can tune them or run the review locally:
32+
33+
```bash
34+
opencode run --agent review "Review PR #556 in modelplaneai/modelplane."
35+
```
36+
37+
## Security
38+
39+
Every PR comes from a fork, and the workflow runs on `pull_request_target`, the
40+
trigger that gives a fork's PR access to secrets. The agent then runs the PR's
41+
code with a shell and network, because it verifies findings by running tests and
42+
reading upstream source. We assume the PR is malicious. Its code must never get
43+
the Anthropic key or any credential that can write to the repo.
44+
45+
**The agent runs in a Firecracker microVM.** GitHub's x86 runners expose KVM.
46+
Escaping the VM takes an exploit in KVM or Firecracker, which is the boundary
47+
AWS Lambda relies on, rather than in the shared kernel a container would give
48+
us.
49+
50+
**The only credential in the VM can't do harm.**
51+
52+
- The Anthropic key stays on the runner. A proxy there adds it to the agent's
53+
model requests, and forwards nothing but `POST /v1/messages`. The key belongs
54+
to a workspace of its own, with a spend limit.
55+
- The VM gets the job's GitHub token, so the agent can read issues and PRs with
56+
`gh`. It can't write anything, everything it can read is public, and it
57+
expires with the job.
58+
- The App's key exists only in a second job with no model in it. That job
59+
validates the review and posts it.
60+
61+
**The VM can only connect to the proxy and the internet.** A firewall on the
62+
runner drops its connections to the runner's other ports and private network,
63+
and to the cloud metadata service. It drops all IPv6. It doesn't share files
64+
with the runner. Its job arrives as a tar on a read-only disk, and it writes the
65+
review as a tar that the post job reads in memory, never extracting it.
66+
67+
**A PR can't change the reviewer's setup.** The VM image, the workflow, the
68+
agents, the skills and every other file opencode reads come from `main`. The
69+
post job escapes any marker the agent writes, so the agent can't set what the
70+
next re-review covers.
71+
72+
What's left: a malicious PR can spend API budget, and post a misleading comment
73+
review. The spend limit bounds the first, a maintainer reads the second, and
74+
non-members get a review only when a maintainer asks for one. PR code running
75+
in the VM can rewrite the agent's instructions mid-review, so a PR can steer its
76+
own review. It's still only a comment, so we accept that.
77+
78+
GitHub doesn't officially support KVM on its runners. About one runner in ten
79+
during testing had none. A small workflow re-runs a review that failed for that
80+
reason, up to twice.
81+
82+
## Alternatives considered
83+
84+
**opencode's GitHub Action.** The agent posts directly, so it holds a token that
85+
can write to the repo.
86+
87+
**A container instead of a VM.** A kernel exploit in the container would give it
88+
the runner, and the key on it. Docker can't run inside an unprivileged container
89+
either, so the agent couldn't create a kind cluster.
90+
91+
**No shell for the agent.** It wouldn't need a sandbox, but it couldn't verify
92+
findings. Unverified findings are most of the noise maintainers decline.

0 commit comments

Comments
 (0)