Skip to content

Commit 9d710e0

Browse files
fix: add audited triage and review publication safeguards (#741)
* fix: add audited triage and review publication safeguards Co-authored-by: openhands <openhands@all-hands.dev> * fix: apply ownership and verdict policy to legacy reviews Co-authored-by: openhands <openhands@all-hands.dev> * docs: simplify triage and review instructions Co-authored-by: openhands <openhands@all-hands.dev> * build: refresh automation bundle sizes after wording edits Co-authored-by: openhands <openhands@all-hands.dev> * docs: distinguish issue readiness from permission and fix evidence Co-authored-by: openhands <openhands@all-hands.dev> * fix: refresh reviewer fixtures and generated skills catalog Co-authored-by: openhands <openhands@all-hands.dev> * Fix triage policy loading and missing-issue review handoff Co-authored-by: openhands <openhands@all-hands.dev> --------- Co-authored-by: openhands <openhands@all-hands.dev>
1 parent ba3f165 commit 9d710e0

19 files changed

Lines changed: 1949 additions & 430 deletions

File tree

‎.agents/skills/custom-codereview-guide.md‎

Lines changed: 44 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,32 @@ Cross-repository work is acceptable when this PR contains only the
2727
extensions-owned portion and relies on public interfaces from the owning
2828
repository.
2929

30-
## Blocking checkpoints
30+
## Issue triage
31+
32+
Issue readiness means the work is clear enough to start—not that the fix is
33+
complete. Do not require a PR, passing implementation tests, or before-and-after
34+
fix evidence. Reconsider readiness when new information leaves scope or expected
35+
behavior unresolved. Read the issue, linked discussion, applicable `AGENTS.md`,
36+
and adjacent source before asking questions.
37+
38+
For API failures, compare the extension request with the provider's reported or
39+
supported version, not upstream main. Use dependency pins or deployment evidence.
40+
Instructions, request arguments, response parsing, and scanner policy belong
41+
here; use the ownership boundaries above for other failures. State required
42+
cross-repository changes and their order. An independent upstream defect need
43+
not block this work. Naming an owner does not authorize edits or duplicate tickets.
44+
45+
Define the smallest useful outcome and explicit non-goals. Cover only affected
46+
entrypoints, environments, and permissions. For scanners, state whether discovery,
47+
commenting, and dispatch are allowed for ineligible candidates or after a cap.
48+
Discovery alone does not authorize unsolicited comments.
49+
50+
Request visual evidence only when needed; otherwise use logs, minimal
51+
reproductions, or failing tests. Do not repeat answered questions. Match managed
52+
criteria to current human decisions and authoritative readiness policy. Report
53+
conflicts instead of repeatedly adding a label that policy removes.
54+
55+
## Implementation review checkpoints
3156

3257
### Executable instructions and runtime parity
3358

@@ -42,6 +67,16 @@ in a clean environment far enough to exercise the changed behavior. Keep local
4267
and cloud behavior identical unless the documentation names and explains a real
4368
platform capability difference.
4469

70+
For provider changes, inspect the supported implementation; mocks and local types
71+
alone do not establish an API contract. Separate static/helper tests, tests with
72+
simulated services, and real integration evidence. If the version or runtime is
73+
unavailable, report that limit, not an assumed defect. For packaging changes,
74+
check source, shipped bundle, and deployed files separately.
75+
76+
For scanners, check eligibility, permissions, caps, retries, unchanged runs, and
77+
recovery for each action listed under triage. A launch cap does not permit
78+
unlimited gate comments.
79+
4580
### Untrusted triggers and credentials
4681

4782
Treat repository events, issue and PR comments, chat messages, catalog fields,
@@ -130,3 +165,11 @@ Do not comment on punctuation, preferred prose tone, optional DRY cleanup, or
130165
hypothetical non-standard configurations without a supported failure mode. Before
131166
raising a finding, verify that the referenced file and behavior are present in
132167
the current PR head and have not already been addressed in review history.
168+
169+
## Review verdict
170+
171+
Do not approve material code defects or unmet blocking merge/evidence checks.
172+
Distinguish code defects from missing merge requirements and optional follow-up.
173+
If GitHub forbids self-approval of a clean PR, use COMMENT and explain why.
174+
Respect accepted scope and non-goals: block on a pre-existing limitation only if
175+
this change exposes it.

‎automations/bundle-index.js‎

Lines changed: 8 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎automations/catalog/github-issue-triage/manifest.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,14 @@
5454
}
5555
},
5656
"bundle": {
57-
"version": "1.3.1",
57+
"version": "1.4.0",
5858
"entrypoint": "python3 worker.py",
5959
"timeout": 300,
6060
"files": {
6161
"worker.py": "skills/github-issue-triage/scripts/worker.py",
6262
"agent_conversation.py": "skills/github/scripts/agent_conversation.py",
63-
"github_client.py": "skills/github/scripts/github_client.py"
63+
"github_client.py": "skills/github/scripts/github_client.py",
64+
"triage_publication.py": "skills/github-issue-triage/scripts/triage_publication.py"
6465
},
6566
"config": {
6667
"repos": "{{form.repositories}}",

‎automations/catalog/github-pr-reviewer/manifest.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,15 +135,16 @@
135135
}
136136
},
137137
"bundle": {
138-
"version": "1.9.2",
138+
"version": "1.10.0",
139139
"entrypoint": "python3 worker.py",
140140
"timeout": 300,
141141
"files": {
142142
"main.py": "skills/github-pr-reviewer/scripts/main.py",
143143
"agent_conversation.py": "skills/github/scripts/agent_conversation.py",
144144
"github_client.py": "skills/github/scripts/github_client.py",
145145
"maintainer_handoff.py": "skills/github-pr-reviewer/scripts/maintainer_handoff.py",
146-
"worker.py": "skills/github-pr-reviewer/scripts/worker.py"
146+
"worker.py": "skills/github-pr-reviewer/scripts/worker.py",
147+
"publication.py": "skills/github-pr-reviewer/scripts/publication.py"
147148
},
148149
"config": {
149150
"repos": "{{form.repositories}}",
Lines changed: 73 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,75 @@
11
# GitHub issue triage
22

3-
Prioritize open issues and establish acceptance criteria before marking them ready for development.
4-
5-
Schedule this automation independently and select its agent profile. The profile
6-
chooses the model, tools, and GitHub credential; the PAT needs Issues write access
7-
only. Explicit issue dependencies must be completed before development starts.
8-
The agent checks repository guidance and adjacent code, preserves the author's
9-
issue description, and maintains accepted criteria in one clearly marked section
10-
at the end of the issue body. This lets repository readiness checks evaluate the
11-
same source that developers and reviewers use. When a material design decision
12-
remains unclear after repository research, the agent leaves the body unchanged,
13-
asks focused follow-up questions in a marked comment, and withholds the readiness
14-
label.
15-
16-
See [SKILL.md](SKILL.md) for configuration and the catalog bundle for its files.
3+
Prioritize issues and define observable acceptance criteria using
4+
[SKILL.md](SKILL.md). The scanner honors dependencies and sends changed issues
5+
to conversations using the selected profile.
6+
7+
## Publication modes
8+
9+
Legacy agent publication is the default. The optional `triage_publication.py`
10+
helper validates JSON recommendations and checks human inputs before every write.
11+
It preserves text outside managed markers and edits/deletes only triage-marked
12+
comments from the authenticated account. It skips identical writes and stops on
13+
ambiguous markers. GitHub Actions readiness comments provide context, but neither
14+
change the human-input fingerprint nor override policy.
15+
16+
This opt-in is **not credential isolation**: the agent can still write directly.
17+
Enforced isolation needs a separate trusted publisher, read-only agent credentials,
18+
and runtime support for passing results to the publisher.
19+
20+
## Operator setup
21+
22+
Bundle `worker.py`, `triage_publication.py`, `github_client.py`, and
23+
`agent_conversation.py`. Entrypoint: `python3 worker.py`.
24+
25+
1. Place the publisher, `github_client.py`, and protected config in the delegated
26+
workspace. Set `triage_publisher_path` and `triage_publisher_config_path` to
27+
absolute file paths. Set both or neither; neither keeps legacy mode.
28+
2. Set `triage_publisher_workspace: "shared-host"` only after checking manually
29+
that the agent can access those paths. The flag asserts access; it does not
30+
prove sandbox visibility. The scanner checks files and the repository allowlist before dispatch, then
31+
loads readiness policies from that same protected publisher config for its
32+
input fingerprints. Do not duplicate policies in the scanner config. Remote/container paths are unsupported. If the
33+
agent cannot invoke the helper, stop; do not fall back to direct writes.
34+
3. Cloud conversations are unsupported: bundling does not provision their files,
35+
and the dispatcher cannot retrieve structured results from them.
36+
37+
Publisher config uses existing `repos` and optional `triage_readiness_policies`:
38+
39+
```json
40+
{
41+
"repos": ["owner/repo"],
42+
"triage_readiness_policies": {
43+
"owner/repo": {
44+
"mode": "authorized-writers",
45+
"files": {".github/workflows/issue-readiness-check.yml": "<audited GitHub blob SHA>"}
46+
}
47+
}
48+
}
49+
```
50+
51+
Audit current policy and pin every relevant file to its actual 40-character
52+
GitHub blob SHA (the file-content identifier). Only writer-only policies are
53+
supported: the helper checks current SHAs and live `write`/`maintain`/`admin`
54+
permission. Never select this mode for a policy that checks issue content.
55+
Unknown, changed, or unsupported policies block new readiness grants, preserve
56+
existing labels, and still allow criteria publication. No repository checker runs.
57+
58+
Precreate `ready-for-dev` and `priority:low/medium/high` labels. Use the same
59+
GitHub account for scanner and publisher. Its fine-grained PAT needs selected-repo
60+
Issues read/write, Contents read, and permission lookup access. Never put token
61+
values in prompts or automation definitions.
62+
63+
## Races and retries
64+
65+
- Checks and writes are not atomic: people or other automations can change inputs
66+
between GET and PATCH. There is no shared lock or all-or-nothing update across
67+
issue bodies, comments, and labels. Changed inputs stop publication; the next
68+
scan recomputes the recommendation.
69+
- The input fingerprint includes human readiness-label changes from the issue
70+
timeline, but excludes the publisher's and GitHub Actions' label changes.
71+
- A completion receipt is saved only after publication finishes. Partial writes
72+
change the guarded-mode delivery key, allowing retries. If an unchanged failure
73+
is marked finished by the dispatcher, manually retry the same command.
74+
- Configured policy changes affect the fingerprint. Upstream policy changes alone
75+
do not automatically retriage unchanged issues with managed markers.

‎skills/github-issue-triage/SKILL.md‎

Lines changed: 26 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -9,49 +9,29 @@ triggers:
99

1010
Prioritize open issues and establish acceptance criteria before marking them ready for development.
1111

12-
Create this automation separately from implementation and review. Use a
13-
fine-grained GitHub PAT limited to the selected repositories with Issues: read and
14-
write. Never put the token value in the automation definition or prompt.
15-
16-
Package `worker.py` with the shared `github_client.py` and
17-
`agent_conversation.py` helpers. Supply `config.json` with `repos` and the saved
18-
GitHub secret name; the entrypoint is `python3 worker.py`. Include that secret in
19-
the selected profile so the delegated agent can use GitHub. Naming it in the
20-
automation does not grant it to the agent.
21-
22-
The scanner is an ordinary Automation host command. It submits selected issues
23-
through the shared KV-backed conversation dispatcher, which creates or resumes
24-
one stable conversation per issue through the Software Agent SDK. Only that
25-
agent workspace is local or Docker. Automation owns scheduling, cancellation,
26-
and cleanup.
27-
28-
Honor `Depends on: #12, #13` lines. A dependency must be closed as completed.
29-
Do not treat an existing `ready-for-dev` label as proof of readiness: inspect the
30-
issue discussion, repository instructions, and the closest relevant or adjacent
31-
implementation. Resolve reasonable ambiguity with a bounded scope and explicit
32-
non-goals. If a material product or design decision still cannot be inferred
33-
safely, ask only the focused follow-up questions needed to resolve it and withhold
34-
`ready-for-dev`; do not invent acceptance criteria around an arbitrary choice.
35-
36-
Acceptance criteria must be observable and sufficient for a reviewer to decide
37-
that the requested behavior is complete. Check the main behavior and every
38-
applicable boundary: failures and edge cases, compatibility or migration,
39-
lifecycle and cleanup, permissions and secrets, user-facing documentation, and
40-
realistic automated or live validation. Avoid subjective criteria and avoid
41-
prescribing an implementation unless repository policy requires one mechanism.
42-
43-
For an issue that is ready, preserve the author's description and maintain one
44-
clearly marked OpenHands AI section at the end of the issue body. Put the triage
45-
scope, any missing repository-required readiness sections, and testable
46-
acceptance criteria there. Replace only that marked section on later runs and do
47-
not also post a triage comment. Add `ready-for-dev` only after the final criteria
48-
pass this standard.
49-
50-
If the design is still unclear, remove any stale managed body section, leave the
51-
human-owned body unchanged, and post the minimum focused questions in one marked
52-
comment below the human discussion. Withhold `ready-for-dev`. Preserve unrelated
53-
labels and never edit or delete human-authored text.
54-
Do not implement code or accept pull requests.
55-
56-
Each scheduled run submits every changed eligible issue. A failure on one issue is
57-
reported and does not prevent the remaining issues from being submitted.
12+
## Setup
13+
14+
Keep triage separate from implementation and review. Follow [README.md](README.md)
15+
for packaging, credentials, publisher setup, and limits. Never put token values
16+
in the automation definition or prompt. Legacy agent publication is the default.
17+
18+
## Triage instructions
19+
20+
- Honor explicit dependencies. Read applicable root/local `AGENTS.md`,
21+
`.agents/skills/custom-codereview-guide.md` if present, human discussion, and
22+
adjacent code. Identify affected versions and the owning component (Canvas,
23+
SDK, automation, or extensions).
24+
- Follow current human decisions over stale generated criteria. Do not repeat
25+
answered questions. Keep scope bounded, name non-goals, and ask only unresolved
26+
material product/design questions.
27+
- Define observable completion, not entry requirements. Do not require tests for
28+
unwritten code, PR artifacts, or images/videos for every issue. Default priority
29+
to low; medium/high need recorded real user pain, not speculation or synthetic
30+
tests.
31+
- Return structured recommendations. If a publisher is configured, use it; never
32+
bypass it or its changed-input checks. Legacy direct mode follows the same rules
33+
without enforcement.
34+
- Preserve human text and unrelated labels. Never run repository checkers with
35+
credentials. Use current repository policy audited by the deployment owner, not
36+
historical checker comments, to decide readiness.
37+
- Do not implement code or accept pull requests.

0 commit comments

Comments
 (0)