Skip to content

Commit 3bfeaf0

Browse files
enystclaude
andauthored
fix(openhands-automation): stop steering agents away from deterministic scripts (#686)
* fix(openhands-automation): stop steering agents away from deterministic scripts The skill's rule 0 tells agents to prefer a no-LLM custom script for deterministic tasks, but the rest of the text pulled the other way: - Prompt-preset examples modelled deterministic tasks (hourly health check, run tests on push, weekly temp-file cleanup). Replace them with tasks that need judgment: error triage, bug reproduction, dependency review, flagging risky pushes. - references/custom-automation.md opened with "Do NOT use this reference unless the user has explicitly requested a custom automation", which contradicted rule 0 and the Reference Files section. Remove it and list deterministic tasks as a first-class reason to use the custom path. - Add a complete no-LLM worked example (hourly health check with Slack alert) next to the prompt-preset examples, so both paths are equally concrete. - Make consent symmetric: present the options side by side and build the one the user picks. The "ready to deploy?" confirmation applies to every path; custom scripts no longer need an extra permission step. Also drops a duplicated references/security.md bullet and regenerates the skills catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(openhands-automation): fix rule cross-reference and rebalance README Address review feedback on #686: - SKILL.md: the polling section pointed at rule 5, which became rule 4 when rules 2-4 were collapsed. - README.md: stop presenting the prompt preset as the recommended default and custom automations as advanced-only; list custom scripts alongside prompt-based creation and point to the no-LLM worked example. - Regenerate skills/index.js. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(openhands-automation): make the no-LLM example's get_secret work on Cloud The worked example added by this PR carried the pre-#709 get_secret, which only reads from the agent server. Match the helper now on main so the example also fetches secrets from the OpenHands API on Cloud and Enterprise runs. Regenerate skills/index.js. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: remove tracker file committed by mistake b588ba6 picked up .beads/issues.jsonl through the local bd pre-commit hook, which stages that file on every commit. It is local task-tracker data and does not belong in this repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 8b05933 commit 3bfeaf0

4 files changed

Lines changed: 113 additions & 30 deletions

File tree

‎skills/index.js‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎skills/openhands-automation/README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,22 @@ This skill is activated by keywords:
1616

1717
## Features
1818

19-
- **Prompt-based creation**: Create automations from a natural language prompt (recommended)
19+
- **Prompt-based creation**: Create automations from a natural language prompt, for tasks that benefit from agent reasoning
20+
- **Custom scripts**: Run your own code, with or without an LLM — the cheaper, more reliable fit for deterministic tasks (fixed data, scheduled HTTP calls, templated messages), and for custom dependencies or full control (see [references/custom-automation.md](references/custom-automation.md))
2021
- **Event-triggered automations**: Trigger on GitHub events (PR opened, issue commented, push, etc.)
2122
- **Custom webhooks**: Register webhooks for any service (Stripe, Slack, Linear, etc.)
2223
- **JMESPath filters**: Match events based on payload content (labels, mentions, repos)
2324
- **Automation management**: List, update, enable/disable, and delete automations
2425
- **Manual dispatch**: Trigger automation runs on-demand
25-
- **Custom automations**: For advanced users who need full control (see [references/custom-automation.md](references/custom-automation.md))
2626

2727
## API Base URL
2828

2929
Use an explicitly provided host first. When a local Agent Canvas automation server is running and no host is provided, use `http://localhost:8001/api/automation/v1`. Otherwise use the cloud default: `https://app.all-hands.dev/api/automation/v1`.
3030

3131
## Quick Start
3232

33+
The examples below use the prompt preset, which suits tasks that need reasoning or judgment. For a deterministic task that needs no LLM, use a custom script instead — see the *Custom Script Example (No LLM)* in [SKILL.md](SKILL.md).
34+
3335
### Cron-Triggered Automation
3436

3537
```bash
@@ -81,9 +83,9 @@ curl -X POST "https://app.all-hands.dev/api/automation/v1/preset/prompt" \
8183
}'
8284
```
8385

84-
The service handles SDK code generation, tarball packaging, upload, and automation creation automatically.
86+
For preset automations, the service handles SDK code generation, tarball packaging, upload, and automation creation automatically.
8587

8688
## See Also
8789

8890
- [SKILL.md](SKILL.md) — Full API reference, agent behavior rules, event keys, filters, and examples
89-
- [references/custom-automation.md](references/custom-automation.md) — Reference for custom automations with user-provided SDK scripts
91+
- [references/custom-automation.md](references/custom-automation.md) — Reference for custom automations with user-provided scripts, SDK-based or deterministic (no LLM)

‎skills/openhands-automation/SKILL.md‎

Lines changed: 103 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -64,14 +64,14 @@ The agent server typically runs inside a **sandbox** (a Docker or Kubernetes con
6464
> 1. **For LLM-appropriate work, default to preset endpoints.** They handle all SDK boilerplate, tarball packaging, and upload automatically:
6565
> - **Prompt preset** (`POST /v1/preset/prompt`) — for tasks expressed as a natural language prompt that benefit from agent reasoning
6666
> - **Plugin preset** (`POST /v1/preset/plugin`) — when plugins with skills, MCP configs, or commands are needed
67-
> 2. **Do not silently create custom scripts.** Do not generate Python code, `setup.sh` files, or tarball uploads without user consent. But *do* proactively recommend the custom path (per rule 0) when the task is deterministic or high-frequency — surface the option and let the user choose.
68-
> 3. **If neither preset is the right fit**, do NOT silently fall back to custom automation. Instead, explain the available options to the user:
69-
> - **Prompt preset** — natural language prompt execution (LLM-driven)
70-
> - **Plugin preset** — load plugins with extended capabilities (skills, MCP, hooks, commands)
71-
> - **Custom script** — full control over code, with or without LLM; point them to `references/custom-automation.md`
72-
> - Let the user choose which approach to use.
73-
> 4. **Only create custom scripts after the user agrees to that path.** Refer to `references/custom-automation.md` for the full reference.
74-
> 5. **Before suggesting event-triggered (webhook) automations, check whether the deployment is publicly reachable.** Check `RUNTIME_URL`. Webhooks require an internet-accessible URL so that external services (GitHub, Slack, Linear, etc.) can deliver events to the automation service. If `RUNTIME_URL` is unset, empty, or resolves to a local or private address (`localhost`, `127.0.0.1`, `0.0.0.0`, or any RFC 1918 range: `10.x.x.x`, `192.168.x.x`, `172.16–31.x.x`), the service cannot receive inbound webhook traffic from the public internet. In that case:
67+
> 2. **Present the options, then build the one the user picks.** Whenever rule 0 applies, or neither preset is a clean fit (custom Python dependencies, a non-Python entrypoint, a multi-file project, direct SDK lifecycle control), lay the choices out side by side with a one-line note on cost and reliability for each, and let the user choose:
68+
> - **Prompt preset** - natural language prompt execution (LLM-driven)
69+
> - **Plugin preset** - load plugins with extended capabilities (skills, MCP, hooks, commands)
70+
> - **Custom script** - full control over code, with or without an LLM; see `references/custom-automation.md`
71+
>
72+
> Do not silently fall back to either side. The "ready to deploy?" confirmation in the creation process applies to every path equally; a custom script needs no extra permission beyond it, and a preset gets no pass on it.
73+
> 3. **Building a custom script:** follow `references/custom-automation.md` for tarball packaging, validation, and upload. The *Custom Script Example (No LLM)* below shows the whole flow end to end.
74+
> 4. **Before suggesting event-triggered (webhook) automations, check whether the deployment is publicly reachable.** Check `RUNTIME_URL`. Webhooks require an internet-accessible URL so that external services (GitHub, Slack, Linear, etc.) can deliver events to the automation service. If `RUNTIME_URL` is unset, empty, or resolves to a local or private address (`localhost`, `127.0.0.1`, `0.0.0.0`, or any RFC 1918 range: `10.x.x.x`, `192.168.x.x`, `172.16–31.x.x`), the service cannot receive inbound webhook traffic from the public internet. In that case:
7575
> - **Recommend a cron-based polling automation instead.** Have the automation run on a schedule and call the external service's API (e.g., the GitHub REST API) to check for new events since the last run.
7676
> - Explain the limitation clearly to the user: "Because this is a local deployment, external services can't reach the webhook endpoint. I'll set up a polling automation using a cron schedule instead."
7777
@@ -255,8 +255,10 @@ curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
255255
Write the prompt as an instruction to an AI agent. The prompt executes inside a sandbox with full tool access (bash, file editing, etc.), the user's configured LLM, stored secrets, and MCP server integrations. Examples:
256256

257257
- `"Generate a weekly status report summarizing the team's GitHub activity and post it to Slack"`
258-
- `"Check the production API health endpoint every hour and alert if it returns non-200"`
259-
- `"Pull the latest data from our analytics API and update the dashboard spreadsheet"`
258+
- `"Read yesterday's new error reports in Sentry, group them by likely root cause, and open one GitHub issue per new failure pattern with a proposed fix"`
259+
- `"Go through support tickets tagged 'bug' since the last run, try to reproduce each against main, and comment on the ticket with what you found"`
260+
261+
Each of these needs judgment: deciding what counts as a pattern, whether a reproduction succeeded, what is worth reporting. A task whose output could be written as a fixed template right now belongs in a custom script instead (see *Custom Script Example (No LLM)* below).
260262

261263
#### Cron Schedule
262264

@@ -296,24 +298,105 @@ curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
296298
}'
297299
```
298300

299-
**Weekly cleanup:**
301+
**Weekly dependency review:**
300302
```bash
301303
curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
302304
-H "Authorization: Bearer ${OPENHANDS_API_KEY}" \
303305
-H "Content-Type: application/json" \
304306
-d '{
305-
"name": "Weekly Cleanup",
306-
"prompt": "Clean up temporary files older than 7 days and send a summary of what was removed",
307+
"name": "Weekly Dependency Review",
308+
"prompt": "Check the repository dependencies for new releases. Read each changelog, open one PR with the upgrades that look safe, and list any release with breaking changes in the PR description so a human can decide on it",
307309
"trigger": {"type": "cron", "schedule": "0 2 * * 0", "timezone": "UTC"},
308-
"timeout": 300
310+
"timeout": 1800
309311
}'
310312
```
311313

314+
### Custom Script Example (No LLM)
315+
316+
The same shape as the prompt preset, for a task that needs no reasoning: check a URL on a schedule and post to Slack only when it fails. Every run takes a moment and costs no tokens. This is the complete flow; `get_secret` and `fire_callback` are the helpers from *No-LLM Script Helpers* above.
317+
318+
```bash
319+
mkdir healthcheck && cd healthcheck
320+
321+
cat > main.py << 'PYEOF'
322+
"""Hourly health check: alert Slack on a non-200 response. No LLM, no SDK."""
323+
import json, os, sys, urllib.request
324+
325+
URL = "https://api.example.com/health"
326+
CHANNEL = "C12345678"
327+
328+
def get_secret(name):
329+
url = os.environ.get("AGENT_SERVER_URL", "").rstrip("/")
330+
key = os.environ.get("SESSION_API_KEY") or os.environ.get("OH_SESSION_API_KEYS_0", "")
331+
if url: # local Agent Canvas
332+
url = f"{url}/api/settings/secrets/{name}"
333+
else: # OpenHands Cloud / Enterprise
334+
api = os.environ["OPENHANDS_CLOUD_API_URL"].rstrip("/")
335+
url = f"{api}/api/v1/sandboxes/{os.environ['SANDBOX_ID']}/settings/secrets/{name}"
336+
req = urllib.request.Request(url,
337+
headers={"X-Session-API-Key": key})
338+
with urllib.request.urlopen(req) as r:
339+
return r.read().decode().strip()
340+
341+
def fire_callback(status="COMPLETED", error=None):
342+
url = os.environ.get("AUTOMATION_CALLBACK_URL", "")
343+
if not url: return
344+
body = {"status": status, "run_id": os.environ.get("AUTOMATION_RUN_ID", "")}
345+
if error: body["error"] = error
346+
req = urllib.request.Request(url, data=json.dumps(body).encode(), headers={
347+
"Content-Type": "application/json",
348+
"Authorization": f"Bearer {os.environ.get('AUTOMATION_CALLBACK_API_KEY', '')}",
349+
})
350+
try: urllib.request.urlopen(req)
351+
except Exception as e: print(f"Callback error: {e}")
352+
353+
def slack(text):
354+
token = get_secret("SLACK_BOT_TOKEN")
355+
req = urllib.request.Request("https://slack.com/api/chat.postMessage",
356+
data=json.dumps({"channel": CHANNEL, "text": text}).encode(),
357+
headers={"Content-Type": "application/json", "Authorization": f"Bearer {token}"})
358+
urllib.request.urlopen(req)
359+
360+
try:
361+
try:
362+
status = urllib.request.urlopen(URL, timeout=10).status
363+
except Exception as e:
364+
status = f"error: {e}"
365+
if status != 200:
366+
slack(f":rotating_light: {URL} returned {status}")
367+
print(f"{URL} -> {status}")
368+
fire_callback("COMPLETED")
369+
except Exception as e:
370+
print(f"ERROR: {e}", file=sys.stderr)
371+
fire_callback("FAILED", str(e))
372+
sys.exit(1)
373+
PYEOF
374+
375+
tar -czf ../healthcheck.tar.gz .
376+
377+
TARBALL_PATH=$(curl -s -X POST "${OPENHANDS_HOST}/api/automation/v1/uploads?name=healthcheck" \
378+
-H "Authorization: Bearer ${OPENHANDS_API_KEY}" \
379+
-H "Content-Type: application/gzip" \
380+
--data-binary @../healthcheck.tar.gz | jq -r '.tarball_path')
381+
382+
curl -X POST "${OPENHANDS_HOST}/api/automation/v1" \
383+
-H "Authorization: Bearer ${OPENHANDS_API_KEY}" \
384+
-H "Content-Type: application/json" \
385+
-d "{
386+
\"name\": \"Hourly Health Check\",
387+
\"trigger\": {\"type\": \"cron\", \"schedule\": \"0 * * * *\", \"timezone\": \"UTC\"},
388+
\"tarball_path\": \"$TARBALL_PATH\",
389+
\"entrypoint\": \"python3 main.py\"
390+
}"
391+
```
392+
393+
For state between runs (last alert time, last seen event), use the KV store helpers in `references/custom-automation.md#state-persistence-kv-store`.
394+
312395
---
313396

314397
## Polling as a Webhook Alternative
315398

316-
When the deployment cannot receive inbound webhook traffic (see rule 5), use a cron-triggered automation that calls the external service’s API on a schedule to check for new events.
399+
When the deployment cannot receive inbound webhook traffic (see rule 4), use a cron-triggered automation that calls the external service’s API on a schedule to check for new events.
317400

318401
### Polling vs. Webhooks at a Glance
319402

@@ -442,15 +525,15 @@ curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
442525
}'
443526
```
444527

445-
#### GitHub: Run tests on push to main
528+
#### GitHub: Flag risky changes pushed to main
446529

447530
```bash
448531
curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
449532
-H "Authorization: Bearer ${OPENHANDS_API_KEY}" \
450533
-H "Content-Type: application/json" \
451534
-d '{
452-
"name": "Run Tests on Main",
453-
"prompt": "Clone the repository and run the test suite. Report any failures.",
535+
"name": "Risky Change Watch",
536+
"prompt": "Review the diff of this push. If it touches authentication, billing, data migrations, or anything that changes behavior for existing users, open an issue that explains the risk in plain terms and mentions the code owners. Otherwise do nothing.",
454537
"trigger": {
455538
"type": "event",
456539
"source": "github",
@@ -942,7 +1025,7 @@ The **prompt preset** is the right default for genuinely agent-shaped work — a
9421025

9431026
**Watch for deterministic, high-frequency patterns.** Requests like "send a daily standup reminder", "ping a healthcheck URL every minute", "post a random quote every 5 minutes", or "rotate a fact-of-the-day message" do not need an LLM. Surface this to the user explicitly with a rough cost framing (e.g. "this schedule will invoke your LLM ~288 times/day") before defaulting to a preset. As a rule of thumb, any cron tighter than hourly deserves a deliberate "should this really be agent-driven?" check.
9441027

945-
**When neither preset is the right fit** (deterministic task, custom Python dependencies, non-Python entrypoint, multi-file project structure, direct SDK lifecycle control), explain the options to the user and let them decide. Do not attempt custom automation without explicit user agreement. If they choose the custom route, refer to `references/custom-automation.md`.
1028+
**When neither preset is the right fit** (deterministic task, custom Python dependencies, non-Python entrypoint, multi-file project structure, direct SDK lifecycle control), explain the options and let the user decide, exactly as for any other automation. If they choose the custom route, follow `references/custom-automation.md` and the *Custom Script Example (No LLM)* above.
9461029

9471030
## Security Considerations
9481031

@@ -955,7 +1038,6 @@ See `references/security.md` — also covers narrowing triggers and sender-level
9551038

9561039
## Reference Files
9571040

958-
- **`references/custom-automation.md`** — Detailed guide for custom automations: tarball uploads, code structure (SDK and no-LLM), state persistence via the KV store, environment variables, validation rules, and complete examples. Consult this whenever you need to evaluate or recommend the custom path (including for deterministic / cost-sensitive tasks per rule 0). Only *implement* a custom automation after the user agrees to that path.
1041+
- **`references/custom-automation.md`** — Detailed guide for custom automations: tarball uploads, code structure (SDK and no-LLM), state persistence via the KV store, environment variables, validation rules, and complete examples. Consult this whenever you need to evaluate, recommend, or build the custom path (including for deterministic / cost-sensitive tasks per rule 0).
9591042
- **`references/ab-testing.md`** — A/B testing for plugin automations: defining variants with weights, experiment configuration, variant selection logic, observability via conversation tags, and complete examples. Consult this when a user wants to compare plugin versions or configurations.
960-
- **`references/security.md`** — Trust boundaries: untrusted content vs. verified sender, least-privilege secrets, trigger scoping, sender authorization, pre-deploy verification. Consult whenever an automation handles external input or forwards secrets to a spawned conversation.
9611043
- **`references/security.md`** — Trust boundaries for automations: untrusted event content vs. verified sender, least-privilege secret scoping for spawned conversations, narrowing triggers, sender-level authorization, and verifying a script actually runs before deploying it. Consult this whenever an automation handles external/untrusted input (GitHub issues/PRs, Slack messages, any public-facing webhook) or forwards secrets to a spawned conversation.

‎skills/openhands-automation/references/custom-automation.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,12 @@
11
# Custom Automation Reference
22

3-
> **⚠️ Do NOT use this reference unless the user has explicitly requested a custom automation.** Always use the preset/prompt endpoint from the main SKILL.md first. If the preset approach cannot satisfy the requirement, explain the options to the user and let them decide.
4-
53
This file contains detailed documentation for creating custom automations with user-provided code, uploads, and entrypoints.
64

7-
**When to use custom automation (only if the user explicitly chooses this):**
5+
**When to use a custom automation:**
6+
- The task is deterministic and needs no LLM (fixed data, scheduled HTTP calls, templated messages), especially on a frequent schedule
87
- Full control over the automation code structure is needed
98
- Custom dependencies or a specific runtime are required
10-
- The user has confirmed that the prompt preset does not meet their requirements
9+
- The user picked this path after seeing the options side by side (see the agent behavior rules in `SKILL.md`)
1110

1211
## Table of Contents
1312

0 commit comments

Comments
 (0)