You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3bfeaf0
Browse filesBrowse the repository at this point in the historyBrowse files
fix(openhands-automation): stop steering agents away from deterministic scripts (#686)
* fix(openhands-automation): stop steering agents away from deterministic scripts
The skill's rule 0 tells agents to prefer a no-LLM custom script for
deterministic tasks, but the rest of the text pulled the other way:
- Prompt-preset examples modelled deterministic tasks (hourly health
check, run tests on push, weekly temp-file cleanup). Replace them with
tasks that need judgment: error triage, bug reproduction, dependency
review, flagging risky pushes.
- references/custom-automation.md opened with "Do NOT use this reference
unless the user has explicitly requested a custom automation", which
contradicted rule 0 and the Reference Files section. Remove it and list
deterministic tasks as a first-class reason to use the custom path.
- Add a complete no-LLM worked example (hourly health check with Slack
alert) next to the prompt-preset examples, so both paths are equally
concrete.
- Make consent symmetric: present the options side by side and build the
one the user picks. The "ready to deploy?" confirmation applies to
every path; custom scripts no longer need an extra permission step.
Also drops a duplicated references/security.md bullet and regenerates
the skills catalog.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(openhands-automation): fix rule cross-reference and rebalance README
Address review feedback on #686:
- SKILL.md: the polling section pointed at rule 5, which became rule 4
when rules 2-4 were collapsed.
- README.md: stop presenting the prompt preset as the recommended default
and custom automations as advanced-only; list custom scripts alongside
prompt-based creation and point to the no-LLM worked example.
- Regenerate skills/index.js.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(openhands-automation): make the no-LLM example's get_secret work on Cloud
The worked example added by this PR carried the pre-#709 get_secret, which
only reads from the agent server. Match the helper now on main so the
example also fetches secrets from the OpenHands API on Cloud and
Enterprise runs. Regenerate skills/index.js.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* chore: remove tracker file committed by mistake
b588ba6 picked up .beads/issues.jsonl through the local bd pre-commit
hook, which stages that file on every commit. It is local task-tracker
data and does not belong in this repository.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: skills/openhands-automation/README.md
+6-4Lines changed: 6 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,20 +16,22 @@ This skill is activated by keywords:
16
16
17
17
## Features
18
18
19
-
-**Prompt-based creation**: Create automations from a natural language prompt (recommended)
19
+
-**Prompt-based creation**: Create automations from a natural language prompt, for tasks that benefit from agent reasoning
20
+
-**Custom scripts**: Run your own code, with or without an LLM — the cheaper, more reliable fit for deterministic tasks (fixed data, scheduled HTTP calls, templated messages), and for custom dependencies or full control (see [references/custom-automation.md](references/custom-automation.md))
-**Custom automations**: For advanced users who need full control (see [references/custom-automation.md](references/custom-automation.md))
26
26
27
27
## API Base URL
28
28
29
29
Use an explicitly provided host first. When a local Agent Canvas automation server is running and no host is provided, use `http://localhost:8001/api/automation/v1`. Otherwise use the cloud default: `https://app.all-hands.dev/api/automation/v1`.
30
30
31
31
## Quick Start
32
32
33
+
The examples below use the prompt preset, which suits tasks that need reasoning or judgment. For a deterministic task that needs no LLM, use a custom script instead — see the *Custom Script Example (No LLM)* in [SKILL.md](SKILL.md).
34
+
33
35
### Cron-Triggered Automation
34
36
35
37
```bash
@@ -81,9 +83,9 @@ curl -X POST "https://app.all-hands.dev/api/automation/v1/preset/prompt" \
81
83
}'
82
84
```
83
85
84
-
The service handles SDK code generation, tarball packaging, upload, and automation creation automatically.
86
+
For preset automations, the service handles SDK code generation, tarball packaging, upload, and automation creation automatically.
85
87
86
88
## See Also
87
89
88
90
-[SKILL.md](SKILL.md) — Full API reference, agent behavior rules, event keys, filters, and examples
89
-
-[references/custom-automation.md](references/custom-automation.md) — Reference for custom automations with user-provided SDK scripts
91
+
-[references/custom-automation.md](references/custom-automation.md) — Reference for custom automations with user-provided scripts, SDK-based or deterministic (no LLM)
@@ -64,14 +64,14 @@ The agent server typically runs inside a **sandbox** (a Docker or Kubernetes con
64
64
> 1.**For LLM-appropriate work, default to preset endpoints.** They handle all SDK boilerplate, tarball packaging, and upload automatically:
65
65
> -**Prompt preset** (`POST /v1/preset/prompt`) — for tasks expressed as a natural language prompt that benefit from agent reasoning
66
66
> -**Plugin preset** (`POST /v1/preset/plugin`) — when plugins with skills, MCP configs, or commands are needed
67
-
> 2.**Do not silently create custom scripts.**Do not generate Python code, `setup.sh` files, or tarball uploads without user consent. But *do* proactively recommend the custom path (per rule 0) when the task is deterministic or high-frequency — surface the option and let the user choose.
68
-
> 3.**If neither preset is the right fit**, do NOT silently fall back to custom automation. Instead, explain the available options to the user:
69
-
> -**Prompt preset**— natural language prompt execution (LLM-driven)
>-**Custom script** — full control over code, with or without LLM; point them to `references/custom-automation.md`
72
-
> - Let the user choose which approach to use.
73
-
> 4.**Only create custom scripts after the user agrees to that path.**Refer to `references/custom-automation.md` for the full reference.
74
-
> 5.**Before suggesting event-triggered (webhook) automations, check whether the deployment is publicly reachable.** Check `RUNTIME_URL`. Webhooks require an internet-accessible URL so that external services (GitHub, Slack, Linear, etc.) can deliver events to the automation service. If `RUNTIME_URL` is unset, empty, or resolves to a local or private address (`localhost`, `127.0.0.1`, `0.0.0.0`, or any RFC 1918 range: `10.x.x.x`, `192.168.x.x`, `172.16–31.x.x`), the service cannot receive inbound webhook traffic from the public internet. In that case:
67
+
> 2.**Present the options, then build the one the user picks.**Whenever rule 0 applies, or neither preset is a clean fit (custom Python dependencies, a non-Python entrypoint, a multi-file project, direct SDK lifecycle control), lay the choices out side by side with a one-line note on cost and reliability for each, and let the user choose:
68
+
> -**Prompt preset** - natural language prompt execution (LLM-driven)
> -**Custom script**- full control over code, with or without an LLM; see `references/custom-automation.md`
71
+
>
72
+
> Do not silently fall back to either side. The "ready to deploy?" confirmation in the creation process applies to every path equally; a custom script needs no extra permission beyond it, and a preset gets no pass on it.
73
+
> 3.**Building a custom script:**follow `references/custom-automation.md` for tarball packaging, validation, and upload. The *Custom Script Example (No LLM)* below shows the whole flow end to end.
74
+
> 4.**Before suggesting event-triggered (webhook) automations, check whether the deployment is publicly reachable.** Check `RUNTIME_URL`. Webhooks require an internet-accessible URL so that external services (GitHub, Slack, Linear, etc.) can deliver events to the automation service. If `RUNTIME_URL` is unset, empty, or resolves to a local or private address (`localhost`, `127.0.0.1`, `0.0.0.0`, or any RFC 1918 range: `10.x.x.x`, `192.168.x.x`, `172.16–31.x.x`), the service cannot receive inbound webhook traffic from the public internet. In that case:
75
75
> -**Recommend a cron-based polling automation instead.** Have the automation run on a schedule and call the external service's API (e.g., the GitHub REST API) to check for new events since the last run.
76
76
> - Explain the limitation clearly to the user: "Because this is a local deployment, external services can't reach the webhook endpoint. I'll set up a polling automation using a cron schedule instead."
77
77
@@ -255,8 +255,10 @@ curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
255
255
Write the prompt as an instruction to an AI agent. The prompt executes inside a sandbox with full tool access (bash, file editing, etc.), the user's configured LLM, stored secrets, and MCP server integrations. Examples:
256
256
257
257
-`"Generate a weekly status report summarizing the team's GitHub activity and post it to Slack"`
258
-
-`"Check the production API health endpoint every hour and alert if it returns non-200"`
259
-
-`"Pull the latest data from our analytics API and update the dashboard spreadsheet"`
258
+
-`"Read yesterday's new error reports in Sentry, group them by likely root cause, and open one GitHub issue per new failure pattern with a proposed fix"`
259
+
-`"Go through support tickets tagged 'bug' since the last run, try to reproduce each against main, and comment on the ticket with what you found"`
260
+
261
+
Each of these needs judgment: deciding what counts as a pattern, whether a reproduction succeeded, what is worth reporting. A task whose output could be written as a fixed template right now belongs in a custom script instead (see *Custom Script Example (No LLM)* below).
260
262
261
263
#### Cron Schedule
262
264
@@ -296,24 +298,105 @@ curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
296
298
}'
297
299
```
298
300
299
-
**Weekly cleanup:**
301
+
**Weekly dependency review:**
300
302
```bash
301
303
curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
302
304
-H "Authorization: Bearer ${OPENHANDS_API_KEY}" \
303
305
-H "Content-Type: application/json" \
304
306
-d '{
305
-
"name": "Weekly Cleanup",
306
-
"prompt": "Clean up temporary files older than 7 days and send a summary of what was removed",
307
+
"name": "Weekly Dependency Review",
308
+
"prompt": "Check the repository dependencies for new releases. Read each changelog, open one PR with the upgrades that look safe, and list any release with breaking changes in the PR description so a human can decide on it",
The same shape as the prompt preset, for a task that needs no reasoning: check a URL on a schedule and post to Slack only when it fails. Every run takes a moment and costs no tokens. This is the complete flow; `get_secret` and `fire_callback` are the helpers from *No-LLM Script Helpers* above.
317
+
318
+
```bash
319
+
mkdir healthcheck &&cd healthcheck
320
+
321
+
cat > main.py << 'PYEOF'
322
+
"""Hourly health check: alert Slack on a non-200 response. No LLM, no SDK."""
For state between runs (last alert time, last seen event), use the KV store helpers in `references/custom-automation.md#state-persistence-kv-store`.
394
+
312
395
---
313
396
314
397
## Polling as a Webhook Alternative
315
398
316
-
When the deployment cannot receive inbound webhook traffic (see rule 5), use a cron-triggered automation that calls the external service’s API on a schedule to check for new events.
399
+
When the deployment cannot receive inbound webhook traffic (see rule 4), use a cron-triggered automation that calls the external service’s API on a schedule to check for new events.
317
400
318
401
### Polling vs. Webhooks at a Glance
319
402
@@ -442,15 +525,15 @@ curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
442
525
}'
443
526
```
444
527
445
-
#### GitHub: Run tests on push to main
528
+
#### GitHub: Flag risky changes pushed to main
446
529
447
530
```bash
448
531
curl -X POST "${OPENHANDS_HOST}/api/automation/v1/preset/prompt" \
449
532
-H "Authorization: Bearer ${OPENHANDS_API_KEY}" \
450
533
-H "Content-Type: application/json" \
451
534
-d '{
452
-
"name": "Run Tests on Main",
453
-
"prompt": "Clone the repository and run the test suite. Report any failures.",
535
+
"name": "Risky Change Watch",
536
+
"prompt": "Review the diff of this push. If it touches authentication, billing, data migrations, or anything that changes behavior for existing users, open an issue that explains the risk in plain terms and mentions the code owners. Otherwise do nothing.",
454
537
"trigger": {
455
538
"type": "event",
456
539
"source": "github",
@@ -942,7 +1025,7 @@ The **prompt preset** is the right default for genuinely agent-shaped work — a
942
1025
943
1026
**Watch for deterministic, high-frequency patterns.** Requests like "send a daily standup reminder", "ping a healthcheck URL every minute", "post a random quote every 5 minutes", or "rotate a fact-of-the-day message" do not need an LLM. Surface this to the user explicitly with a rough cost framing (e.g. "this schedule will invoke your LLM ~288 times/day") before defaulting to a preset. As a rule of thumb, any cron tighter than hourly deserves a deliberate "should this really be agent-driven?" check.
944
1027
945
-
**When neither preset is the right fit** (deterministic task, custom Python dependencies, non-Python entrypoint, multi-file project structure, direct SDK lifecycle control), explain the options to the user and let them decide. Do not attempt custom automation without explicit user agreement. If they choose the custom route, refer to `references/custom-automation.md`.
1028
+
**When neither preset is the right fit** (deterministic task, custom Python dependencies, non-Python entrypoint, multi-file project structure, direct SDK lifecycle control), explain the options and let the user decide, exactly as for any other automation. If they choose the custom route, follow `references/custom-automation.md` and the *Custom Script Example (No LLM)* above.
946
1029
947
1030
## Security Considerations
948
1031
@@ -955,7 +1038,6 @@ See `references/security.md` — also covers narrowing triggers and sender-level
955
1038
956
1039
## Reference Files
957
1040
958
-
-**`references/custom-automation.md`** — Detailed guide for custom automations: tarball uploads, code structure (SDK and no-LLM), state persistence via the KV store, environment variables, validation rules, and complete examples. Consult this whenever you need to evaluateor recommend the custom path (including for deterministic / cost-sensitive tasks per rule 0). Only *implement* a custom automation after the user agrees to that path.
1041
+
-**`references/custom-automation.md`** — Detailed guide for custom automations: tarball uploads, code structure (SDK and no-LLM), state persistence via the KV store, environment variables, validation rules, and complete examples. Consult this whenever you need to evaluate, recommend, or build the custom path (including for deterministic / cost-sensitive tasks per rule 0).
959
1042
-**`references/ab-testing.md`** — A/B testing for plugin automations: defining variants with weights, experiment configuration, variant selection logic, observability via conversation tags, and complete examples. Consult this when a user wants to compare plugin versions or configurations.
960
-
-**`references/security.md`** — Trust boundaries: untrusted content vs. verified sender, least-privilege secrets, trigger scoping, sender authorization, pre-deploy verification. Consult whenever an automation handles external input or forwards secrets to a spawned conversation.
961
1043
-**`references/security.md`** — Trust boundaries for automations: untrusted event content vs. verified sender, least-privilege secret scoping for spawned conversations, narrowing triggers, sender-level authorization, and verifying a script actually runs before deploying it. Consult this whenever an automation handles external/untrusted input (GitHub issues/PRs, Slack messages, any public-facing webhook) or forwards secrets to a spawned conversation.
Copy file name to clipboardExpand all lines: skills/openhands-automation/references/custom-automation.md
+3-4Lines changed: 3 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,13 +1,12 @@
1
1
# Custom Automation Reference
2
2
3
-
> **⚠️ Do NOT use this reference unless the user has explicitly requested a custom automation.** Always use the preset/prompt endpoint from the main SKILL.md first. If the preset approach cannot satisfy the requirement, explain the options to the user and let them decide.
4
-
5
3
This file contains detailed documentation for creating custom automations with user-provided code, uploads, and entrypoints.
6
4
7
-
**When to use custom automation (only if the user explicitly chooses this):**
5
+
**When to use a custom automation:**
6
+
- The task is deterministic and needs no LLM (fixed data, scheduled HTTP calls, templated messages), especially on a frequent schedule
8
7
- Full control over the automation code structure is needed
9
8
- Custom dependencies or a specific runtime are required
10
-
- The user has confirmed that the prompt preset does not meet their requirements
9
+
- The user picked this path after seeing the options side by side (see the agent behavior rules in `SKILL.md`)
0 commit comments