Skip to content

Latest commit

 

History

History
588 lines (439 loc) · 33.7 KB

AUTOMATIONS.md

File metadata and controls

588 lines (439 loc) · 33.7 KB

Automations

Automations let you run coding agents either on a recurring schedule or when an external event arrives. Define the repository configuration, model, and instructions once, then Open-Inspect starts a new session whenever the trigger fires.

Trigger types:

Trigger Type Description Availability
Schedule Run on a cron schedule Available
Inbound Webhook Trigger from any system with an HTTP POST Available
Sentry Alert Trigger from a Sentry Custom Integration Available
Slack Message Trigger on messages in watched channels Available
GitHub Event Trigger on GitHub activity Available (opt-in)
Linear Event Trigger on Linear activity Planned

Common use cases include nightly dependency updates, reacting to deploy or incident events, triaging new Sentry issues, and recurring report generation.


Creating an Automation

Navigate to Automations in the sidebar, then click Create Automation.

Creation requires both automations.create and sessions.create, plus permission to use the selected targets. The creator becomes the initial executor. The web creation and template entry points also require both permissions; automation creation permission alone is insufficient.

Start by choosing a Trigger Type. The rest of the form adjusts based on that choice.

Choose workspace ownership or an owning team; team pages also expose an Automations tab. The creator must belong to the selected active team, and that team's grants must cover the selected repositories, including repositories resolved from environment targets. Newly created teams have no repository grants. When requireTeamOnCreate is enabled, new automation definitions must have a team. Existing workspace automations are not migrated or blocked from running by that setting alone.

Ownership is fixed at creation: an automation cannot move between teams or between a team and the workspace. Its executor, initially the creator, is a separate user identity that can be reassigned without changing ownership.

Required Fields

Field Description
Trigger Type How the automation starts: schedule, inbound webhook, Sentry alert, GitHub event, or Slack message.
Name A short label for the automation (max 200 characters). Appears in the automations list and in session titles prefixed with [Auto].
Repository Configuration Pick no repository, one repository, or (for scheduled automations) up to 10 repositories. Selecting several fans each firing out into one session per repository. Only repositories installed on the GitHub App are available.
Instructions The prompt sent to the coding agent each time the automation fires (max 15,000 characters). Write this as you would a normal session prompt and reference the trigger context when useful. Multi-repo automations share one prompt across repos.

Optional Fields

Field Description
Branch The base branch for each session (shown when exactly one repository is selected). Multi-repo selections use each repository's default branch.
Model The AI model to use. Defaults to the system default model.
Reasoning Optional reasoning level for models that support it.
Conditions Optional trigger filters for event-driven automations such as inbound webhooks, GitHub events, and Sentry alerts.

Trigger-Specific Fields

Trigger Type Additional Fields
Schedule Schedule and Timezone
Inbound Webhook No extra required fields
Sentry Alert Event Type and Sentry Client Secret
GitHub Event Event Type and optional Conditions
Slack Message Conditions (a Slack Channel condition is required; a Message Text condition is optional)

For non-schedule automations, schedule fields are not used.


Repository Context

Automations can run with zero, one, or several repositories:

  • No repository: no repository is cloned. The agent still starts a normal session and can use configured tools such as MCP servers, but repo workspace actions like opening pull requests require repository context.
  • One repository: clone the configured repository and branch for each run.
  • Multiple repositories (scheduled automations only, up to 10): each firing starts one independent session per repository, concurrently. Every session clones its own repository on that repository's default branch, runs the same instructions, and opens its own pull request if needed. Event-driven triggers stay limited to one repository for now — an event arrives scoped to a repository, and fanning it out to unrelated repositories has no defined semantics yet.

You can change the repository selection at any time, including while runs are in flight: history always shows the repositories a run actually used, in-flight runs are unaffected, and the next firing uses the new selection.


Inbound Webhooks

Use Inbound Webhook when you want any external system to trigger an automation with a JSON payload. This is the most flexible event-driven option and works well for internal tools, deployments, monitoring systems, scheduled jobs, and custom integrations.

How It Works

  1. Create an automation with Trigger Type = Inbound Webhook.
  2. Copy the generated webhook URL and API key shown after creation.
  3. Send an authenticated HTTP POST request with a JSON body.
  4. Open-Inspect prepends a webhook context block to your automation instructions and starts a new session if the request matches the automation's conditions. A request with a sessionKey can continue the session an earlier request started instead (see Continuing a Session).

Setup Notes

  • The webhook URL and API key are shown after the automation is created.
  • The API key is only shown once, so store it when you create the automation.
  • The automation detail page shows the webhook path for reference.
  • Webhook automations do not use schedule or timezone settings.

Request Requirements

Inbound webhooks must meet all of the following requirements:

Requirement Value
Method POST
Content-Type application/json
Authentication Authorization: Bearer <api-key>
Maximum payload size 64 KB

Any valid JSON body is accepted.

Example:

curl -X POST "https://<your-worker-url>/webhooks/automation/<automation-id>" \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"event":"deploy.failed","service":"api","environment":"prod"}'

What the Agent Receives

For each accepted request, Open-Inspect prepends a context block to your instructions that includes:

  • The fact that the automation was triggered by an inbound webhook
  • The time the webhook was received
  • The JSON payload, truncated if necessary
  • A warning to treat the payload as untrusted input data, not as instructions

Write your automation instructions assuming the agent will read both your saved prompt and the incoming payload together.

Filtering with Conditions

Webhook automations support one condition type: JSONPath Filter.

Use conditions when you want the automation to run only for specific payload shapes or values, such as:

  • Only production events
  • Only deploy failures
  • Only payloads that include a specific field

Each JSONPath condition contains one or more filters, and all filters must match for the automation to run.

Supported comparisons:

Comparison Meaning
eq Equal to
neq Not equal to
gt Greater than
gte Greater than or equal to
lt Less than
lte Less than or equal to
contains String contains substring
exists Field is present

Supported path syntax is limited to simple dot notation such as $.event.type or $.deployment.environment.

Not supported:

  • Array indexing
  • Recursive descent
  • Full JSONPath expressions

Example filters:

Goal Filter
Run only for production $.environment eq "production"
Run only for failed deploys $.status eq "failed"
Run only when a field is present $.pull_request.number exists

Idempotency and Duplicate Deliveries

If your sender may retry the same event, include an idempotencyKey field in the JSON body.

When present, Open-Inspect uses that value to deduplicate repeated deliveries of the same logical event. Re-sending the same idempotencyKey will not create duplicate runs.

The idempotencyKey remains in the stored webhook body, but it is omitted from the context block shown to the agent.

If you do not provide an idempotencyKey, each webhook delivery gets its own concurrency key. That means separate deliveries for the same automation can run at the same time.

Continuing a Session

Include a sessionKey string in the JSON body to send several deliveries to one session. The first delivery with a key starts a run as usual. A later delivery with the same key is queued as a follow-up prompt on that run's session instead of starting a run, whether the run is still starting, working, or finished, as long as the session was active within the last 7 days. The follow-up carries the payload's context block without the saved instructions, and it is sent as the automation's executor, who must be able to collaborate on the session; without that access the delivery is skipped.

Conditions still apply to follow-ups. The sessionKey stays in the stored body, is removed from the context block, and must be a non-empty string of at most 256 characters, or the request is rejected with 400. Pair it with an idempotencyKey: the session recognizes a retried follow-up and the retry counts as skipped. A follow-up that arrives while the first run is still starting is held and queued right after the run's own first prompt. If the session will not take a follow-up (archived, cancelled, out of budget, needing sandbox recovery, or unreachable), the request fails with 409, 422, 429, or 503 instead of starting over in a new session. After 7 days without activity in the session, the next delivery starts a new run.

Responses

Successful requests return JSON in this shape:

{ "ok": true, "triggered": 1, "skipped": 0, "steered": 0, "invocationId": "3f2a…" }

triggered is the number of automation runs started, and steered the number of follow-ups routed into an existing session by a sessionKey. invocationId identifies the firing this request belongs to (for a repeated idempotencyKey, the original firing, and for a follow-up, the firing of the run whose session took it), or is null when nothing was recorded. Read its status with the same API key at GET /webhooks/automation/<automation-id>/invocations/<invocation-id>, which returns { invocationId, status, runs: [{ id, status, sessionId }] } and never session content.

skipped includes runs not started because of duplicate delivery, concurrency protection, or runtime authorization denial. Authorization denial does not pause an event-driven automation or record a run-history invocation; restore the required executor/team/target access before sending another delivery.

Error Responses

Status Meaning
400 Invalid JSON body, or an invalid sessionKey
401 Missing or invalid API key
404 Automation not found or not a webhook automation
409 The sessionKey session cannot take prompts: SESSION_NOT_PROMPTABLE, BUDGET_EXHAUSTED, SANDBOX_RECOVERY_REQUIRED
413 Payload too large
415 Content-Type was not application/json
422 The sessionKey session rejected the follow-up for another reason; a retry is rejected the same way
429 The sessionKey session's queue is full (PROMPT_QUEUE_FULL)
503 The sessionKey session could not be reached (SESSION_UNAVAILABLE); safe to retry with the same idempotencyKey

A refused follow-up returns { "error", "code", "invocationId" }, where invocationId is the run whose session refused it, and starts nothing.


GitHub Event Triggers

A GitHub Event automation starts a session when a supported webhook event arrives for its repository. Pick one repository and one event type. The GitHub App must subscribe to that event and have its required repository permission. Set enable_github_bot = true, then complete the GitHub bot setup to deploy the webhook worker and configure event delivery.

Workflow Run Completed handles completed GitHub Actions workflow runs. It requires the GitHub App's read-only Actions permission and the Workflow runs event subscription. Use these conditions to restrict it:

  • Workflow Name matches the workflow name exactly.
  • Conclusion matches success, failure, neutral, cancelled, timed_out, action_required, stale, or skipped.

All configured conditions must match. Open-Inspect includes the workflow name, conclusion, and run ID in the untrusted event context sent to the agent. When GitHub supplies them, the context also includes the branch, commit SHA, workflow path, and run URL. Open-Inspect includes the attempt number in its deduplication key, so it can admit each GitHub rerun once.

Draft matches on the pull request's draft state. Set it to Not draft to skip draft pull requests before Open-Inspect starts a session. Pair it with a PR Ready for Review automation, which fires when the author marks a draft pull request ready, so the pull request still gets a run when it leaves draft without a new push.

Conclusion also works for completed check suites. Check Conclusion remains available for existing check-suite automations.

Each event type offers only the conditions its payload can answer, so the choices change with the event you pick — pull requests offer branch, target branch, label, and actor; issues offer label and actor; comments offer actor. PR opened and PR updated also offer Draft. Changing the event type removes conditions the new one cannot answer, and the form says which. GitHub webhook payloads carry no file list, so path-pattern filtering is not offered for any GitHub event.


Slack Message Triggers

A Slack Message automation starts a session when someone posts a matching message in a watched Slack channel. Unlike @mention sessions (which are explicit, interactive requests), these triggers fire on ambient channel messages that match the conditions you define.

The Slack app must be configured to deliver channel messages. See the Slack integration guide for setup and the threat model.

Conditions

A Slack automation must define at least a Slack Channel condition; the rest are optional filters.

Slack Message automations ingest text only. A message posted with an attachment (a file_share message) is matched on its text like any other, but the attachment is not added to the prompt, so an image-only message with no text triggers nothing. Use an interactive bot DM or @mention for image input.

When the triggering message is a reply, the thread it belongs to is passed to the agent alongside it — up to 20 earlier messages total, preserving the thread's opening message alongside the most recent replies — so a reply is read in context instead of on its own. The thread is fetched only after a run is admitted, so unmatched messages, follow-ups that steer an existing session, and skipped or duplicate firings cost nothing. Conditions still match against the triggering message's text only, never the thread.

  • Slack Channel (required) — the channels to watch. Pick channels by name in the web form; channel IDs (for example C0123ABCD) also work as a fallback when channel listing is unavailable. Only messages in these channels are considered, and the bot must be a member of each.
  • Message Text (optional) — filter on the message text. Without it, every message in the watched channels triggers the automation. Pick a mode:
    • contains — the message contains the substring (optionally case-insensitive).
    • exact — the message equals the text.
    • regex — the message matches a regular expression. Patterns are capped in length and limited to the i and m flags; an invalid pattern is rejected when you save.
  • Slack User (optional) — include or exclude specific Slack user IDs (an allowlist is the recommended way to limit who can trigger a run).

A message runs the automation only when every condition passes. The bot-mention token is stripped before matching, and messages that @mention the bot are handled by the interactive @mention flow instead — they never double-fire as triggers.

Run feedback

A triggering message is marked with the 👀 reaction while its run is in flight. When the run finishes, the agent's final response is posted as a reply in that message's thread — with links to any pull requests it opened and to the full web session — and the reaction is cleared. A failed run posts a short failure notice in the thread instead.

A run can also decline to reply: if the agent's entire final message is NO_REPLY (or empty), nothing is posted and only the reaction is cleared. Because every reply in a watched thread wakes the automation (see below), an automation that watches a busy channel will otherwise answer messages that need nothing from it. Mention the sentinel in the automation's instructions — for example "if the message needs nothing from you, reply with exactly NO_REPLY and nothing else" — since the agent will answer everything it is woken for unless told otherwise. A run that opened a pull request or produced other artifacts always posts.

Every reply in a thread continues the same session — while the run starts, during it, and after it finishes — until the session has been idle for 7 days, exactly like replying in an @mention thread. The reply is enqueued as a follow-up turn on that session (re-spawning it from a snapshot if it had gone idle), and the agent posts its response in-thread when the turn finishes. A reply that arrives while the run is still starting is held and queued right after the run's own first prompt. A follow-up does not need to match the trigger condition — conditions gate new runs, not replies that continue an existing thread. If a reply races the very first trigger before its run exists, it falls back to an ephemeral "a run is already active" notice (reason concurrent_run_active); a reply after 7 idle days starts a fresh run.

These automation follow-ups also forward text only. Attachments on a thread reply are not delivered to the session.


Schedule Options

The schedule picker offers four presets and a custom mode:

Preset Description Controls
Every hour Runs once per hour at the top of the hour None
Daily Runs once per day at a chosen time Hour picker (12-hour AM/PM)
Weekly Runs once per week on a chosen day and time Day-of-week + hour picker
Monthly Runs once per month on a chosen day and time Day-of-month (1st–28th) + hour picker
Custom Any valid 5-field cron expression Text input with live validation

The picker shows a live preview of the next scheduled run time below the controls.

Custom Cron Expressions

Custom expressions must use the standard 5-field format:

minute  hour  day-of-month  month  day-of-week

Examples:

Expression Meaning
*/15 * * * * Every 15 minutes
0 9 * * * Daily at 9:00 AM
30 14 * * 1-5 Weekdays at 2:30 PM
0 0 1 * * First of every month at midnight

Note: The minimum schedule interval is 15 minutes. Expressions that fire more frequently (e.g., */5 * * * *) are rejected.

Note: Six-field expressions (with seconds) are not supported.


Managing Automations

Workspace automations are readable with automations.read; team automations additionally require owning-team membership or a workspace Owner/Administrator role. Executors and owning-team leads can manage and trigger eligible automations with the corresponding own permissions; any permissions allow those actions across eligible automations. Built-in Owners and Administrators have the latter; Viewers cannot manage or trigger. These resource checks apply even in off or shadow session enforcement mode.

Scheduled and event runs use the executor's authority. Each run checks that the execution user is active, can create sessions and use its targets, and, for team automations, is a current member of the active owning team. Current team grants must cover all repositories being launched. Generated sessions inherit the automation's owning team and that team's default visibility; workspace runs remain workspace-owned and workspace-visible. Environment targets must have the same ownership as the automation; unlike the team session picker, a team automation cannot select a workspace-owned environment.

If a scheduled run is denied execution authorization, it is recorded as Skipped and the automation is paused immediately, clearing its next run time without adding a failure strike. Restore the required access or reassign the executor, then click Resume. Restoring access or reassigning alone does not resume the schedule. Event-driven authorization denials skip the event without pausing the automation.

Executor Reassignment

A team lead or workspace Owner/Administrator with automation management access can use PATCH /automations/:id with {"userId":"<canonical-user-id>"} to change the executor. Being the executor alone does not permit reassignment. The replacement must be active, able to launch the stored targets, and a member of the active owning team for a team automation. The change is audited as automation.executor_changed. Reassignment allows recovery when the old executor loses access without recreating the definition; it does not move ownership or rewrite existing sessions.

Pause and Resume

Pausing an automation stops it from firing. Scheduled automations will not run on their cron, and event-driven automations will ignore incoming events until resumed. You can pause from the automations list or the detail page.

Resuming reactivates the automation. Scheduled automations calculate the next run time from the current moment. Event-driven automations resume listening immediately. Resuming also resets the consecutive failure counter (see Auto-Pause below).

Trigger Now

Click Trigger Now to fire a one-off run immediately across the automation's full repository selection. For scheduled automations, this does not affect the next scheduled run time. Manual triggers follow the same concurrency rules as all other runs: if a run is already active, the trigger is rejected. Trigger Now also works while the automation is paused, so you can verify a fix before resuming.

Manual runs use the requester's authority and linked source-control credentials, not the stored executor's. In addition to trigger permission, the requester must pass runtime session/target checks and be a current member of the active owning team for a team automation, even if they are a workspace Owner or Administrator.

Edit

You can change an automation's name, repository selection, branch, model, and instructions at any time — including adding or removing repositories while a run is active (in-flight sessions are unaffected; the next firing uses the new selection). For scheduled automations, you can also change the schedule and timezone. Repository-scoped triggers require exactly one repository; other trigger types can be changed to No repository.

If you update the schedule or timezone, the next run time is recalculated automatically.

Delete

Deleting an automation stops all future runs and removes it from the list. Existing run history and any sessions it created are preserved.


Run History

Each automation's detail page shows a chronological list of runs — one row per recorded invocation — with status, duration, and links to the underlying sessions. Generic event execution-authorization denials do not create invocation records and do not appear in this history.

GitHub repository-grant denials are an exception. Matching events without an owning-team grant for the repository can record a sessionless Unauthorized invocation with reason repo_not_granted before executor authorization, including when the executor cannot be resolved. A missing executor alone does not create that record. These denials increment the event response's skipped count, do not pause the automation, and neither add a failure strike nor reset existing failures. See packages/control-plane/src/automation/github-event-admission.ts and packages/control-plane/src/db/github-automation-store.ts for this pre-admission path.

A single-repository firing renders as a flat row, exactly as before. A multi-repository firing renders as one expandable row summarizing its repositories (for example "10 repositories — 8 completed, 1 failed, 1 running"); expanding it shows each repository with its own status, failure reason, and session link.

Run Statuses

Status Meaning
Starting A session is being created for this run.
Running At least one session is actively executing.
Completed Every session finished successfully.
Failed Every session encountered an error. The failure reason is shown on the run.
Partial failure A multi-repository run where some repositories completed and some failed.
Skipped A previous run was still active, or a scheduled firing was denied execution authorization.
Unauthorized GitHub event admission found no owning-team grant for the repository (repo_not_granted); no session was launched.

When authorized, click View session to open the session with its output and artifacts. Automation read access is not session read access: history redacts session IDs, titles, and artifact summaries for sessions the viewer cannot read. Lists do not expose private sessions solely through an Owner's break-glass privilege; a qualifying single-run read audits that access.


Automation Status

Automations display one of three statuses:

Status Meaning
Enabled Running normally and ready to respond to its trigger.
Degraded Enabled but has recent consecutive failures. The failure count is shown on the badge.
Paused Not firing. Manually paused, auto-paused after repeated failures, or immediately paused after scheduled execution authorization denial.

Concurrent Runs

For scheduled and manual triggers, only one run per automation can be active at a time. If one of those triggers fires while a previous run is still in progress, the new run is recorded as Skipped with reason "concurrent run active".

Event-driven automations use concurrency keys instead. For inbound webhooks, retries with the same idempotencyKey are treated as the same event, but separate deliveries without a shared idempotencyKey can overlap. Deliveries that share a sessionKey are keyed by it: a later one continues the first run's session rather than overlapping it.

Slack Message triggers key concurrency by thread. Replies in a thread are not skipped — until the session has been idle for 7 days they continue the same session (while the run starts, during it, and after it finishes), routed to that session as follow-up prompts (see the Run feedback note under Slack Message Triggers).

This prevents overlapping sessions from interfering with each other on the same repository.


Auto-Pause

If an automation fails 3 consecutive times, it is automatically paused to prevent runaway failures. The status changes to Paused and no further runs will start until you resume it.

To re-enable the automation, click Resume. This resets the failure counter. Scheduled automations also compute their next run at that point.

Consecutive failures are tracked across both scheduled and manually triggered runs. For multi-repository runs, a firing with any failed repository counts as one failure — a weekly sweep that fails the same repository every week is still broken. The counter resets only when a firing finishes with every repository completed; partial failures never reset it. Skipped runs count neither way. Auto-pause stops future firings but never cancels repository sessions that already started.

Runs that time out (sessions running longer than 90 minutes) also count as failures toward the auto-pause threshold.


Limits

Limit Value
Automation name length 200 characters
Instructions length 15,000 characters
Repositories per automation 10 (multi-select on schedule triggers)
Minimum schedule interval 15 minutes
Webhook payload size 64 KB
Concurrent runs per automation 1 for scheduled/manual triggers only
Consecutive failures before auto-pause 3
Run execution timeout 90 minutes