Skip to content

feat(ai): AI-task reliability, robustness & security hardening (27 fixes) - #1241

Draft
ocervell wants to merge 100 commits into
canaryfrom
ai-resiliency
Draft

feat(ai): AI-task reliability, robustness & security hardening (27 fixes)#1241
ocervell wants to merge 100 commits into
canaryfrom
ai-resiliency

Conversation

@ocervell

@ocervell ocervell commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

AI-task reliability / robustness / security hardening — the full remediation of the
2026-06-30 deep review (docs/superpowers/specs/2026-06-30-ai-task-reliability-review.md).

27 / 29 findings fixed, one focused PR per finding, merged here across 7 rounds.
H8 excluded (overlaps feat/sensitive-options); H3 deferred to a design decision (event-driven
human-wait — see the review doc's H3 proposal).

🔴 Critical

🟠 High

🟡 Medium

🔵 Pertinence / cleanup


Every PR added focused tests and proved no regression against a git-stash baseline (guardrail
tests needing shfmt/safecmd fail identically on base — CI is the arbiter). ~20 follow-up
items the agents surfaced are ranked in the review doc's "round-2 candidates" backlog — the
standout being the permission_engine-unset full bypass (candidate Critical).

ocervell and others added 30 commits June 17, 2026 01:13
The api driver POSTs targets, raw command output and the Bearer API key to
addons.api.url. force_ssl (default True) only governed TLS cert verification,
so an operator setting force_ssl:false would ship that data over cleartext
HTTP to a remote host (MITM / key-leak risk).

_make_request now hard-fails any http:// URL whose host is not loopback
(localhost / 127.0.0.1 / ::1), regardless of force_ssl. https:// and
loopback-http stay working; force_ssl still controls cert verification only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed tasks

When a worker is killed mid-task (cgroup OOMKill of the child, or a node
memory-pressure eviction), task_acks_late + task_reject_on_worker_lost
cause the task to be redelivered with the same Celery id. A task that
OOMs on every run would loop forever (OOM -> redeliver -> OOM) and the
surrounding chord/workflow could never complete.

Add task_max_retries (-1 = disabled). run_command counts redeliveries on
the Celery result backend, keyed by the stable Celery id, and once the
cap is exceeded abandons the task: it returns a forwarded result list
with a FAILURE Error appended instead of re-running, so the chord
proceeds and the workflow finishes with a clear 'abandoned after N
retries' error (mirroring the inner memory-limit warning).

The counter is backend-agnostic: it uses the result backend's generic
key/value interface (atomic incr when available e.g. Redis/Memcached,
else a get/set read-modify-write that every KV backend implements -
filesystem, S3, GCS, ...). It is not tied to Redis or to any Secator
data backend (Mongo/Postgres/SQLite). Backends with neither (database/
RPC) degrade safely to 'cap disabled'.

Inert by default (task_max_retries=-1 and task_acks_late=False); enabling
requires task_acks_late + task_reject_on_worker_lost and raising
broker_visibility_timeout above the max task lifetime.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…loss cancel flag

Completes the worker-loss redelivery cap with the review fixes + the L1 delivery flag:

- Off-by-one: the initial delivery no longer counts as a retry. Extracted
  worker_loss_retries_exhausted(delivery_count, max_retries) — redeliveries =
  delivery_count - 1, so task_max_retries=3 allows the initial run + 3
  redeliveries before abandoning (was 2). Unit-tested at the boundary.
- Key expiry: worker-loss counter keys now get a best-effort TTL tied to
  result_expires (via _expire_worker_loss_key) on both the atomic-incr and the
  get/set paths, so they don't accumulate forever.
- L1: add worker_cancel_long_running_tasks_on_connection_loss config flag
  (default False) wired into app.conf, alongside the existing task_acks_late /
  task_reject_on_worker_lost. Enables clean redelivery of a connection-lost
  worker's in-flight tasks. Inert until enabled in deployment.

Note: the terminal-doc no-op guard (a redelivered task whose runner doc is
already done) is deferred — it depends on stable runner identity (on_build, the
L2 PR) to find 'its' doc, and is ineffective without it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The abandon message said 'after N retries' using task_max_retries, which read
oddly at max_retries=0 ('after 0 retries') because a redelivery still occurs
even with 0 retries — the broker redelivers a lost task under acks_late; that
is NOT a re-run of the work. Thread the actual delivery_count through and
report it separately from the cap:

  Task httpx abandoned after 5 delivery attempts (retry cap: 3; worker
  repeatedly lost — likely OOM kill or node eviction).

Reads sensibly for any cap value (incl. 0).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_on_build.py imported secator.hooks.mongodb at module level, which pulls
pymongo — an optional addon not installed in the base CI test env — so the whole
file failed to collect. Guard the import behind ADDONS_ENABLED['mongodb'] and
mark the 4 mongodb-dependent classes with pytest.skipif on the same flag (works
for both the unittest.TestCase and plain pytest classes). The hook-registration
and sqlite tests don't need pymongo and keep running.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… proceeds

On a K8s pod eviction (SIGTERM -> grace -> SIGKILL) the worker running a task
dies. With task_acks_late on the Redis broker, its message is only redelivered
after the broker visibility timeout (hours, on the long-task pool) — stalling the
surrounding chord/workflow that whole time. Unlike a child OOM (where the master
survives and task_reject_on_worker_lost requeues immediately), a whole-pod
eviction has no surviving master, so Redis can only fall back to the visibility
timeout.

Catch the worker_shutting_down signal and raise a flag the running task's monitor
thread polls. On shutdown it stops the task early via the existing stop_process
path, so the task returns its partial results through the normal completion path
and the chord proceeds in seconds instead of waiting for the visibility timeout.

- celery_signals.py: SHUTDOWN_FLAG + worker_shutting_down_handler (wired
  unconditionally); stale flag cleared on worker boot.
- command.py: the monitor thread checks the flag (same self-stop used for
  timeout/memory limits) and caps its poll interval (MONITOR_POLL_SECONDS=5) so
  an eviction is caught well within the pod's terminationGracePeriodSeconds.
- tests: flag lifecycle + a behavioral test (a running command stops early and
  emits the eviction warning).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…late test

The integration suite caught a real leak: SHUTDOWN_FLAG is a machine-global file,
so once any worker fired worker_shutting_down (e.g. between test files) the flag
persisted and every later task whose monitor polled once self-aborted — slow
tasks failed, fast ones that finished before the first poll passed. Prod never
hit it (1 task = 1 pod = fresh /tmp), but any shared/long-lived worker does.

- command.py: clear the flag at monitor start, so it only means "shutdown raised
  *during this run*", not a stale flag from a previous worker/task.
- celery_signals.py (CodeRabbit): ensure the flag's parent dir exists and catch +
  log OSError instead of swallowing all exceptions.
- test_eviction.py (CodeRabbit): isolate SHUTDOWN_FLAG to a per-test temp path;
  add a regression test that a flag set *before* a task starts does not stop it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…ace)

The behavioral test set the flag once after a fixed 2s wait, but the monitor now
clears any pre-existing flag at startup; on a slow CI runner the monitor started
*after* that single set and wiped it, so the task never stopped and the test
failed. Re-raise the flag in a loop until the task stops, so the monitor's poll
sees it once it is running regardless of start timing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
The real-subprocess behavioral test was environment-dependent: in CI the monitor
thread didn't stop the live `sleep` reliably, so the test flaked (and the stale
test passed for the wrong reason). Replace both with deterministic tests that
exercise _monitor_process directly against a bare Command — one asserts it calls
stop_process(exit_ok=True) + emits the eviction Warning when the flag is set,
the other asserts it clears a stale flag at start (the integration-leak
regression). No subprocess, no timing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
The real-file version of the stale-flag test flaked on CI (patched temp path +
bare-__new__ Command), while the real behaviour is already proven green by the
integration suite. Assert that _monitor_process calls clear_shutdown_flag() at
start via a mock instead of inspecting the filesystem — deterministic and
environment-independent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
Add a `status` field to the Vulnerability output type (NEW / ACKNOWLEDGED /
FIXED, default NEW), marked compare=False so dedup identity (name/id/matched_at)
is unchanged. Normalize/validate in __post_init__ (coerce empty/None/unknown to
NEW, uppercase) and display it via _table_fields.

Carry status across re-scans: add `status` to the duplicate_main_copy_fields
defaults of MongodbAddon and SqliteAddon, and in compute_duplicate_updates treat
a status of ''/None/'NEW' as unset (per-field sentinel) so a prior
ACKNOWLEDGED/FIXED carries forward onto a re-found main whose status is still the
default NEW, while never-touched vulns stay NEW. Other fields keep the generic
`not value` emptiness check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
# Conflicts:
#	.github/workflows/publish-canary.yml
Adds the two secator-core gaps for the Workspace AI Assistant
(Mongo-channel chat), repo 1/3:

- `restore_history_from_db(session_id, query_engine, model, encryptor,
  system_prompt)` in `secator/ai/session.py`: rebuilds a `ChatHistory`
  from the workspace `_type:"ai"` docs (queried by session_id, ordered by
  `_timestamp`) — `prompt`→user, `response`→assistant, system prompt set,
  re-encrypted when an encryptor is active. Headless: no local files, no TUI.
- Wire a remote-resume branch in `ai.py:yielder`: when `interactive="remote"`
  and the session has prior `_type:"ai"` docs, restore from Mongo and continue;
  fresh conversations (no docs) start as before. The local CLI
  `replay_session`/`show_session_picker` path is untouched.
- `session_id` now prefers `run_opts.context.session_id` so a respawned task
  finds its prior docs.
- `save_history` (local `history.json`) is skipped on the remote path via a
  `_save_history()` helper — the Mongo docs are the source of truth.
- Query-engine guard: warn when `interactive="remote"` but the resolved query
  backend is not mongodb/api (the web answer channel can't work otherwise).

History-fidelity finding: persisted `_type:"ai"` docs capture only text turns
(prompt/response) plus action *display* records — not the litellm assistant
`tool_calls` messages or their `tool` results. Restore is therefore text-only.
This is valid and sufficient for `mode="chat"` continuation; fabricating
partial tool-call messages would produce a malformed transcript providers
reject, so tool activity is deliberately collapsed. Richer assistant
persistence for `mode="attack"` replay is a documented follow-up.

Tests: `tests/unit/test_ai_session.py` — restore rebuilds equivalent History
(order/roles/system/encryption/empty-docs/search-failure), and the remote-resume
branch picks Mongo restore for prior docs / fresh otherwise / warns on
non-Mongo backend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…nscript

The web UI correlates an AI chat conversation by session_id (across respawns),
but only the resume-prompt and follow_up items set it — the prompt/response/
token_usage/chat_compacted message items did not, so they persisted to Mongo
without session_id and the UI's {_type:"ai", session_id} query returned nothing
(empty transcript despite the task running fine). Wrap yielder to stamp
session_id on every Ai item centrally (self.session_id is set in _init_options
before the first yield).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
ocervell and others added 5 commits July 4, 2026 20:05
… ship shfmt via the ai addon (#1274)

## Problem (hit during CLI testing)

In attack mode every shell command the LLM issues hit the guardrail
shell parser, which printed:
```
[ERR] Missing ai addon: please run "secator install addons ai".
```
…even though the ai addon **was** installed. `_parse_subcommands`
(`guardrails.py`) shells out to `shfmt` via `safecmd`; the misleading
message came from its `ImportError` branch. The env had `litellm` (so
`ADDONS_ENABLED['ai']=True`) but not `safecmd`/`shfmt` — a partial ai
install. Worse, the *other* failure mode (safecmd present, `shfmt`
binary not on PATH) was **silent** (`FileNotFoundError` swallowed by
`except Exception: return []`).

## Fix
1. **Message** — replace the `Error("Missing ai addon")` with a
**one-shot `Warning`** that names the real gap: `"Missing safecmd shell
parser"` (ImportError) or `"Missing shfmt binary"` (FileNotFoundError).
Both fall back to the **non-shfmt path** — an empty sub-command list, so
`_check_action_type` asks the user to approve the whole command (safe,
just coarser). Warn once per process (no per-command spam).
2. **Install** — add `shfmt-py` explicitly to the `ai` extra so `secator
install addons ai` (`pip install secator[ai]`) always ships the `shfmt`
**binary**, not just the `safecmd` Python package (which only pulled it
transitively).

## Tests
`TestShellParserFallback`: safecmd-missing → `Warning` naming `safecmd`,
never `"ai addon"`; shfmt-binary-missing → `Warning` naming `shfmt`;
warn-once across multiple commands. Full AI suite: **no new failures**
vs branch baseline.

Found while testing `ai-testing`; targets `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…chat conversation id (#1272)

Two commits, one theme: **`_context.session_id` becomes the single
source of truth** for the remote AI chat conversation id.

## 1. `fix(ai): stamp session_id on runner context so remote transcript
restores`

The remote (web) AI channel is headless — a respawned `ai` task rebuilds
the conversation from its `_type:"ai"` Mongo docs, and
`restore_history_from_db` + the `RemoteBackend` poll both scope by
**`_context.session_id`**. Tool-action docs get that key via
`_get_result_context`, but the `prompt`/`response` turns only inherit
`_context` from `self.context` (`Runner._process_item` copies it).
`self.session_id` was resolved into a local var but **never written back
to `self.context`**, so a locally-derived id left the transcript turns
unqueryable → resume restored an empty history. It only worked on the
platform because the dispatcher supplies `session_id` in the context.

**Fix:** write the resolved `session_id` back onto `self.context` in
`_init_options` — one line, stamps every persisted item uniformly,
idempotent on the platform. Also fixes two pre-existing stale tests
(asserted the old top-level query shape) and adds
`TestSessionIdStampedOnContext`.

## 2. `refactor(ai): retire top-level Ai.session_id field`

With #1 guaranteeing `_context.session_id` on every doc, the top-level
`Ai.session_id` field is redundant — it was **write-only** in core and
duplicated the id that `restore_history_from_db`, the poll,
`_expire_stale_pending`, `poll_steers`, and secator-api all already key
on via `_context.session_id`. Removes the field + its 3 constructor
writes (resume prompt, turn_completed, remote pending prompt).
`ActionContext.session_id` and the `build_pending_prompt`/`ask_user`
params (the *working* id) are unchanged.

## Cross-repo (lockstep)
- **secator-api `feat/ai-chat` (PR #199)** — updated so
`answer_ai_prompt` resolves the pending prompt by `_context.session_id`
(was top-level) and the steer doc stops writing a redundant top-level
`session_id`. Must merge no later than this.
- **secator-ui `feat/ai-chat`** — no change; `getChatTranscript` already
queries `_context.session_id`.

## Verification
- Full AI unit suite: **no new failures** vs branch baseline (the ~68
remaining are the `shfmt`/`safecmd`-parser env failures common to all
branches).
- Empirically re-ran a remote turn: `prompt`/`response`/`follow_up` docs
all carry `_context.session_id` and **zero** top-level `session_id`;
answering via `_context.session_id` (the way updated secator-api does)
is picked up by the poll and continues the turn.

Part of the AI-task reliability series → `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#1273)

## Bug (hit during CLI testing)

```
🟢Query({"_type": "url", "verified": true})
[ERR] ai Action failed with error: AttributeError: 'str' object has no attribute 'items'
  File ".../secator/ai/actions.py", line 731, in _handle_query
    query_filter = _decrypt_dict(query_filter, ctx.encryptor)
  File ".../secator/ai/actions.py", line 1130, in _decrypt_dict
    for k, v in d.items()
```

The `query_workspace` tool schema **correctly** declares `query` as
`type: object`, but the model returned it as a JSON **string**
(`'{"_type":"url","verified":true}'`) — a well-known tool-calling quirk
where providers serialize nested object params as strings.
`_handle_query` assumed a dict and handed it to `_decrypt_dict`, which
does `d.items()` → `AttributeError`. It's caught by
`safe_dispatch_action` (so non-fatal — the model retried with `curl`),
but it wastes an iteration and **any** model that stringifies object
args can never use `query_workspace`. (`_decrypt_dict` only fires when
the encryptor is active, i.e. the default `sensitive=True`.)

## Fix
- `_handle_query`: if `query` is a `str`, `json.loads` it before
decrypt/search. On a non-JSON string or a non-dict, return a clean,
LLM-actionable `Error` (`"query must be a JSON object …"`) instead of
crashing. Mirrors the existing `add_finding` scalar coercion.
- `_decrypt_dict`: no-op backstop on non-dict input.

## Tests
`TestHandleQuery`: stringified query is coerced+searched **with the
encryptor active** (the exact original crash condition); non-JSON string
and non-dict each return one clean `Error`. `TestDecryptDict`: non-dict
input returned unchanged. Full AI suite: **no new failures** vs branch
baseline.

Found while testing `ai-testing`; targets `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…args, + LLM-response fuzz harness (#1275)

Makes the agent loop resilient to weird LLM tool-call responses — and
adds the harness that proves it.

## The fixes
1. **Coerce stringified object/array args**
(`opts`/`query`/`targets`/`choices`/…). Some models serialize nested
object params as JSON strings even though the schema says
`object`/`array`. A stringified `opts` crashed `_get_action_label`
(`'str' has no attribute 'get'`) and silently vanished in `_run_runner`.
`coerce_stringified_args()` runs at the tool-call boundary (before
decrypt/convert) and `json.loads`-es any such arg once. Malformed values
are left as-is for a clean handler error.
2. **Reject non-object arguments** — a model emitting a bare JSON
int/array/string (`12345`, `["nmap"]`) made `tool_call_to_action` call
`.items()` on a non-dict → `AttributeError` → the top-level catch-all →
**whole conversation aborted**. Now rejected cleanly to `None` so the
caller feeds an error back and the loop continues. (Surfaced by the
harness below.)

## The harness — `tests/unit/test_ai_resilience.py`
Fakes the LLM (patches `call_llm`) and drives the **real `_run_loop`**
with a battery of weird responses, asserting the invariant:

> a malformed response is handled turn-locally and the loop **survives**
— it never aborts the session via the top-level `except →
Error.from_exception; return` catch-all (monitored precisely), and never
raises.

Two layers:
- **Curated table** — stringified opts/query/targets/choices, wrong-type
args, broken JSON, missing fields, unknown tools, empty/huge/no-usage
responses.
- **Seeded fuzzer** — 200 random malformed `arguments` strings across
every tool; deterministic so any failure reproduces.

This is the tool that would have caught the stringified-`opts`/`query`
crashes *before* they were hit by hand — and it found the
non-object-args gap on its first run.

## Tests
`TestWeirdToolCalls` + `TestWeirdContentResponses` +
`TestMalformedArgFuzzer` (28) all green; `TestCoerceStringifiedArgs` +
`tool_call_to_action` non-object guard. Full AI suite: **no new
failures** vs branch baseline.

Targets `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ports (#1216)

The MarkdownExporter (run for every AI task report) did `item.content`
over `report.data[\"results\"][\"ai\"]`, which holds **serialized
dicts**, not `Ai` objects — raising `AttributeError: 'dict' object has
no attribute 'content'` and failing the export for all ai-task runs.

Fix: read `content` defensively (dict `.get` or object `getattr`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved Markdown report generation to handle content more reliably.
* The exporter now supports multiple result shapes and skips empty
entries, reducing the chance of missing or broken output.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…tryable 400) (#1276)

## Bug (hit during CLI testing, after compaction)

```
[WRN] Chat history trimmed: dropped 10 messages to fit under 100000 tokens.
[ERR] LLM call failed with non-retryable 400: ... each tool_use must have a single result.
      Found multiple `tool_result` blocks with id: toolu_bdrk_01Bi8...
```

Providers fold consecutive `tool` messages into a single user turn and
reject more than one `tool_result` per `tool_use` id — a
**non-retryable** 400 that aborts the whole run.

## Root cause
`_dispatch_and_collect` grouped batch results with `itertools.groupby`,
which groups only **consecutive** equal keys. In batch mode
(`_run_batch`) results **interleave** by `tool_call_id` (e.g. `[X, Y,
X]`), so `groupby` produced *several* groups for one id →
`add_tool_result` fired more than once for it → duplicate consecutive
`tool` messages → litellm folds them into one user turn with duplicate
`tool_result` blocks → 400. History trim/compaction restructures the
window the same way (hence the correlation with compaction).

## Fix (source + safety net)
1. **Order-preserving grouping** — group `collected` by an ordered
`dict` instead of `groupby`, so each `tool_call_id` yields exactly one
result regardless of arrival order (also fixes token accounting for
interleaved batches).
2. **`_dedupe_tool_results`** in `_repair_orphan_tool_uses` — within
each run of consecutive `tool` messages, keep the first result per id,
drop the rest. Runs **proactively** before every LLM call *and* on the
**400-repair-retry** path, so the "multiple tool_result blocks" 400 is
now repaired + retried instead of failing fast.

## Tests
`TestDedupeToolResults` (drop consecutive dup, no-op when unique,
per-run scoping, repair integration) + `call_llm` duplicate-400 →
repaired-and-retried. Full AI suite: **no new failures** vs branch
baseline.

Targets `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…by session_id (#1277)

## What / why (PR 2 of the runner-parenting design — core foundation)

AI-task-spawned sub-runners (nmap, httpx, subagents, …) run and their
**findings** persist, but the sub-runner itself never created its
**own** runner doc — so it never appeared in runner history. Root cause:
the child inherited the parent ai task's `context.task_id`, so
`update_runner`/`runner_id` targeted the *parent's* doc instead of
minting a new one.

## Fix
`_get_result_context` now builds a **clean child context**: strips the
parent's runner-identity keys
(`task_id`/`workflow_id`/`scan_id`/`task_chunk_id`), keeps
`session_id`/`drivers`/`workspace_*`. Each child now takes
`update_runner`'s insert branch → mints its own doc, linked to the
conversation by `context.session_id`.

## Verified (empirical)
Probe (`secator x ai -p "Run nmap … on scanme.nmap.org" --dangerous
-driver mongodb -ws …`):
- **Before:** 0 non-ai task docs under the session.
- **After:** the `nmap` child persists with its own `_id` (≠ parent),
`context.task_id` = its own, `context.session_id` = the conversation,
`status=SUCCESS`.

## Tests
`TestChildContextParenting` (child context keeps
session/drivers/workspace, sets `has_parent`, strips the four identity
keys). `test_ai_actions.py` 85/85; no new failures vs branch baseline.

## Not in this PR (design's later phases, go through the user)
- **secator-api:** a `GET /ai/conversations/{session_id}/runners` list
resolver (unions tasks/workflows/scans by `context.session_id`).
- **secator-ui:** group a conversation's runners under it.
- **Open decision (API/UI phase):** whether ai children should also
carry the *behavioral* top-level `has_parent` (controls whether they
show as top-level in the general runner history vs nested). The
persisted `has_parent` is currently `false`; parenting-by-`session_id`
works regardless.

Part of the AI-task reliability series → `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)
…r union) (#1278)

## Why (diagnosed from a live run)
Asked "What's in my workspace", the model correctly ran
`query_workspace` — then `cd`'d into
`~/.secator/reports/.../tasks/18/.outputs/` and `cat | jq`'d local JSON
to count finding types. Two causes:
1. **Local-driver gap:** the JSON exporter writes findings to disk only
at *end-of-run*, so mid-run `query_workspace` (local backend) sees
nothing — the prompt compensated by telling the model where the report
files live.
2. **Prompt framing:** "the runner folder … is where we store all
inputs/outputs" invited the model to *read* those files to find data — a
different, possibly stale store than the workspace it queried (outright
wrong under `--driver mongodb`/cloud).

## Fix — Query is the single source of truth
- **Local driver only:** `_handle_query` unions the backend results with
this run's in-memory findings (`ctx.results`), filtered by the same
query and deduped by `_uuid`. **mongodb/api are unchanged** (hooks
persist live → no union needed). The local driver is also exempted from
the `workspace_id` guard (it can answer from in-memory results).
- **Prompt (`common.txt`):** keep "write generated outputs to
`$workspace_path/.outputs/`"; drop the "we store all inputs/outputs
here" framing; add "ALWAYS use `query_workspace` — the single source of
truth; do NOT read local report files to find findings."

## Tests
`_union_live_results` (filter+merge+dedup), local-driver unions live
results, mongodb does NOT union, local exempt from the workspace guard,
and the updated `no_workspace` guard test (now scoped to non-local
backends). Full AI suite: no new failures.

Targets `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…eritance (PR1 1.b/1.c) (#1279)

## What (PR 1, parts 1.b + 1.c + a subagent-runnability fix)

**1.b/1.c — structured, evidence-backed subagent prompt.** When the ai
task spawns a subagent, its prompt becomes a **structured** template
whose "already known" section is **auto-assembled** from workspace
findings for the subagent's target(s), so it doesn't redo work.
- `build_subagent_prompt(objective, targets, evidence)` — wraps the
LLM-supplied objective (**verbatim**) in `## Objective / ## Scope / ##
Already known (do not re-run) / ## Expected output`.
- `_gather_subagent_evidence(ctx, targets, limit=40)` — queries the
workspace (single source of truth incl. this run's live findings, per
#1278) for findings matching the targets; token-bounded; **best-effort**
(failure → `""`, never breaks the spawn).
- Wired into `_run_runner`'s `name=="ai"` branch. **1.a (permissions)
untouched** — out of scope.

**Subagent LLM inheritance (fold-in).** E2E testing surfaced that a
spawned subagent fell back to `CONFIG.addons.ai.default_model` — a
different provider than the parent (anthropic-direct vs openrouter) with
no key → `AuthenticationError` before it ran, so **subagents never
actually ran**. Fixed: carry the parent's resolved
`model`/`api_key`/`api_base` on `ActionContext` and `setdefault` them
onto the child opts at spawn (explicit LLM-supplied model still wins).

## Verification
- Unit: `TestBuildSubagentPrompt`, `TestGatherSubagentEvidence`, and
`TestRunRunner` tests for the structured prompt + LLM inheritance (+
explicit-model-wins). `test_ai_actions.py` **97 passed**; no new
failures vs baseline.
- **E2E** (live subagent run): subagent runs to **SUCCESS** (no
AuthError), receives the structured prompt with **all four sections**,
and its spawned nmap **nests** under the conversation via `session_id`.

## Known limitation (accepted, v1)
The `$or` host/ip/url evidence match is exact — `matched_at`-only
findings on URL-only targets aren't gathered yet (deliberate follow-up).

Part of the AI-task series → `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ackend) (#1280)

## Bug
```
🟢Query({'_type': 'ip', 'host': 'cachyos.local'}) -> failed results (limit: 10)
[ERR] ai TypeError: '>=' not supported between instances of 'int' and 'str'
  File ".../query/json.py", line 182, in _execute_search
    if limit and len(matched) >= limit:
```
The model sent `query_workspace`'s `limit` as a **string** (`"10"`); it
reached the backend and broke the `>=` comparison. Same "model
stringifies args" class as #1275, but `coerce_stringified_args` only
handles object/array params — not scalar ints like `limit`.

## Fix
Coerce `limit` to `int` in `_handle_query`; bad/None values fall back to
the default (100).

## Tests
Stringified `"10"` → `10` reaches the backend; a non-numeric limit falls
back to 100. `test_ai_actions.py` 99 passed; the original crash
reproduces as resolved.

Targets `ai-resiliency` (#1241).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
ChatHistory.trim() fed self.messages straight to litellm's trim_messages,
whose shorten-to-fit path does len(msg["content"]) — which raises
"TypeError: object of type 'NoneType' has no len()" for an assistant turn
carrying only tool_calls (content=None), killing the AI loop. Coerce None
content to "" before trimming (equivalent for the LLM, safe for len()) and
wrap the trim in a guard so any trimmer failure degrades to untrimmed
history (handled downstream by the context_length_exceeded 400-repair)
instead of crashing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNjPggRSVZ2xnLb7ZxWP5H
ocervell added a commit that referenced this pull request Jul 6, 2026
…ed tasks (#1284)

**Re-extracted from the closed #1199** — this generic Celery
worker-reliability fix was folded into the `ai-resiliency` branch
(#1241, AI-task hardening, base `canary`) and its standalone PR closed.
It's general worker-pool infra (nothing AI-specific), so it belongs on
`main` on its own timeline. Cherry-picked clean onto current `main`.

## What it does
Under `task_acks_late` + `task_reject_on_worker_lost` (prod config), a
task that dies with its worker (SIGKILL / OOM / node eviction) is
redelivered with the **same Celery id**. A task that OOMs on *every* run
would loop forever (OOM → redeliver → OOM), blocking its chord
indefinitely. This caps redeliveries: `run_command` counts worker-loss
redeliveries (Redis `celery-task-meta-worker-loss-<id>`); once
`task_max_retries` is exceeded it **abandons** the task with a clear
`abandoned after N delivery attempts` result so the chord/workflow can
finish. `task_max_retries=-1` disables the cap (default). Also adds the
connection-loss cancel flag.

## Validation
Exercised end-to-end earlier (prefork child-loss → requeue → abandon at
cap, 6/6). **`tests/unit/test_celery.py`: 23 passed** on current `main`;
flake8 clean.

## Note
`feature:worker-reliability`. Companion PR extracts the
eviction-finalize half (was #1203). Prod workers run `-P prefork -c 1`,
so the fast child-loss path is live.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a limit for repeated worker-loss retries so long-running jobs
can stop retrying after a configurable cap.
* Introduced an option to cancel in-flight tasks when the broker
connection is lost, helping prevent stuck background work.

* **Bug Fixes**
* Improved handling of interrupted tasks so workflows can complete with
a clear failure result instead of hanging indefinitely.
* Added safer retry counting to better track redelivered tasks across
worker losses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ocervell
ocervell marked this pull request as draft July 8, 2026 08:37
@ocervell ocervell added the superseded Folded into a unified/newer PR; do not merge label Jul 8, 2026
ocervell and others added 2 commits July 8, 2026 10:57
Shell commands the AI runs become first-class secator runners —
persisted as runner docs, visible in history, parented under the
conversation via `context.session_id`. Built on `origin/ai-resiliency`
(PR 3 of the AI subagent/parenting/bash-runner design).

## What's in it
- **`command` task** (`secator/tasks/command.py`) — a `Command` subclass
that runs an arbitrary command line verbatim in shell mode
(`input_types=[]` so bare commands like `whoami` aren't type-stripped;
`_build_cmd` restores `shell=True`). stdout captured via the base
runner's `self.output`.
- **`command.from_result(...)`** — a populate-without-execute import
seam (build a runner doc from an already-run command's
cmd/output/status/timing without re-running it — the future path for
importing externally-run commands into Cloud). Execute vs populate paths
cleanly separated.
- **`_handle_shell` rewired** (`secator/ai/actions.py`) — dispatches the
`command` task in-process instead of raw `subprocess.run`, so the
command persists as a runner doc (mongodb `update_runner` hooks)
parented by `session_id`, while preserving the `shell`/`shell_output`
transcript contract, the 60s timeout, and the output cap.
- **env sanitization preserved & hardened** — a backward-compatible
`env` run-opt on base `Command` (`env = self.run_opts.get('env',
os.environ)`) lets `_handle_shell` pass `_sanitized_env()` (strips
`ANTHROPIC_`/`AWS_`/`SECRET_`/API-key vars) so an AI-run `env` can't
leak creds into output; `env` is kept **runtime-only** (excluded from
persisted `resolved_opts`) so it never lands in the runner doc.

## Testing
- Unit: `tests/unit/test_command_task.py` (verbatim exec, shell
metachars, bare-command, from_result success/failure/no-exec, env
honored/not-persisted), plus rewired
`test_ai_actions.py`/`test_ai_loop.py` shell tests. Full `tests/unit`:
**1071 passed, 0 regressions** vs baseline.
- E2E: verified against a live Mongo — an AI shell command persists a
`command` runner doc with cmd/output/status, parented by `session_id`,
and the `shell_output` appears in the transcript.

Built via subagent-driven-development (per-task reviews + a final
whole-branch review; the review loop caught a shell-mode clobber,
input-type stripping, output corruption, an env-sanitization regression,
and — via the live-Mongo E2E — a silent hook-resolution no-op).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Restore an AI conversation from **one code path** for both the local
(JSON) and remote (Mongo) drivers, rebuilding the **full litellm
transcript including tool calls / tool results** — not just the
prompt/response text it reconstructed before. Built on `ai-resiliency`;
to be merged there once validated.

## How
Every transcript turn persists its exact litellm message on a new
`Ai.message` field. `restore_history_from_db` collects those by
`session_id` (ordered by `_timestamp`) and sets `history.messages`
verbatim — a faithful, valid transcript continuation, backend-agnostic.
Local resume adopts the picked session's `session_id` and routes through
the same restore.

## Commits
1. `Ai.message` transcript carrier + `cap_message` (BSON-safety
backstop, far under 16MB).
2. Persist prompt/assistant turns as raw messages — incl.
previously-dropped **tool-call-only** turns.
3. Persist tool results as new `tool_result` docs (with `runner_id`) at
all 4 sites.
4. Rewrite `restore_history_from_db` to rebuild the full transcript from
`message` (legacy docs → text-only fallback; orphan-repair retained).
5. Unify local resume through `restore_history_from_db` + adopt the
prior `session_id`; **legacy (pre-session_id) sessions fall back to
`replay_session`** so they don't restore empty.
6. Scoping / cross-backend (real JsonBackend + Mongo-mock) /
size-backstop / UI-safety coverage.

## Key properties
- **Encryption:** `message.content` is encrypted exactly once (persisted
verbatim, no re-encryption on restore); legacy fallback re-encrypts its
plaintext. A restore yields the same in-memory (encrypted) history a
fresh run would.
- **Scoping:** every `_type:"ai"` doc carries `_context.session_id`;
restore is scoped to it (verified: zero cross-conversation bleed on both
backends).
- **No new truncation:** tool-result content is already bounded upstream
by `truncate_to_tokens`; `cap_message` is only a hard envelope backstop.

## Testing
Built via subagent-driven-development (per-task reviews + a whole-branch
review that caught & fixed a blocker: legacy local sessions restoring
empty). Full `tests/unit`: **0 new failures** vs the `ai-resiliency`
baseline (the ~125 failures are pre-existing env issues — missing
`shfmt`/`safecmd`).

## Known follow-ups (out of scope, parked)
- Driver-aware spill pointer for truncated tool output (+ extend
`query_workspace` to query runners, not just findings).
- Rich UI rendering of `tool_result` docs in the chat transcript
(secator-ui); tool-call-only turns now emit an empty-content `response`
doc the UI should filter.
- Local resume no longer console-prints the prior conversation for
new-format sessions (small UX consideration).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request superseded Folded into a unified/newer PR; do not merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant