feat(ai): AI-task reliability, robustness & security hardening (27 fixes) - #1241
Draft
ocervell wants to merge 100 commits into
Draft
feat(ai): AI-task reliability, robustness & security hardening (27 fixes)#1241ocervell wants to merge 100 commits into
ocervell wants to merge 100 commits into
Conversation
The api driver POSTs targets, raw command output and the Bearer API key to addons.api.url. force_ssl (default True) only governed TLS cert verification, so an operator setting force_ssl:false would ship that data over cleartext HTTP to a remote host (MITM / key-leak risk). _make_request now hard-fails any http:// URL whose host is not loopback (localhost / 127.0.0.1 / ::1), regardless of force_ssl. https:// and loopback-http stay working; force_ssl still controls cert verification only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed tasks When a worker is killed mid-task (cgroup OOMKill of the child, or a node memory-pressure eviction), task_acks_late + task_reject_on_worker_lost cause the task to be redelivered with the same Celery id. A task that OOMs on every run would loop forever (OOM -> redeliver -> OOM) and the surrounding chord/workflow could never complete. Add task_max_retries (-1 = disabled). run_command counts redeliveries on the Celery result backend, keyed by the stable Celery id, and once the cap is exceeded abandons the task: it returns a forwarded result list with a FAILURE Error appended instead of re-running, so the chord proceeds and the workflow finishes with a clear 'abandoned after N retries' error (mirroring the inner memory-limit warning). The counter is backend-agnostic: it uses the result backend's generic key/value interface (atomic incr when available e.g. Redis/Memcached, else a get/set read-modify-write that every KV backend implements - filesystem, S3, GCS, ...). It is not tied to Redis or to any Secator data backend (Mongo/Postgres/SQLite). Backends with neither (database/ RPC) degrade safely to 'cap disabled'. Inert by default (task_max_retries=-1 and task_acks_late=False); enabling requires task_acks_late + task_reject_on_worker_lost and raising broker_visibility_timeout above the max task lifetime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…ests from CONFIG pollution
…loss cancel flag Completes the worker-loss redelivery cap with the review fixes + the L1 delivery flag: - Off-by-one: the initial delivery no longer counts as a retry. Extracted worker_loss_retries_exhausted(delivery_count, max_retries) — redeliveries = delivery_count - 1, so task_max_retries=3 allows the initial run + 3 redeliveries before abandoning (was 2). Unit-tested at the boundary. - Key expiry: worker-loss counter keys now get a best-effort TTL tied to result_expires (via _expire_worker_loss_key) on both the atomic-incr and the get/set paths, so they don't accumulate forever. - L1: add worker_cancel_long_running_tasks_on_connection_loss config flag (default False) wired into app.conf, alongside the existing task_acks_late / task_reject_on_worker_lost. Enables clean redelivery of a connection-lost worker's in-flight tasks. Inert until enabled in deployment. Note: the terminal-doc no-op guard (a redelivered task whose runner doc is already done) is deferred — it depends on stable runner identity (on_build, the L2 PR) to find 'its' doc, and is ineffective without it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The abandon message said 'after N retries' using task_max_retries, which read
oddly at max_retries=0 ('after 0 retries') because a redelivery still occurs
even with 0 retries — the broker redelivers a lost task under acks_late; that
is NOT a re-run of the work. Thread the actual delivery_count through and
report it separately from the cap:
Task httpx abandoned after 5 delivery attempts (retry cap: 3; worker
repeatedly lost — likely OOM kill or node eviction).
Reads sensibly for any cap value (incl. 0).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_on_build.py imported secator.hooks.mongodb at module level, which pulls pymongo — an optional addon not installed in the base CI test env — so the whole file failed to collect. Guard the import behind ADDONS_ENABLED['mongodb'] and mark the 4 mongodb-dependent classes with pytest.skipif on the same flag (works for both the unittest.TestCase and plain pytest classes). The hook-registration and sqlite tests don't need pymongo and keep running. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… proceeds On a K8s pod eviction (SIGTERM -> grace -> SIGKILL) the worker running a task dies. With task_acks_late on the Redis broker, its message is only redelivered after the broker visibility timeout (hours, on the long-task pool) — stalling the surrounding chord/workflow that whole time. Unlike a child OOM (where the master survives and task_reject_on_worker_lost requeues immediately), a whole-pod eviction has no surviving master, so Redis can only fall back to the visibility timeout. Catch the worker_shutting_down signal and raise a flag the running task's monitor thread polls. On shutdown it stops the task early via the existing stop_process path, so the task returns its partial results through the normal completion path and the chord proceeds in seconds instead of waiting for the visibility timeout. - celery_signals.py: SHUTDOWN_FLAG + worker_shutting_down_handler (wired unconditionally); stale flag cleared on worker boot. - command.py: the monitor thread checks the flag (same self-stop used for timeout/memory limits) and caps its poll interval (MONITOR_POLL_SECONDS=5) so an eviction is caught well within the pod's terminationGracePeriodSeconds. - tests: flag lifecycle + a behavioral test (a running command stops early and emits the eviction warning). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…late test The integration suite caught a real leak: SHUTDOWN_FLAG is a machine-global file, so once any worker fired worker_shutting_down (e.g. between test files) the flag persisted and every later task whose monitor polled once self-aborted — slow tasks failed, fast ones that finished before the first poll passed. Prod never hit it (1 task = 1 pod = fresh /tmp), but any shared/long-lived worker does. - command.py: clear the flag at monitor start, so it only means "shutdown raised *during this run*", not a stale flag from a previous worker/task. - celery_signals.py (CodeRabbit): ensure the flag's parent dir exists and catch + log OSError instead of swallowing all exceptions. - test_eviction.py (CodeRabbit): isolate SHUTDOWN_FLAG to a per-test temp path; add a regression test that a flag set *before* a task starts does not stop it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…ace) The behavioral test set the flag once after a fixed 2s wait, but the monitor now clears any pre-existing flag at startup; on a slow CI runner the monitor started *after* that single set and wiped it, so the task never stopped and the test failed. Re-raise the flag in a loop until the task stops, so the monitor's poll sees it once it is running regardless of start timing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
The real-subprocess behavioral test was environment-dependent: in CI the monitor thread didn't stop the live `sleep` reliably, so the test flaked (and the stale test passed for the wrong reason). Replace both with deterministic tests that exercise _monitor_process directly against a bare Command — one asserts it calls stop_process(exit_ok=True) + emits the eviction Warning when the flag is set, the other asserts it clears a stale flag at start (the integration-leak regression). No subprocess, no timing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
The real-file version of the stale-flag test flaked on CI (patched temp path + bare-__new__ Command), while the real behaviour is already proven green by the integration suite. Assert that _monitor_process calls clear_shutdown_flag() at start via a mock instead of inspecting the filesystem — deterministic and environment-independent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
Add a `status` field to the Vulnerability output type (NEW / ACKNOWLEDGED / FIXED, default NEW), marked compare=False so dedup identity (name/id/matched_at) is unchanged. Normalize/validate in __post_init__ (coerce empty/None/unknown to NEW, uppercase) and display it via _table_fields. Carry status across re-scans: add `status` to the duplicate_main_copy_fields defaults of MongodbAddon and SqliteAddon, and in compute_duplicate_updates treat a status of ''/None/'NEW' as unset (per-field sentinel) so a prior ACKNOWLEDGED/FIXED carries forward onto a re-found main whose status is still the default NEW, while never-touched vulns stay NEW. Other fields keep the generic `not value` emptiness check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
# Conflicts: # .github/workflows/publish-canary.yml
Adds the two secator-core gaps for the Workspace AI Assistant (Mongo-channel chat), repo 1/3: - `restore_history_from_db(session_id, query_engine, model, encryptor, system_prompt)` in `secator/ai/session.py`: rebuilds a `ChatHistory` from the workspace `_type:"ai"` docs (queried by session_id, ordered by `_timestamp`) — `prompt`→user, `response`→assistant, system prompt set, re-encrypted when an encryptor is active. Headless: no local files, no TUI. - Wire a remote-resume branch in `ai.py:yielder`: when `interactive="remote"` and the session has prior `_type:"ai"` docs, restore from Mongo and continue; fresh conversations (no docs) start as before. The local CLI `replay_session`/`show_session_picker` path is untouched. - `session_id` now prefers `run_opts.context.session_id` so a respawned task finds its prior docs. - `save_history` (local `history.json`) is skipped on the remote path via a `_save_history()` helper — the Mongo docs are the source of truth. - Query-engine guard: warn when `interactive="remote"` but the resolved query backend is not mongodb/api (the web answer channel can't work otherwise). History-fidelity finding: persisted `_type:"ai"` docs capture only text turns (prompt/response) plus action *display* records — not the litellm assistant `tool_calls` messages or their `tool` results. Restore is therefore text-only. This is valid and sufficient for `mode="chat"` continuation; fabricating partial tool-call messages would produce a malformed transcript providers reject, so tool activity is deliberately collapsed. Richer assistant persistence for `mode="attack"` replay is a documented follow-up. Tests: `tests/unit/test_ai_session.py` — restore rebuilds equivalent History (order/roles/system/encryption/empty-docs/search-failure), and the remote-resume branch picks Mongo restore for prior docs / fresh otherwise / warns on non-Mongo backend. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
…nscript
The web UI correlates an AI chat conversation by session_id (across respawns),
but only the resume-prompt and follow_up items set it — the prompt/response/
token_usage/chat_compacted message items did not, so they persisted to Mongo
without session_id and the UI's {_type:"ai", session_id} query returned nothing
(empty transcript despite the task running fine). Wrap yielder to stamp
session_id on every Ai item centrally (self.session_id is set in _init_options
before the first yield).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm
… ship shfmt via the ai addon (#1274) ## Problem (hit during CLI testing) In attack mode every shell command the LLM issues hit the guardrail shell parser, which printed: ``` [ERR] Missing ai addon: please run "secator install addons ai". ``` …even though the ai addon **was** installed. `_parse_subcommands` (`guardrails.py`) shells out to `shfmt` via `safecmd`; the misleading message came from its `ImportError` branch. The env had `litellm` (so `ADDONS_ENABLED['ai']=True`) but not `safecmd`/`shfmt` — a partial ai install. Worse, the *other* failure mode (safecmd present, `shfmt` binary not on PATH) was **silent** (`FileNotFoundError` swallowed by `except Exception: return []`). ## Fix 1. **Message** — replace the `Error("Missing ai addon")` with a **one-shot `Warning`** that names the real gap: `"Missing safecmd shell parser"` (ImportError) or `"Missing shfmt binary"` (FileNotFoundError). Both fall back to the **non-shfmt path** — an empty sub-command list, so `_check_action_type` asks the user to approve the whole command (safe, just coarser). Warn once per process (no per-command spam). 2. **Install** — add `shfmt-py` explicitly to the `ai` extra so `secator install addons ai` (`pip install secator[ai]`) always ships the `shfmt` **binary**, not just the `safecmd` Python package (which only pulled it transitively). ## Tests `TestShellParserFallback`: safecmd-missing → `Warning` naming `safecmd`, never `"ai addon"`; shfmt-binary-missing → `Warning` naming `shfmt`; warn-once across multiple commands. Full AI suite: **no new failures** vs branch baseline. Found while testing `ai-testing`; targets `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…chat conversation id (#1272) Two commits, one theme: **`_context.session_id` becomes the single source of truth** for the remote AI chat conversation id. ## 1. `fix(ai): stamp session_id on runner context so remote transcript restores` The remote (web) AI channel is headless — a respawned `ai` task rebuilds the conversation from its `_type:"ai"` Mongo docs, and `restore_history_from_db` + the `RemoteBackend` poll both scope by **`_context.session_id`**. Tool-action docs get that key via `_get_result_context`, but the `prompt`/`response` turns only inherit `_context` from `self.context` (`Runner._process_item` copies it). `self.session_id` was resolved into a local var but **never written back to `self.context`**, so a locally-derived id left the transcript turns unqueryable → resume restored an empty history. It only worked on the platform because the dispatcher supplies `session_id` in the context. **Fix:** write the resolved `session_id` back onto `self.context` in `_init_options` — one line, stamps every persisted item uniformly, idempotent on the platform. Also fixes two pre-existing stale tests (asserted the old top-level query shape) and adds `TestSessionIdStampedOnContext`. ## 2. `refactor(ai): retire top-level Ai.session_id field` With #1 guaranteeing `_context.session_id` on every doc, the top-level `Ai.session_id` field is redundant — it was **write-only** in core and duplicated the id that `restore_history_from_db`, the poll, `_expire_stale_pending`, `poll_steers`, and secator-api all already key on via `_context.session_id`. Removes the field + its 3 constructor writes (resume prompt, turn_completed, remote pending prompt). `ActionContext.session_id` and the `build_pending_prompt`/`ask_user` params (the *working* id) are unchanged. ## Cross-repo (lockstep) - **secator-api `feat/ai-chat` (PR #199)** — updated so `answer_ai_prompt` resolves the pending prompt by `_context.session_id` (was top-level) and the steer doc stops writing a redundant top-level `session_id`. Must merge no later than this. - **secator-ui `feat/ai-chat`** — no change; `getChatTranscript` already queries `_context.session_id`. ## Verification - Full AI unit suite: **no new failures** vs branch baseline (the ~68 remaining are the `shfmt`/`safecmd`-parser env failures common to all branches). - Empirically re-ran a remote turn: `prompt`/`response`/`follow_up` docs all carry `_context.session_id` and **zero** top-level `session_id`; answering via `_context.session_id` (the way updated secator-api does) is picked up by the poll and continues the turn. Part of the AI-task reliability series → `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#1273) ## Bug (hit during CLI testing) ``` 🟢Query({"_type": "url", "verified": true}) [ERR] ai Action failed with error: AttributeError: 'str' object has no attribute 'items' File ".../secator/ai/actions.py", line 731, in _handle_query query_filter = _decrypt_dict(query_filter, ctx.encryptor) File ".../secator/ai/actions.py", line 1130, in _decrypt_dict for k, v in d.items() ``` The `query_workspace` tool schema **correctly** declares `query` as `type: object`, but the model returned it as a JSON **string** (`'{"_type":"url","verified":true}'`) — a well-known tool-calling quirk where providers serialize nested object params as strings. `_handle_query` assumed a dict and handed it to `_decrypt_dict`, which does `d.items()` → `AttributeError`. It's caught by `safe_dispatch_action` (so non-fatal — the model retried with `curl`), but it wastes an iteration and **any** model that stringifies object args can never use `query_workspace`. (`_decrypt_dict` only fires when the encryptor is active, i.e. the default `sensitive=True`.) ## Fix - `_handle_query`: if `query` is a `str`, `json.loads` it before decrypt/search. On a non-JSON string or a non-dict, return a clean, LLM-actionable `Error` (`"query must be a JSON object …"`) instead of crashing. Mirrors the existing `add_finding` scalar coercion. - `_decrypt_dict`: no-op backstop on non-dict input. ## Tests `TestHandleQuery`: stringified query is coerced+searched **with the encryptor active** (the exact original crash condition); non-JSON string and non-dict each return one clean `Error`. `TestDecryptDict`: non-dict input returned unchanged. Full AI suite: **no new failures** vs branch baseline. Found while testing `ai-testing`; targets `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…args, + LLM-response fuzz harness (#1275) Makes the agent loop resilient to weird LLM tool-call responses — and adds the harness that proves it. ## The fixes 1. **Coerce stringified object/array args** (`opts`/`query`/`targets`/`choices`/…). Some models serialize nested object params as JSON strings even though the schema says `object`/`array`. A stringified `opts` crashed `_get_action_label` (`'str' has no attribute 'get'`) and silently vanished in `_run_runner`. `coerce_stringified_args()` runs at the tool-call boundary (before decrypt/convert) and `json.loads`-es any such arg once. Malformed values are left as-is for a clean handler error. 2. **Reject non-object arguments** — a model emitting a bare JSON int/array/string (`12345`, `["nmap"]`) made `tool_call_to_action` call `.items()` on a non-dict → `AttributeError` → the top-level catch-all → **whole conversation aborted**. Now rejected cleanly to `None` so the caller feeds an error back and the loop continues. (Surfaced by the harness below.) ## The harness — `tests/unit/test_ai_resilience.py` Fakes the LLM (patches `call_llm`) and drives the **real `_run_loop`** with a battery of weird responses, asserting the invariant: > a malformed response is handled turn-locally and the loop **survives** — it never aborts the session via the top-level `except → Error.from_exception; return` catch-all (monitored precisely), and never raises. Two layers: - **Curated table** — stringified opts/query/targets/choices, wrong-type args, broken JSON, missing fields, unknown tools, empty/huge/no-usage responses. - **Seeded fuzzer** — 200 random malformed `arguments` strings across every tool; deterministic so any failure reproduces. This is the tool that would have caught the stringified-`opts`/`query` crashes *before* they were hit by hand — and it found the non-object-args gap on its first run. ## Tests `TestWeirdToolCalls` + `TestWeirdContentResponses` + `TestMalformedArgFuzzer` (28) all green; `TestCoerceStringifiedArgs` + `tool_call_to_action` non-object guard. Full AI suite: **no new failures** vs branch baseline. Targets `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ports (#1216) The MarkdownExporter (run for every AI task report) did `item.content` over `report.data[\"results\"][\"ai\"]`, which holds **serialized dicts**, not `Ai` objects — raising `AttributeError: 'dict' object has no attribute 'content'` and failing the export for all ai-task runs. Fix: read `content` defensively (dict `.get` or object `getattr`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01P5vSjfkBuGAAHdKxHS3ySm <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Markdown report generation to handle content more reliably. * The exporter now supports multiple result shapes and skips empty entries, reducing the chance of missing or broken output. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…tryable 400) (#1276) ## Bug (hit during CLI testing, after compaction) ``` [WRN] Chat history trimmed: dropped 10 messages to fit under 100000 tokens. [ERR] LLM call failed with non-retryable 400: ... each tool_use must have a single result. Found multiple `tool_result` blocks with id: toolu_bdrk_01Bi8... ``` Providers fold consecutive `tool` messages into a single user turn and reject more than one `tool_result` per `tool_use` id — a **non-retryable** 400 that aborts the whole run. ## Root cause `_dispatch_and_collect` grouped batch results with `itertools.groupby`, which groups only **consecutive** equal keys. In batch mode (`_run_batch`) results **interleave** by `tool_call_id` (e.g. `[X, Y, X]`), so `groupby` produced *several* groups for one id → `add_tool_result` fired more than once for it → duplicate consecutive `tool` messages → litellm folds them into one user turn with duplicate `tool_result` blocks → 400. History trim/compaction restructures the window the same way (hence the correlation with compaction). ## Fix (source + safety net) 1. **Order-preserving grouping** — group `collected` by an ordered `dict` instead of `groupby`, so each `tool_call_id` yields exactly one result regardless of arrival order (also fixes token accounting for interleaved batches). 2. **`_dedupe_tool_results`** in `_repair_orphan_tool_uses` — within each run of consecutive `tool` messages, keep the first result per id, drop the rest. Runs **proactively** before every LLM call *and* on the **400-repair-retry** path, so the "multiple tool_result blocks" 400 is now repaired + retried instead of failing fast. ## Tests `TestDedupeToolResults` (drop consecutive dup, no-op when unique, per-run scoping, repair integration) + `call_llm` duplicate-400 → repaired-and-retried. Full AI suite: **no new failures** vs branch baseline. Targets `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…by session_id (#1277) ## What / why (PR 2 of the runner-parenting design — core foundation) AI-task-spawned sub-runners (nmap, httpx, subagents, …) run and their **findings** persist, but the sub-runner itself never created its **own** runner doc — so it never appeared in runner history. Root cause: the child inherited the parent ai task's `context.task_id`, so `update_runner`/`runner_id` targeted the *parent's* doc instead of minting a new one. ## Fix `_get_result_context` now builds a **clean child context**: strips the parent's runner-identity keys (`task_id`/`workflow_id`/`scan_id`/`task_chunk_id`), keeps `session_id`/`drivers`/`workspace_*`. Each child now takes `update_runner`'s insert branch → mints its own doc, linked to the conversation by `context.session_id`. ## Verified (empirical) Probe (`secator x ai -p "Run nmap … on scanme.nmap.org" --dangerous -driver mongodb -ws …`): - **Before:** 0 non-ai task docs under the session. - **After:** the `nmap` child persists with its own `_id` (≠ parent), `context.task_id` = its own, `context.session_id` = the conversation, `status=SUCCESS`. ## Tests `TestChildContextParenting` (child context keeps session/drivers/workspace, sets `has_parent`, strips the four identity keys). `test_ai_actions.py` 85/85; no new failures vs branch baseline. ## Not in this PR (design's later phases, go through the user) - **secator-api:** a `GET /ai/conversations/{session_id}/runners` list resolver (unions tasks/workflows/scans by `context.session_id`). - **secator-ui:** group a conversation's runners under it. - **Open decision (API/UI phase):** whether ai children should also carry the *behavioral* top-level `has_parent` (controls whether they show as top-level in the general runner history vs nested). The persisted `has_parent` is currently `false`; parenting-by-`session_id` works regardless. Part of the AI-task reliability series → `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…r union) (#1278) ## Why (diagnosed from a live run) Asked "What's in my workspace", the model correctly ran `query_workspace` — then `cd`'d into `~/.secator/reports/.../tasks/18/.outputs/` and `cat | jq`'d local JSON to count finding types. Two causes: 1. **Local-driver gap:** the JSON exporter writes findings to disk only at *end-of-run*, so mid-run `query_workspace` (local backend) sees nothing — the prompt compensated by telling the model where the report files live. 2. **Prompt framing:** "the runner folder … is where we store all inputs/outputs" invited the model to *read* those files to find data — a different, possibly stale store than the workspace it queried (outright wrong under `--driver mongodb`/cloud). ## Fix — Query is the single source of truth - **Local driver only:** `_handle_query` unions the backend results with this run's in-memory findings (`ctx.results`), filtered by the same query and deduped by `_uuid`. **mongodb/api are unchanged** (hooks persist live → no union needed). The local driver is also exempted from the `workspace_id` guard (it can answer from in-memory results). - **Prompt (`common.txt`):** keep "write generated outputs to `$workspace_path/.outputs/`"; drop the "we store all inputs/outputs here" framing; add "ALWAYS use `query_workspace` — the single source of truth; do NOT read local report files to find findings." ## Tests `_union_live_results` (filter+merge+dedup), local-driver unions live results, mongodb does NOT union, local exempt from the workspace guard, and the updated `no_workspace` guard test (now scoped to non-local backends). Full AI suite: no new failures. Targets `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…eritance (PR1 1.b/1.c) (#1279) ## What (PR 1, parts 1.b + 1.c + a subagent-runnability fix) **1.b/1.c — structured, evidence-backed subagent prompt.** When the ai task spawns a subagent, its prompt becomes a **structured** template whose "already known" section is **auto-assembled** from workspace findings for the subagent's target(s), so it doesn't redo work. - `build_subagent_prompt(objective, targets, evidence)` — wraps the LLM-supplied objective (**verbatim**) in `## Objective / ## Scope / ## Already known (do not re-run) / ## Expected output`. - `_gather_subagent_evidence(ctx, targets, limit=40)` — queries the workspace (single source of truth incl. this run's live findings, per #1278) for findings matching the targets; token-bounded; **best-effort** (failure → `""`, never breaks the spawn). - Wired into `_run_runner`'s `name=="ai"` branch. **1.a (permissions) untouched** — out of scope. **Subagent LLM inheritance (fold-in).** E2E testing surfaced that a spawned subagent fell back to `CONFIG.addons.ai.default_model` — a different provider than the parent (anthropic-direct vs openrouter) with no key → `AuthenticationError` before it ran, so **subagents never actually ran**. Fixed: carry the parent's resolved `model`/`api_key`/`api_base` on `ActionContext` and `setdefault` them onto the child opts at spawn (explicit LLM-supplied model still wins). ## Verification - Unit: `TestBuildSubagentPrompt`, `TestGatherSubagentEvidence`, and `TestRunRunner` tests for the structured prompt + LLM inheritance (+ explicit-model-wins). `test_ai_actions.py` **97 passed**; no new failures vs baseline. - **E2E** (live subagent run): subagent runs to **SUCCESS** (no AuthError), receives the structured prompt with **all four sections**, and its spawned nmap **nests** under the conversation via `session_id`. ## Known limitation (accepted, v1) The `$or` host/ip/url evidence match is exact — `matched_at`-only findings on URL-only targets aren't gathered yet (deliberate follow-up). Part of the AI-task series → `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ackend) (#1280) ## Bug ``` 🟢Query({'_type': 'ip', 'host': 'cachyos.local'}) -> failed results (limit: 10) [ERR] ai TypeError: '>=' not supported between instances of 'int' and 'str' File ".../query/json.py", line 182, in _execute_search if limit and len(matched) >= limit: ``` The model sent `query_workspace`'s `limit` as a **string** (`"10"`); it reached the backend and broke the `>=` comparison. Same "model stringifies args" class as #1275, but `coerce_stringified_args` only handles object/array params — not scalar ints like `limit`. ## Fix Coerce `limit` to `int` in `_handle_query`; bad/None values fall back to the default (100). ## Tests Stringified `"10"` → `10` reaches the backend; a non-numeric limit falls back to 100. `test_ai_actions.py` 99 passed; the original crash reproduces as resolved. Targets `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 5, 2026
ChatHistory.trim() fed self.messages straight to litellm's trim_messages, whose shorten-to-fit path does len(msg["content"]) — which raises "TypeError: object of type 'NoneType' has no len()" for an assistant turn carrying only tool_calls (content=None), killing the AI loop. Coerce None content to "" before trimming (equivalent for the LLM, safe for len()) and wrap the trim in a guard so any trimmer failure degrades to untrimmed history (handled downstream by the context_length_exceeded 400-repair) instead of crashing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNjPggRSVZ2xnLb7ZxWP5H
This was referenced Jul 6, 2026
ocervell
added a commit
that referenced
this pull request
Jul 6, 2026
…ed tasks (#1284) **Re-extracted from the closed #1199** — this generic Celery worker-reliability fix was folded into the `ai-resiliency` branch (#1241, AI-task hardening, base `canary`) and its standalone PR closed. It's general worker-pool infra (nothing AI-specific), so it belongs on `main` on its own timeline. Cherry-picked clean onto current `main`. ## What it does Under `task_acks_late` + `task_reject_on_worker_lost` (prod config), a task that dies with its worker (SIGKILL / OOM / node eviction) is redelivered with the **same Celery id**. A task that OOMs on *every* run would loop forever (OOM → redeliver → OOM), blocking its chord indefinitely. This caps redeliveries: `run_command` counts worker-loss redeliveries (Redis `celery-task-meta-worker-loss-<id>`); once `task_max_retries` is exceeded it **abandons** the task with a clear `abandoned after N delivery attempts` result so the chord/workflow can finish. `task_max_retries=-1` disables the cap (default). Also adds the connection-loss cancel flag. ## Validation Exercised end-to-end earlier (prefork child-loss → requeue → abandon at cap, 6/6). **`tests/unit/test_celery.py`: 23 passed** on current `main`; flake8 clean. ## Note `feature:worker-reliability`. Companion PR extracts the eviction-finalize half (was #1203). Prod workers run `-P prefork -c 1`, so the fast child-loss path is live. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a limit for repeated worker-loss retries so long-running jobs can stop retrying after a configurable cap. * Introduced an option to cancel in-flight tasks when the broker connection is lost, helping prevent stuck background work. * **Bug Fixes** * Improved handling of interrupted tasks so workflows can complete with a clear failure result instead of hanging indefinitely. * Added safer retry counting to better track redelivered tasks across worker losses. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ocervell
marked this pull request as draft
July 8, 2026 08:37
Shell commands the AI runs become first-class secator runners —
persisted as runner docs, visible in history, parented under the
conversation via `context.session_id`. Built on `origin/ai-resiliency`
(PR 3 of the AI subagent/parenting/bash-runner design).
## What's in it
- **`command` task** (`secator/tasks/command.py`) — a `Command` subclass
that runs an arbitrary command line verbatim in shell mode
(`input_types=[]` so bare commands like `whoami` aren't type-stripped;
`_build_cmd` restores `shell=True`). stdout captured via the base
runner's `self.output`.
- **`command.from_result(...)`** — a populate-without-execute import
seam (build a runner doc from an already-run command's
cmd/output/status/timing without re-running it — the future path for
importing externally-run commands into Cloud). Execute vs populate paths
cleanly separated.
- **`_handle_shell` rewired** (`secator/ai/actions.py`) — dispatches the
`command` task in-process instead of raw `subprocess.run`, so the
command persists as a runner doc (mongodb `update_runner` hooks)
parented by `session_id`, while preserving the `shell`/`shell_output`
transcript contract, the 60s timeout, and the output cap.
- **env sanitization preserved & hardened** — a backward-compatible
`env` run-opt on base `Command` (`env = self.run_opts.get('env',
os.environ)`) lets `_handle_shell` pass `_sanitized_env()` (strips
`ANTHROPIC_`/`AWS_`/`SECRET_`/API-key vars) so an AI-run `env` can't
leak creds into output; `env` is kept **runtime-only** (excluded from
persisted `resolved_opts`) so it never lands in the runner doc.
## Testing
- Unit: `tests/unit/test_command_task.py` (verbatim exec, shell
metachars, bare-command, from_result success/failure/no-exec, env
honored/not-persisted), plus rewired
`test_ai_actions.py`/`test_ai_loop.py` shell tests. Full `tests/unit`:
**1071 passed, 0 regressions** vs baseline.
- E2E: verified against a live Mongo — an AI shell command persists a
`command` runner doc with cmd/output/status, parented by `session_id`,
and the `shell_output` appears in the transcript.
Built via subagent-driven-development (per-task reviews + a final
whole-branch review; the review loop caught a shell-mode clobber,
input-type stripping, output corruption, an env-sanitization regression,
and — via the live-Mongo E2E — a silent hook-resolution no-op).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Restore an AI conversation from **one code path** for both the local (JSON) and remote (Mongo) drivers, rebuilding the **full litellm transcript including tool calls / tool results** — not just the prompt/response text it reconstructed before. Built on `ai-resiliency`; to be merged there once validated. ## How Every transcript turn persists its exact litellm message on a new `Ai.message` field. `restore_history_from_db` collects those by `session_id` (ordered by `_timestamp`) and sets `history.messages` verbatim — a faithful, valid transcript continuation, backend-agnostic. Local resume adopts the picked session's `session_id` and routes through the same restore. ## Commits 1. `Ai.message` transcript carrier + `cap_message` (BSON-safety backstop, far under 16MB). 2. Persist prompt/assistant turns as raw messages — incl. previously-dropped **tool-call-only** turns. 3. Persist tool results as new `tool_result` docs (with `runner_id`) at all 4 sites. 4. Rewrite `restore_history_from_db` to rebuild the full transcript from `message` (legacy docs → text-only fallback; orphan-repair retained). 5. Unify local resume through `restore_history_from_db` + adopt the prior `session_id`; **legacy (pre-session_id) sessions fall back to `replay_session`** so they don't restore empty. 6. Scoping / cross-backend (real JsonBackend + Mongo-mock) / size-backstop / UI-safety coverage. ## Key properties - **Encryption:** `message.content` is encrypted exactly once (persisted verbatim, no re-encryption on restore); legacy fallback re-encrypts its plaintext. A restore yields the same in-memory (encrypted) history a fresh run would. - **Scoping:** every `_type:"ai"` doc carries `_context.session_id`; restore is scoped to it (verified: zero cross-conversation bleed on both backends). - **No new truncation:** tool-result content is already bounded upstream by `truncate_to_tokens`; `cap_message` is only a hard envelope backstop. ## Testing Built via subagent-driven-development (per-task reviews + a whole-branch review that caught & fixed a blocker: legacy local sessions restoring empty). Full `tests/unit`: **0 new failures** vs the `ai-resiliency` baseline (the ~125 failures are pre-existing env issues — missing `shfmt`/`safecmd`). ## Known follow-ups (out of scope, parked) - Driver-aware spill pointer for truncated tool output (+ extend `query_workspace` to query runners, not just findings). - Rich UI rendering of `tool_result` docs in the chat transcript (secator-ui); tool-call-only turns now emit an empty-content `response` doc the UI should filter. - Local resume no longer console-prints the prior conversation for new-format sessions (small UX consideration). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI-task reliability / robustness / security hardening — the full remediation of the
2026-06-30 deep review (
docs/superpowers/specs/2026-06-30-ai-task-reliability-review.md).27 / 29 findings fixed, one focused PR per finding, merged here across 7 rounds.
H8 excluded (overlaps
feat/sensitive-options); H3 deferred to a design decision (event-drivenhuman-wait — see the review doc's H3 proposal).
🔴 Critical
dangerous=False)celery_ididempotency marker; redelivery no longer replays/re-bills a turn🟠 High
rm -rf /*-class destructive-command denyprompt_uuid(no stale auto-approve)🟡 Medium
usageis absentadd_findingtypes (block injected trusted-target minting)curl -o/wget -Ooutput paths as writesallow_allgrants real session-wide scope vs a one-shot allow🔵 Pertinence / cleanup
run_queryexampleexploitmode; wire + document itACTION_TOOL_MAP, orphan_maybe_encrypt, dead render branch)Every PR added focused tests and proved no regression against a git-stash baseline (guardrail
tests needing
shfmt/safecmdfail identically on base — CI is the arbiter). ~20 follow-upitems the agents surfaced are ranked in the review doc's "round-2 candidates" backlog — the
standout being the
permission_engine-unset full bypass (candidate Critical).