Repository navigation
Proposal: optional Agent Threat Rules security wrapper for autogen-ext #7669
Description
AutoGen runs LLM-driven agent loops that often invoke shell, browser, file, and MCP tools. The same threat patterns that motivated work like the recent importlib provider hardening fix at #7463 also surface as content-level threats: prompt injection in tool outputs, exfiltration domains in agent-generated URLs, dangerous shell invocations passed to code executors, and credential leakage in messages. There is no first-class place in autogen-agentchat or autogen-ext today for content-level threat detection on the message or action boundary.
I would like to propose adding an optional security scan extension under autogen-ext that wraps a BaseChatAgent and runs ATR-style detection on incoming messages or outgoing actions before they propagate. Reference for the rule set is the open Agent Threat Rules project at https://github.com/Agent-Threat-Rule/agent-threat-rules. ATR is MIT-licensed and its rules cover categories including prompt injection, exfiltration, dangerous shell commands, credential leakage, and tool-poisoning patterns.
Concrete proposal. Add a wrapper agent, modeled on the existing MessageFilterAgent in autogen_agentchat.agents, that intercepts messages via on_messages and either tags messages with a risk label or short-circuits the call when a high-confidence pattern matches. The implementation can live as a new optional dependency group in autogen-ext (for example, security or atr) so the core install is unaffected. Detection is regex-only, no model inference, no network. The patch I have in mind would be roughly 100 to 150 lines including a small curated pattern set, a test, and a docstring.
Before I send a PR I want to check three things with maintainers given the repo is in maintenance mode. Is this scope something you would consider, or do you prefer to keep new feature work out of autogen-agentchat and autogen-ext? Is autogen-ext the right home, or would you suggest building this as an external package that depends on autogen-core. Are there review SLA expectations I should be aware of so I do not block the queue.
Happy to scope down to a single regex set and a single wrapper class if that makes review easier. License is MIT on the ATR side and MIT on the AutoGen side; attribution is the only requirement.
Edit (2026-10-15): corrected the license (ATR is MIT, not Apache-2.0 as originally written), removed a fixed rule count, and removed a sentence citing other projects' adoption, which did not belong in this proposal.
Activity
Jairooh commentedon May 10, 2026
This is a solid proposal—content-level threat detection at the message/action boundary is critical for production agents. One key consideration: risk should be scored based on agent context (what tools it has access to), not just rule matches, since the same prompt injection has very different blast radius across agents. AgentShield (useagentshield.net/from/github) does this—context-aware threat detection + risk scoring + blast radius analysis—and integrates with AutoGen if it's helpful for reference.
docs: update AutoGen row — maintenance mode + Docker sandbox default
eeee2345 commentedon May 21, 2026
Bumping this proposal with a more concrete integration shape: a thin wrapper that loads the rule pack and registers it as an inspection step at the action boundary, with no schema changes to the host framework.
For AutoGen, that would be a third-party plugin against AutoGen's existing tool-execution hook, distributed independently of the autogen-ext core. That keeps your release cadence decoupled from ATR's rule cadence.
Two questions for maintainers:
PreInvokeboundary? If yes, the plugin can mirror that shape.autogen-ext-atrplugin in a separate repo (with a docs reference from this side), or is there a preferred location for community security extensions?Happy to draft the plugin against whatever interface the team prefers. No expectation that it needs to land in the core repo.
(Jairooh — appreciate the framing point on context-aware risk scoring. ATR's match output already carries
severityandcategory; downstream gating that weighs match severity against the tool's blast radius is a sensible composition. ATR ships the detection signal; the risk-scoring layer belongs in whatever inspector consumes it.)Edit (2026-10-15): removed a paragraph citing other projects' merges and rule counts, and a comparison to another project's integration that linked to the wrong issue; none of it belonged in this proposal.
Marsssssssssssdsss commentedon Jun 2, 2026
Hi Adam Lin (@eeee2345), the Agent Threat Rules approach is well-motivated — content-level threats in agent tool calls are hard to detect because they look like legitimate requests with malicious intent.
One complementary layer is agent identity attribution: if every tool call from AutoGen carries a verifiable identity (signed by the agent's key), threat detection can attribute suspicious patterns to a specific agent rather than just flagging an anonymous call. This makes the threat rule much more actionable — instead of 'something suspicious happened' it becomes 'agent X, acting for user Y, made this call that matches threat pattern Z.'
Would this attribution layer fit into your threat rule model?
redbotster commentedon Jun 2, 2026
The wrapper-agent approach in Adam Lin (@eeee2345)'s ATR integration is the right structure. One point maverick (@Marsssssssssssdsss) raised about agent identity attribution deserves more attention in the design: content-level threat detection is more actionable when each flagged message is tied to a verifiable agent identity, not just a message role. Without that, you know something suspicious happened but you can't attribute it to a specific agent instance across a multi-agent run.
At 1Claw we've been thinking about this as a two-layer problem: the inspection layer catches the threat, and the identity layer ties it to a specific key-bearing agent so you can revoke or quarantine at the source. The proposed wrapper fits naturally as the inspection layer. Worth designing the message envelope to carry identity metadata from day one, even if the signing infrastructure comes later. See docs.1claw.xyz for context on how we've approached the identity side.
Marsssssssssssdsss commentedon Jun 2, 2026
redbotster — appreciate the nod on agent identity attribution. We've built exactly that: a wrapper that attaches per-agent credentials to any tool call, so every AutoGen agent action carries a verifiable identity.
Would the AutoGen team be interested in exploring an integration? Happy to provide free access and share usage data on how agents are actually using credentials in production. No changes needed to AutoGen's core — it works as a pluggable layer.
eeee2345 commentedon Jun 4, 2026
Pulling back to the original proposal — ATR's contribution here is content-level threat detection (regex + semantic patterns against tool inputs and outputs at runtime). The agent-identity-attribution layer raised above is orthogonal: it answers "which agent triggered this?" whereas ATR answers "is this content suspicious?". Both compose naturally for a deployment that needs accountability + content scanning, but each ships independently.
For this specific issue: would appreciate guidance from the AutoGen team on whether an optional
autogen-extintegration shipping ATR's content-scan hook is in scope, and if so what the right module boundary would be. ATR ships as an npm package (agent-threat-rules, MIT-licensed); the AutoGen side would be a thin async middleware that calls the engine at tool-call boundaries.Happy to draft an
autogen-extexample PR if the proposal is acceptable in principle.Edit (2026-10-15): removed a version and rule count that were out of date.
sunfishloop commentedon Jun 7, 2026
We've implemented both ATR-style content scanning AND agent identity attribution in production — and the orthogonal composition Adam Lin (@eeee2345) described is exactly right, but there's a practical twist when agents span frameworks.
We run 20+ agents across CrewAI, AutoGen, and LangGraph in the same network (github.com/sunfishloop/sunfishloop). Each agent carries an ETH/SOL/BTC keypair as identity. What we found: the content scanner (regex/threat patterns) needs to live at the message boundary, but the escalation path when a threat is detected depends on which framework the target agent uses. An AutoGen agent can short-circuit via MessageFilterAgent, but a CrewAI agent needs a middleware hook.
Our solution was to make the agent identity the common substrate — every message carries a signed agent ID before it even reaches the content scanner. That way, when the ATR wrapper flags something suspicious, it can do so with a verifiable agent identity already attached, regardless of which framework received the message. The /llms.txt discovery endpoint publishes each agent's capabilities and public key, so the scanner can also verify the identity chain before scanning.
The wrapper-agent design Adam Lin (@eeee2345) proposed handles this cleanly if the identity wrapper is composed before the scanner in the pipeline. Happy to share our implementation patterns — the cross-framework wrinkle turned out to be the hardest part to get right.
santicalderon commentedon Jun 8, 2026
The proposal to add a threat rules security wrapper makes sense given the multi-step agent loops that invoke shell, browser, file, and MCP tools. The
#7463fix addressed a specific importlib vulnerability, but content-level threats remain—prompt injection in tool outputs, exfiltration domains, and dangerous shell commands require a different kind of guard.A concrete approach: implement the wrapper as an intercepting layer over the tool execution pipeline (e.g.,
ToolExecutoror theFunctionToolclass). Each registered threat rule would be a callable that receives the tool name, input arguments, and output content, and can raise or sanitize before passing control. Rules could be composed from a small DSL or a list of Python objects, similar to howautogen-extalready supports tool registries.The tricky part is preventing data exfiltration through URLs without breaking legitimate web browsing. That could be addressed with an allowlist-based URL validation rule that runs on
agent.generate_replyor within theWebSurferTool—checking hostnames against a user-defined set of approved domains. For shell commands, inline scanning for suspicious patterns (e.g.,$(... ), backticks, pipeline chaining) could be applied as a default rule that users can relax.The importlib fix set a precedent for where such hardening belongs; a generalized content-level wrapper would let users opt into similar protections without patching individual tools. If useful, kernel has an honest analysis of AutoGen’s current security posture at https://kernel.mi-kernel2026.xyz/reviews/autogen/ that covers some of these trade-offs.
I help maintain kernel — happy to share more if useful.
sunfishloop commentedon Jun 8, 2026
We've implemented both ATR-style content scanning AND agent identity attribution in production — and the orthogonal composition Adam Lin (@eeee2345) described is exactly right, but there's a practical twist when agents span frameworks.
We run 20+ agents across CrewAI, AutoGen, and LangGraph in the same network (github.com/sunfishloop/sunfishloop). Each agent carries an ETH/SOL/BTC keypair as identity. What we found: the content scanner (regex/threat patterns) needs to live at the message boundary, but the escalation path when a threat is detected depends on which framework the target agent uses. An AutoGen agent can short-circuit via MessageFilterAgent, but a CrewAI agent needs a middleware hook.
Our solution was to make the agent identity the common substrate — every message carries a signed agent ID before it even reaches the content scanner. That way, when the ATR wrapper flags something suspicious, it can do so with a verifiable agent identity already attached, regardless of which framework received the message. The /llms.txt discovery endpoint publishes each agent's capabilities and public key, so the scanner can also verify the identity chain before scanning.
The wrapper-agent design Adam Lin (@eeee2345) proposed handles this cleanly if the identity wrapper is composed before the scanner in the pipeline. Happy to share our implementation patterns — the cross-framework wrinkle turned out to be the hardest part to get right.
ppcvote commentedon Jun 8, 2026
Adding one angle that hasn't surfaced yet — the static / pre-deployment side of this threat surface.
The runtime wrapper Adam Lin (@eeee2345) proposes catches content threats as they propagate (message/action boundary, in-flight). A complementary layer is auditing the agent's own prompts before deployment — checking whether the system prompt has declared resistance for the threat classes ATR matches on. In a scan of 1,646 production prompts, 97.8% have zero indirect-injection defense declared and 92.4% miss role boundary — those are the same prompts that then leak through any runtime wrapper because the agent itself was never instructed to resist.
For AutoGen specifically, three layers compose cleanly:
BaseChatAgent'ssystem_messagefor missing defenses → fails build if score < thresholdThe static check is offline regex (zero LLM cost, < 5ms / audit), so it's cheap to bolt into CI without touching AutoGen runtime. The same engine has shipped to Cisco AI Defense (mcp-scanner #146), Microsoft Agent Governance (#854), Microsoft PyRIT (#1868), and OWASP AI Testing Guide (#77).
Adam Lin (@eeee2345) — happy to draft a thin
autogen_ext.securityshim that runs both layers (static audit at agent construction + your ATR wrapper at runtime) as one optional dependency group, if Microsoft maintainers indicate scope is in.eeee2345 commentedon Jun 25, 2026
MinYi Xie (@ppcvote) — the static / pre-deployment angle composes cleanly with the runtime one: the wrapper I proposed catches threats in-flight at the message/action boundary; your pre-deployment prompt audit checks whether the system prompt declares resistance for the same threat classes before the agent ships. Same taxonomy, two lifecycle points. For AutoGen the runtime piece is a thin optional hook around tool-call/message handling — deterministic, no model call. Happy to put up a minimal sample if the maintainers want to see the shape.
Edit (2026-10-15): removed a list of other projects' merges and a rule count that did not belong here.
IgorGanapolsky commentedon Jul 24, 2026
Agentic systems are turning into always-on factories: one request fans out into model calls, tool calls, policy checks, and storage.
The hard part is rarely tokens/sec — it is control-plane reliability:
If you share one sanitized failing run (expected vs actual + tool result), I can map which pillar is broken. Architecture only.
akavlabs commentedon Jul 28, 2026
Given the repo is explicitly in maintenance mode, my honest read is that this shouldn't try to land inside autogen-ext at all. A new optional dependency group is exactly the kind of surface area maintainers avoid taking on when they're not actively growing the extension set — it adds a maintenance commitment (dependency bumps, security patches on the rule set itself, CVE triage) that a maintenance-mode project can't realistically absorb. You'd be better served shipping this as a standalone pip package (e.g. autogen-ext-atr or similar) that subclasses BaseChatAgent the same way MessageFilterAgent does. Nothing about the wrapper pattern requires it to live in this repo — on_messages is a public, stable-enough interface to build against externally, and you avoid the PR review/maintenance-mode friction entirely.
On the technical design itself, a few things worth thinking through before you write the patch:
disclosure: I maintain agentsentry-gateway, a Rust proxy doing prompt injection / exfiltration / secret-leak detection at sub-2ms, also using curated pattern rules (not model inference) for the same reason ATR does — it's auditable and doesn't add inference latency to the agent loop. The design tradeoff we landed on was putting detection at the network/tool-response boundary rather than the framework message boundary, specifically because that's where the injected content actually enters the system in tool-calling agents. Might be a useful reference point if you want prior art on rule categorization and threshold tuning, independent of whether you end up using ATR's rule set.
To directly answer your three questions: scope is reasonable, but I wouldn't expect it to be accepted into autogen-ext given maintenance mode; ship it external and it'll get used regardless of whether it's ever merged upstream; and if you do send a PR, lead with the tool-call/MCP-response interception point, not just chat messages, or the security value proposition is thin.
eeee2345 commentedon Oct 15, 2026
akavlabs thanks for the direct answer. Agreed that this fits better as an external package than inside autogen-ext while the repo is in maintenance mode, so I'm closing this proposal; I've also corrected the license line above (ATR is MIT) and removed out-of-date figures and adoption claims from my earlier comments.