Skip to content

Add adversarial-verification plugin#148

Open
wizardengineer wants to merge 4 commits into
mainfrom
adversarial-verification-plugin
Open

Add adversarial-verification plugin#148
wizardengineer wants to merge 4 commits into
mainfrom
adversarial-verification-plugin

Conversation

@wizardengineer

@wizardengineer wizardengineer commented Apr 17, 2026

Copy link
Copy Markdown

Summary

Adds a new adversarial-verification plugin that stress-tests claims, designs, and bug findings by dispatching two isolated sub-agents (advocate + skeptic) and synthesizing their arguments into a structured verdict. Counters sycophancy and single-agent agreement bias by forcing maximal disagreement before the caller commits.

What it does

Two modes:

  • Decision mode — free-form arguments organized by evaluation dimensions (for approach/design choices)
  • Proof mode — N null hypotheses the skeptic tries to prove and the advocate tries to refute (for verifying bug findings and security claims)

Core principle: isolated sub-agent contexts are non-negotiable. An agent that sees the other side's arguments will soften to accommodate them. The adversarial value comes from each agent arguing without knowledge of the counter-argument, with synthesis happening separately.

Structure

  • SKILL.md (main entry point with decision tree for mode selection)
  • references/decision-mode.md (structure for approach selection)
  • references/proof-mode.md (N-null-hypothesis structure for finding verification)
  • references/prompt-templates.md (advocate/skeptic templates enforcing anti-balance)
  • references/synthesis.md (verdict table format + recommendation structure)
  • references/anti-patterns.md (10 common failure modes with diagnoses)

When to use

  • Choosing between competing technical approaches
  • Verifying a bug finding is real (not a false positive)
  • Reviewing a design decision before commit
  • Any claim the caller is inclined to agree with by default

Test plan

  • python3 .github/scripts/validate_codex_skills.py passes (verified locally)
  • Plugin registered in .claude-plugin/marketplace.json
  • CODEOWNERS entry added
  • README table entry added under Verification section
  • Codex symlink created at .codex/skills/adversarial-verification
  • Install the plugin locally and invoke the skill on a real claim
  • Verify both decision-mode and proof-mode paths produce useful verdicts

🤖 Generated with Claude Code

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@wizardengineer wizardengineer marked this pull request as ready for review April 17, 2026 07:35
@wizardengineer wizardengineer requested a review from dguido as a code owner April 17, 2026 07:35

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@wizardengineer wizardengineer marked this pull request as draft April 17, 2026 17:31
Stress-tests claims, designs, and bug findings by dispatching two
isolated sub-agents (advocate + skeptic) and synthesizing their
arguments into a structured verdict. Counters sycophancy and single-
agent agreement bias by forcing maximal disagreement before commit.

Two modes:
- Decision mode: free-form arguments organized by evaluation dimensions
  for approach/design choices
- Proof mode: N null hypotheses the skeptic proves and advocate refutes,
  for verifying bug findings and security claims

Includes SKILL.md + 5 reference docs:
- anti-patterns.md (10 common failure modes with diagnoses)
- decision-mode.md (structure for approach selection)
- proof-mode.md (N-null-hypothesis structure for finding verification)
- prompt-templates.md (advocate/skeptic templates enforcing anti-balance)
- synthesis.md (verdict table format and recommendation structure)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@wizardengineer wizardengineer force-pushed the adversarial-verification-plugin branch from 7f05877 to 833dbef Compare April 17, 2026 17:46
@dguido

dguido commented May 8, 2026

Copy link
Copy Markdown
Member

@claude review once

Comment thread plugins/adversarial-verification/skills/adversarial-verification/SKILL.md Outdated
@wizardengineer wizardengineer marked this pull request as ready for review May 11, 2026 16:45

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@dguido

dguido commented Jun 29, 2026

Copy link
Copy Markdown
Member

@claude review once

@@ -0,0 +1 @@
../../plugins/adversarial-verification/skills/adversarial-verification No newline at end of file

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This PR adds a .codex/skills/adversarial-verification symlink, but AGENTS.md:41 explicitly prohibits .codex/ sidecar directories — Codex already supports the canonical .claude-plugin/marketplace.json directly (AGENTS.md:35-37). The README.md update in this same PR even states "Codex supports Claude plugin marketplaces directly, so this repository does not need Codex-specific sidecar metadata," directly contradicting the new symlink. Fix: delete the .codex/ directory entirely; the marketplace entry already added to .claude-plugin/marketplace.json handles Codex discovery.

Extended reasoning...

What the bug is

The PR creates a new top-level .codex/ directory containing a symlink:

.codex/skills/adversarial-verification -> ../../plugins/adversarial-verification/skills/adversarial-verification

AGENTS.md:41 (which is @-included by CLAUDE.md and is therefore the binding contributor guide) explicitly states:

Do not add .agents/plugins/marketplace.json, .codex/, or plugins/<name>/.codex-plugin/.

AGENTS.md:35-37 explains the rationale: "This repository uses Claude plugin marketplace metadata as the canonical source for both Claude Code and Codex. Codex supports .claude-plugin/marketplace.json and plugins/<name>/.claude-plugin/plugin.json directly, so do not add duplicate Codex-only sidecar metadata."

Why existing code does not prevent it

There is no CI check that fails on .codex/ paths — validate_plugin_metadata.py and check_codex_loadability.py validate plugin contents, not the absence of forbidden sidecar directories. The rule lives only in the contributor guide, and the PR's test plan even lists "Codex symlink created at .codex/skills/adversarial-verification" as a deliberate step, indicating the author was working from outdated knowledge of the repo conventions. (Relatedly, the test plan references validate_codex_skills.py, which does not exist — .github/scripts/ only contains check_claude_loadability.py, check_codex_loadability.py, and validate_plugin_metadata.py.)

Step-by-step proof

  1. Read AGENTS.md:41 → forbidden patterns include .codex/.
  2. find . -maxdepth 5 -name '.codex*' on the PR branch returns exactly one hit: the new .codex/skills/adversarial-verification from this PR. No other plugin in the marketplace uses this pattern.
  3. The PR also updates README.md to add: "Codex supports Claude plugin marketplaces directly, so this repository does not need Codex-specific sidecar metadata." This is the same PR that adds the sidecar — the documentation update directly contradicts the symlink it ships alongside.
  4. The plugin is also registered in .claude-plugin/marketplace.json (lines 395-406 of the diff), which is the canonical discovery mechanism Codex consumes via marketplace compatibility. The symlink adds nothing functional.

Impact

Sets a bad precedent. If merged, future plugin contributors will see .codex/ in the tree and replicate the pattern, drifting the repo away from the single-source-of-truth Claude marketplace metadata that AGENTS.md establishes. There is no functional benefit — Codex already discovers the plugin through .claude-plugin/marketplace.json.

How to fix

Delete the .codex/ directory entirely:

git rm .codex/skills/adversarial-verification
rmdir .codex/skills .codex

The plugin remains discoverable for both Claude Code and Codex through the marketplace entry already added in this PR. Also remove the "Codex symlink created at .codex/skills/adversarial-verification" line from the PR description's test plan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants