Skip to content

[codex] design complex synthetic benchmark fixture#27

Merged
InfoSecHack merged 1 commit into
mainfrom
codex/complex-synthetic-benchmark-design
Jun 3, 2026
Merged

[codex] design complex synthetic benchmark fixture#27
InfoSecHack merged 1 commit into
mainfrom
codex/complex-synthetic-benchmark-design

Conversation

@InfoSecHack

Copy link
Copy Markdown
Owner

Summary

  • Add a docs-only design for complex_shared_uncertainty_iam_benchmark_001.
  • Define the future synthetic graph shape, verdict categories, shared uncertainty classes, fixture files, tests, and runner integration plan.
  • Preserve local-only, bounded-evidence, non-score, non-pass/fail framing.

Boundaries

  • Design/spec only, no fixture JSON, tests, runner changes, reasoner changes, benchmark semantic changes, live AWS, Terraform, AWS CLI, STS, Lambda API, or iam:PassRole calls.
  • No new live evidence claims, broad correctness claims, exploitability claims, composite benchmark score, or pass/fail benchmark label.

Validation

  • grep -nEi 'complex_shared_uncertainty_iam_benchmark_001|validated|blocked|precondition-only|inconclusive|shared_passrole_target_resource_scope_unknown|shared_cross_account_trust_condition_unknown|shared_boundary_or_session_policy_context_missing|Recommended next slice|not broad IAMScope correctness|not exploitability|no composite benchmark score|no pass/fail benchmark label' docs/specs/complex-synthetic-benchmark-design.md
  • grep -RInE '[0-9]{12}' docs tests | grep -v '000000000000' || true
  • grep -RInE 'arn:aws:iam::[0-9]{12}' docs tests | grep -v '000000000000' || true
  • ind . -name 'result.json' -o -name 'terraform.tfstate*' -o -name '.terraform.lock.hcl' -o -name '*.tfplan' -o -name 'terraform-outputs.json'
  • ./scripts/check.sh
  • ./scripts/test_fast.sh
  • git diff --check

@InfoSecHack InfoSecHack marked this pull request as ready for review June 3, 2026 22:13
@InfoSecHack InfoSecHack merged commit 419c8a2 into main Jun 3, 2026
6 checks passed
@InfoSecHack InfoSecHack deleted the codex/complex-synthetic-benchmark-design branch June 3, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant