Skip to content

[codex] summarize complex synthetic benchmark in demo review#29

Merged
InfoSecHack merged 1 commit into
mainfrom
codex/wire-complex-synthetic-runner-summary
Jun 3, 2026
Merged

[codex] summarize complex synthetic benchmark in demo review#29
InfoSecHack merged 1 commit into
mainfrom
codex/wire-complex-synthetic-runner-summary

Conversation

@InfoSecHack

Copy link
Copy Markdown
Owner

Summary

  • Add a report-only complex synthetic benchmark section to summary.md from scripts/run_public_demo_review.py.
  • Add complex_synthetic_benchmark to the generated manifest with exact frozen-oracle counts and non-score flags.
  • Update focused runner tests and the public demo review runbook.

Boundaries

  • Runner/report-only integration only.
  • No fixture changes, live AWS, Terraform, AWS CLI, STS, Lambda API, or iam:PassRole calls.
  • No reasoner changes, benchmark semantic changes, live evidence claims, composite benchmark score, or pass/fail benchmark label.

Validation

  • python -m pytest -q tests/test_public_demo_review_runner.py tests/test_complex_synthetic_benchmark_fixture.py
  • python scripts/run_public_demo_review.py --out /tmp/iamscope-public-demo-review
  • grep -nEi 'Complex synthetic benchmark|42|18|validated|blocked|precondition_only|inconclusive|shared_passrole_target_resource_scope_unknown|shared_cross_account_trust_condition_unknown|shared_boundary_or_session_policy_context_missing|frozen synthetic oracle|not generated/replayed by IAMScope|composite score|pass/fail benchmark label' /tmp/iamscope-public-demo-review/summary.md
  • jq '.complex_synthetic_benchmark' /tmp/iamscope-public-demo-review/manifest.json
  • grep -RInE '[0-9]{12}' docs tests | grep -v '000000000000' || true
  • grep -RInE 'arn:aws:iam::[0-9]{12}' docs tests | grep -v '000000000000' || true
  • ind . -name 'result.json' -o -name 'terraform.tfstate*' -o -name '.terraform.lock.hcl' -o -name '*.tfplan' -o -name 'terraform-outputs.json'
  • ./scripts/check.sh
  • ./scripts/test_fast.sh
  • git diff --check

@InfoSecHack InfoSecHack marked this pull request as ready for review June 3, 2026 22:54
@InfoSecHack InfoSecHack merged commit 6f98712 into main Jun 3, 2026
6 checks passed
@InfoSecHack InfoSecHack deleted the codex/wire-complex-synthetic-runner-summary branch June 3, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant