Skip to content

[codex] add complex synthetic benchmark fixture#28

Merged
InfoSecHack merged 1 commit into
mainfrom
codex/complex-synthetic-benchmark-fixture
Jun 3, 2026
Merged

[codex] add complex synthetic benchmark fixture#28
InfoSecHack merged 1 commit into
mainfrom
codex/complex-synthetic-benchmark-fixture

Conversation

@InfoSecHack

Copy link
Copy Markdown
Owner

Summary

  • Add the local-only frozen synthetic oracle fixture complex_shared_uncertainty_iam_benchmark_001.
  • Pin 42 naive candidates, 18 findings, verdict breakdown 4/5/3/6, and uncertainty group counts 3/2/1.
  • Add focused oracle tests for fixture shape, synthetic-only IDs/ARNs, metadata, non-claims, pattern coverage, and artifact hygiene.

Boundaries

  • Fixture/test slice only, no runner integration yet.
  • No live AWS, Terraform, AWS CLI, STS, Lambda API, or iam:PassRole calls.
  • No reasoner changes, benchmark semantic changes, live evidence claims, composite benchmark score, or pass/fail benchmark label.
  • indings.json is explicitly static_fixture_authoring and rozen_synthetic_oracle, not generated/replayed by IAMScope.

Validation

  • python -m pytest -q tests/test_complex_synthetic_benchmark_fixture.py
  • grep -RInE '[0-9]{12}' docs tests | grep -v '000000000000' || true
  • grep -RInE 'arn:aws:iam::[0-9]{12}' docs tests | grep -v '000000000000' || true
  • ind . -name 'result.json' -o -name 'terraform.tfstate*' -o -name '.terraform.lock.hcl' -o -name '*.tfplan' -o -name 'terraform-outputs.json'
  • ./scripts/check.sh
  • ./scripts/test_fast.sh
  • git diff --check

@InfoSecHack InfoSecHack marked this pull request as ready for review June 3, 2026 22:36
@InfoSecHack InfoSecHack merged commit 34cec86 into main Jun 3, 2026
6 checks passed
@InfoSecHack InfoSecHack deleted the codex/complex-synthetic-benchmark-fixture branch June 3, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant