Add API/export contract enforcement checks#24
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new synthetic API/dashboard/export contract fixture along with automated generation and validation scripts. Key additions include scripts/generate_contract_fixtures.py and scripts/validate_api_exports.py, which are integrated into the agent workflow and CI guardrails to ensure deterministic, privacy-compliant validation without requiring a live server. Feedback focuses on significant logic duplication between the new validation script and existing tools, missing support for numeric types in the validator, and redundant manual checks for array fields that are already covered by schema-based validation.
| #!/usr/bin/env python3 | ||
| """Validate synthetic API/dashboard/export fixtures against the contract schema. | ||
|
|
||
| This is a lightweight, deterministic validator for the generated fixture. It uses | ||
| only the Python standard library and performs a small subset of JSON Schema-like | ||
| checks that match the current machine-readable contract needs. | ||
| """ |
There was a problem hiding this comment.
This script introduces significant logic duplication with scripts/validate_contracts.py (e.g., JSON loading, type checking, and schema validation loops). The only unique validation performed here is the enforcement of the synthetic: true flag. To improve maintainability and ensure a single source of truth for contract validation, consider integrating the synthetic check into scripts/validate_contracts.py and removing this redundant script.
| SUPPORTED_TYPES = {"string", "boolean", "object", "array"} | ||
| ARRAY_FIELDS = {"api_routes", "dashboard_views", "export_formats", "report_integration_points"} |
There was a problem hiding this comment.
The SUPPORTED_TYPES set is missing number and integer, which are supported in scripts/validate_contracts.py. This inconsistency makes the validator fragile if the schema is updated to include numeric fields. Additionally, the ARRAY_FIELDS set and its associated manual check (lines 110-112) are redundant because the schema-based validation loop (lines 97-105) already verifies that these fields match the array type defined in the schema. Note that ARRAY_FIELDS also currently omits security_notes.
| SUPPORTED_TYPES = {"string", "boolean", "object", "array"} | |
| ARRAY_FIELDS = {"api_routes", "dashboard_views", "export_formats", "report_integration_points"} | |
| SUPPORTED_TYPES = {"string", "number", "integer", "boolean", "object", "array"} |
| def type_name(value: Any) -> str: | ||
| if isinstance(value, bool): | ||
| return "boolean" | ||
| if isinstance(value, str): | ||
| return "string" | ||
| if isinstance(value, dict): | ||
| return "object" | ||
| if isinstance(value, list): | ||
| return "array" | ||
| if value is None: | ||
| return "null" | ||
| return type(value).__name__ | ||
|
|
||
|
|
||
| def matches_type(value: Any, expected: str) -> bool: | ||
| if expected == "string": | ||
| return isinstance(value, str) | ||
| if expected == "boolean": | ||
| return isinstance(value, bool) | ||
| if expected == "object": | ||
| return isinstance(value, dict) | ||
| if expected == "array": | ||
| return isinstance(value, list) | ||
| return False |
There was a problem hiding this comment.
These utility functions are duplicated from scripts/validate_contracts.py but lack support for numeric types. If this script is retained, these functions should be updated to handle int and float (mapping to integer and number) to maintain consistency across the repository's validation tools.
def type_name(value: Any) -> str:
if isinstance(value, bool):
return "boolean"
if isinstance(value, str):
return "string"
if isinstance(value, int):
return "integer"
if isinstance(value, float):
return "number"
if isinstance(value, dict):
return "object"
if isinstance(value, list):
return "array"
if value is None:
return "null"
return type(value).__name__
def matches_type(value: Any, expected: str) -> bool:
if expected == "string":
return isinstance(value, str)
if expected == "number":
return isinstance(value, (int, float)) and not isinstance(value, bool)
if expected == "integer":
return isinstance(value, int) and not isinstance(value, bool)
if expected == "boolean":
return isinstance(value, bool)
if expected == "object":
return isinstance(value, dict)
if expected == "array":
return isinstance(value, list)
return False| for field in sorted(ARRAY_FIELDS): | ||
| if field in fixture and not isinstance(fixture[field], list): | ||
| errors.append(f"fixture field {field} must be an array") |
There was a problem hiding this comment.
As noted previously, this manual check for array fields is redundant. The validation loop at lines 97-105 already checks the fixture fields against the types specified in contracts/api-dashboard.schema.json. Since the schema defines these fields as array, they are already being verified.
1 participant
Motivation
contracts/to catch structural regressions early.main(links issue Add API/export contract enforcement checks #23).Description
scripts/generate_contract_fixtures.pywhich emits a deterministic synthetic fixture atcontracts/examples/api-dashboard.example.json(withgenerated_at: "2026-01-01T00:00:00Z") and writesdocs/reports/contract-fixture-generation-report.md.scripts/validate_api_exports.pywhich loadscontracts/api-dashboard.schema.jsonand the generated example and performs lightweight standard-library checks for required fields, simple types,synthetic: true, and thatapi_routes,dashboard_views,export_formats, andreport_integration_pointsare arrays, then writesdocs/reports/api-export-validation-report.md..github/workflows/agent-checks.ymltopy_compilethe new scripts and to runpython scripts/generate_contract_fixtures.pyandpython scripts/validate_api_exports.pyduring CI.docs/API_SURFACE.mdanddocs/AGENT_WORKFLOW.mdto document the new scripts, the examplecontracts/examples/api-dashboard.example.json, and the generated reports indocs/reports/.Testing
python -m py_compile scripts/generate_contract_fixtures.py scripts/validate_api_exports.py(passed).python scripts/generate_contract_fixtures.pyto generatecontracts/examples/api-dashboard.example.jsonanddocs/reports/contract-fixture-generation-report.md(passed).python scripts/validate_api_exports.pyto validate the generated fixture againstcontracts/api-dashboard.schema.jsonand producedocs/reports/api-export-validation-report.md(passed).python scripts/validate_contracts.pyto refresh the contract validation report including the new example (passed).Codex Task