[Feature] gitagent test -- run against example scenarios

[shreyas-lyzr]

Summary

Add a gitagent test command that runs an agent against predefined test scenarios.

Problem

There is no standard way to verify that an agent behaves correctly before deploying it. Agent developers need a testing framework analogous to unit tests for code.

Proposed Solution

# Run all tests for the current agent
gitagent test

# Run a specific test scenario
gitagent test --scenario greeting

# Run with a specific adapter
gitagent test --adapter openai

Test scenarios would live in tests/ or examples/ within the agent repo:

# tests/greeting.yaml
name: greeting-test
input: "Hello, who are you?"
expect:
  contains: ["agent name"]
  tone: professional
  max_tokens: 500

Acceptance Criteria

• gitagent test command implemented
• YAML-based test scenario format defined
• Works with multiple adapters
• Clear pass/fail output
• Documentation added

[shreyas-lyzr]

Apologies for the long silence — this is still on the roadmap and it's still a real gap.

Updated thinking: gapman test should sit closer to gapman audit semantically — both are validation, the difference is audit is static (against the spec) and test is behavioral (against scenarios).

Minimum viable version I'd accept:

gapman test                 # run all scenarios in tests/
gapman test --scenario X    # one scenario
gapman test --adapter <a>   # pick runtime
gapman test --record        # capture outputs as new golden
gapman test --golden        # compare against captured outputs

Where a scenario is tests/<name>/{input.md, expected.md, fixtures/}.

Open implementation questions worth answering before code:

1. Determinism. LLM outputs aren't reproducible byte-for-byte. Default comparison should be either (a) keyword-presence checks or (b) LLM-as-judge with a separate, locked model. Both are valid; default should be (a) — cheaper and more deterministic.
2. Runtime cost. gapman test will burn API tokens; CI usage needs an opt-in --ci mode that runs against cached fixtures rather than live models.
3. Hook mocking. A pre_tool_use hook with fail_open: false is meant to enforce — in tests, it must run with the real semantics, not be skipped.

Tagging this good first issue for anyone who wants to take it on. Happy to review a small first PR that just gets the verb wired up + runs a single scenario against examples/standard/ — and we can iterate from there.

[shreyas-lyzr]

Strong feature request — a testing layer is a natural complement to gitagent validate. A few design questions to resolve before implementation:

Test scenario format — where do test scenarios live? Proposing tests/scenarios/<name>.yaml with fields: prompt, expected_contains (substring match), expected_not_contains, expected_tool_calls (list of tool names that must be invoked), and max_turns. This keeps tests git-native and diffable.

Determinism — LLM outputs are non-deterministic. The test runner should support a --seed flag (where the adapter supports it) and assertion modes: contains, regex, schema (for structured output agents). Avoid exact-match assertions by default.

Adapter parity — gitagent test --adapter openai implies the test runner dispatches through the same adapter path as gitagent run. That is the right design. Make sure the test command reuses the runner infrastructure from src/runners/ rather than duplicating it.

CI integration — the test command should exit non-zero on any failed scenario so it works as a CI gate. A --reporter json flag for machine-readable output would also be useful for dashboards.

Happy to review a draft schema or implementation PR.

[shreyas-lyzr]

A gitagent test command is a natural fit alongside the existing export/run commands. Key design questions before implementation:

• Test scenarios should probably live in a tests/ directory inside the agent repo (e.g., tests/greeting.yaml) so they are version-controlled alongside the agent definition.
• The scenario format needs at least: an input prompt, expected output criteria (exact match, regex, semantic similarity threshold), and optionally a set of adapter-specific overrides.
• Semantic similarity checking requires an embedding model call — for offline/CI use cases, regex or substring matching should be the default with semantic as opt-in.
• The --adapter flag is important because behavior can differ between adapters; tests should ideally be adapter-agnostic by default but allow adapter-specific overrides.

PR #81 (LangChain adapter) and PR #80 (LangGraph/DeepAgents) are close to merge — once those land, a test command that covers all adapters uniformly would be high value. Assigning @shreyas-lyzr to track.