Implement cross-model replay CLI#86
Conversation
AI Quality Gate ReportMode:
Duplication
Security
Logic
Advisory mode does not block merge. Review findings before strict mode is enabled. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af3afab447
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| provider=target.provider, | ||
| model=target.model, | ||
| ) |
There was a problem hiding this comment.
When a replay includes two targets from the same provider with different models (for example --targets openai:gpt-4o,openai:gpt-4o-mini), both calls share the same LLMRouter; LLMRouter._get_adapter caches adapters only by provider (adapters/router.py:293-326), so whichever model creates the provider adapter first is reused for the other target and the diff silently compares two calls to the same model. This undermines the main cross-model replay use case; use a per-target router/adapter cache key or otherwise force the requested model per call.
Useful? React with 👍 / 👎.
| target = raw_target.strip() | ||
| if not target: | ||
| continue | ||
| separator = ":" if ":" in target else "/" if "/" in target else "" |
There was a problem hiding this comment.
For common Ollama targets that include a tag, such as ollama/llama3.1:70b, this separator choice sees the : in the model tag first and splits the provider as ollama/llama3.1, causing parse_model_targets to reject the target as an unknown provider. Since the parser documents slash-form targets specifically for provider/model input, check for / before falling back to : so tagged Ollama models can be replayed.
Useful? React with 👍 / 👎.
|
|
||
| from adapters.base import LLMMessage, LLMProvider | ||
|
|
||
| DEFAULT_REPLAY_ROOT = Path(".titan") / "replays" |
There was a problem hiding this comment.
With this default, titan replay capture writes prompt and context JSON under the current repo, but the repo's .gitignore does not exclude .titan/; in normal workflows those captured prompts or context files become untracked files that can be added and committed accidentally. Since replay inputs may include security-review prompts, source snippets, or secrets, use an ignored location (for example under the user's home directory) or add the replay path to .gitignore.
Useful? React with 👍 / 👎.
1 participant
Summary
Implements the cross-model replay/diff workflow described in
docs/cross-model-replay.md.Changes
titan.replayfor JSON-backed replay capture, multi-target dispatch, stored run results, and pairwise diff generation.titan replay capture,titan replay run, andtitan replay diffCLI commands.Related Issues
Fixes #26
Testing
uv run --python 3.11 --with '.[dev,dashboard,auth,ratelimit]' ruff check .uv run --python 3.11 --with '.[dev,dashboard,auth,ratelimit]' mypy .uv run --python 3.11 --with '.[dev,dashboard,auth,ratelimit]' pytest tests/test_replay.py tests/test_adapters.pyUV_NO_CACHE=1 uv run --python 3.11 --with '.[dev,dashboard,auth,ratelimit]' titan replay --helpChecklist