The provider adapters (OpenAI, Anthropic, Google, Ollama) are only tested against mocks — the httptest unit tests and the --llm=mock testscripts. Nothing hits the real APIs, so a response-format change on a provider side goes unnoticed until a user hits it.
Add a scheduled (nightly) workflow that runs vens generate against each real provider using keys from repo secrets, on a small report, and asserts a valid scored VEX. Gate on the secrets being present so it never runs on PRs.
The provider adapters (OpenAI, Anthropic, Google, Ollama) are only tested against mocks — the httptest unit tests and the
--llm=mocktestscripts. Nothing hits the real APIs, so a response-format change on a provider side goes unnoticed until a user hits it.Add a scheduled (nightly) workflow that runs
vens generateagainst each real provider using keys from repo secrets, on a small report, and asserts a valid scored VEX. Gate on the secrets being present so it never runs on PRs.