fix: /answer judge saw empty excerpts when no reranker is configured#31
Open
andreimarinescu wants to merge 1 commit into
Open
fix: /answer judge saw empty excerpts when no reranker is configured#31andreimarinescu wants to merge 1 commit into
andreimarinescu wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
rerankText is only populated when a reranker runs. The /answer judge used it as the candidate excerpt unconditionally, so with rerank disabled every candidate was scored against an empty string and dropped — /answer always returned 'none were judged relevant'. Fall back to title + retrieval excerpt when rerankText is empty.
andreimarinescu
force-pushed
the
fix/judge-empty-excerpt
branch
from
53d5c46 to
9116f27
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/answer's LLM relevance judge usesc.rerankTextas the candidate excerpt, but that field is only populated when a reranker runs. With rerank disabled, every candidate is judged against an empty string, scored irrelevant, and dropped —/answeralways returns "none were judged relevant" even when retrieval found good candidates.Found on a local dev instance running without a reranker configured; deployments with a reranker never hit this, which is why it went unnoticed.
Fix
Extract the excerpt choice into
judgeExcerpt(): usererankTextwhen present, otherwise fall back totitle + "\n" + retrieval excerpt(the same text the synth prompt already uses). Regression test covers both paths.