Fix address-reviews duplicate detection and add real-data evals#558
Open
bryan-cox wants to merge 2 commits into
Open
Fix address-reviews duplicate detection and add real-data evals#558bryan-cox wants to merge 2 commits into
bryan-cox wants to merge 2 commits into
Conversation
openshift-ci
Bot
added
the
approved
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughExtends the address-reviews skill with a CI push-override rule that skips manual pushing when the prompt disables it, and expands bot detection to include GitHub Actions accounts. Introduces a comprehensive 14-case eval suite with Python judges validating categorization, reply format, and decision flags, plus an LLM quality rubric. Increments the plugin version to 0.0.12 and adds a 14.00 eval budget entry. Changesaddress-reviews skill behavior and evaluation suite
Plugin version bump and eval budget
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (2 errors)
✅ Passed checks (8 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/utils/commands/address-reviews.md (1)
259-269:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winPush flow still lacks required safety gates.
Line 259 adds CI opt-out, but the default path still allows unconditional force-push behavior and origin-assumed verification. This command must require explicit user approval before any push, reject main/master targets, reject force variants, and discover remotes before push/verification.
Suggested patch
-**CI Override**: If the user prompt explicitly says "Do NOT push" or "pushing will be handled automatically", skip Steps 4b and 4c entirely. The CI harness manages pushing separately. +**CI Override**: If the user prompt explicitly says "Do NOT push" or "pushing will be handled automatically", skip Steps 4b and 4c entirely. The CI harness manages pushing separately. + +**Mandatory Push Gate**: If CI override is not active, ask for explicit user approval before pushing: +`Replies are posted and commits are ready. Push now? (yes/no)` +If the user does not explicitly approve, skip Steps 4b and 4c. After all replies are posted, push all committed changes in a single push: ```bash -git push --force-with-lease +CURRENT_BRANCH=$(git branch --show-current) +case "$CURRENT_BRANCH" in + main|master|*/main|*/master) + echo "Refusing push to protected branch: $CURRENT_BRANCH" >&2 + exit 1 + ;; +esac +TARGET_REMOTE=$(git branch -vv | awk '/^\*/ {print $3}' | cut -d/ -f1) +if [ -z "$TARGET_REMOTE" ]; then + TARGET_REMOTE=$(git remote -v | awk 'NR==1 {print $1}') +fi +git push "$TARGET_REMOTE" "$CURRENT_BRANCH"4c. Verify push
-- Run
git log -1 --format='%H'locally andgit ls-remote origin <branch>to confirm the remote has your commit
+- Rungit log -1 --format='%H'locally andgit ls-remote "$TARGET_REMOTE" "$CURRENT_BRANCH"to confirm the remote has your commit</details> As per coding guidelines: commands/skills must ask before pushing, must never force-push, must never push main/master, and must discover remotes instead of assuming names. <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@plugins/utils/commands/address-reviews.mdaround lines 259 - 269, The push
flow in the address-reviews.md file (specifically in section 4b where git push
--force-with-lease is used) lacks required safety gates. Replace the force-push
command with a safer implementation that: first rejects pushes to protected
branches (main, master, and their variants) by checking the current branch name
and exiting if it matches these patterns, dynamically discovers the target
remote by querying the branch tracking info or falling back to the first remote
from git remote output instead of assuming "origin", performs a regular push to
the discovered remote and current branch without force flags, and updates the
verification step in section 4c to use the dynamically discovered TARGET_REMOTE
and CURRENT_BRANCH variables instead of hardcoded "origin" and branch name in
both the git log and git ls-remote commands.</details> <!-- cr-comment:v1:e59874e9f69bca2e5f5d9346 --> _Source: Coding guidelines_ </blockquote></details> </blockquote></details>🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. Inline comments: In `@plugins/openshift/commands/api-review.md`: - Line 10: Three fenced code blocks in the file lack language specification: the code block containing `/openshift:api-review [pr_url]` at line 10 (Step 1 prefix), the code block containing the example URL at line 212, and the code block at line 218. Add `bash` language specification to each of these three code blocks by changing the opening triple backticks from ``` to ```bash to comply with the markdownlint enforcement in the repository. - Around line 80-84: The code block that adds a git remote blindly assumes the "upstream" name is available without checking if it already exists. Before executing the git remote add upstream command, check whether a remote named "upstream" already exists using git remote. If it already exists, either use a different remote name, prompt the user to specify an alternative, or dynamically discover an available remote name. Only proceed with adding the remote if the name is confirmed to be available or use an alternative that does not conflict with existing remotes. --- Outside diff comments: In `@plugins/utils/commands/address-reviews.md`: - Around line 259-269: The push flow in the address-reviews.md file (specifically in section 4b where git push --force-with-lease is used) lacks required safety gates. Replace the force-push command with a safer implementation that: first rejects pushes to protected branches (main, master, and their variants) by checking the current branch name and exiting if it matches these patterns, dynamically discovers the target remote by querying the branch tracking info or falling back to the first remote from git remote output instead of assuming "origin", performs a regular push to the discovered remote and current branch without force flags, and updates the verification step in section 4c to use the dynamically discovered TARGET_REMOTE and CURRENT_BRANCH variables instead of hardcoded "origin" and branch name in both the git log and git ls-remote commands.🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID:
9a91e115-2a60-4552-a5de-af2613f50319📒 Files selected for processing (4)
plugins/openshift/commands/api-review.mdplugins/utils/commands/address-reviews.mdplugins/utils/evals/address-reviews.yamlplugins/utils/scripts/check_replied.py
bryan-cox
force-pushed
the
fix/address-reviews-dedup-and-evals
branch
3 times, most recently
from
c6ea632 to
18c0496
Compare
2 tasks
Contributor
Author
|
/test ? |
Contributor
Author
|
/test eval-address-reviews |
enxebre
reviewed
Jun 22, 2026
|
|
||
| Author: JoelSpeed | ||
| Comment: "We are moving away from crd-schema-checker towards the upstream crdify project, and in fact, in o/api we have disabled the majority of checks in crd-schema-checker because they are implemented in crdify already. Could you weigh in on whether to switch this PR to crdify?" | ||
| Type: issue_comment (general PR comment) |
Collaborator
There was a problem hiding this comment.
are you intentionally adding promptfoo coverage here vs eval harness?
bryan-cox
force-pushed
the
fix/address-reviews-dedup-and-evals
branch
from
18c0496 to
9738f1b
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/utils/evals/eval-address-reviews.yaml`:
- Around line 155-201: The review points out that schema fields
expected_should_filter and expected_priority_order are defined but lack
corresponding evaluation checks, unlike should_change_code_correct and
should_push_correct. Create two new evaluation checks following the same pattern
as the existing ones: add a check named should_filter_correct that compares the
predicted should_filter value from the analysis.json file against the
expected_should_filter annotation, and add a check named priority_order_correct
that compares the predicted priority_order value from the analysis.json file
against the expected_priority_order annotation. Both checks should reuse the
same JSON parsing and file retrieval logic as the existing checks and return
appropriate pass/fail messages with the compared values.
- Around line 99-103: The valid_output_json function's required fields list is
incomplete and does not match the full output contract defined in the file.
Update the required list to include all required fields specified in the output
contract definition (lines 19-26 and 68-75), which should include: category,
should_reply, should_change_code, rationale, should_push, should_filter,
priority_order, and reply_draft. This ensures the validation properly enforces
the complete output schema and prevents malformed analysis.json files from
passing validation.
- Around line 203-233: The reply_format_correct check is using incorrect
annotation key names that don't match the actual annotations used. Instead of
checking for expected_starts_with and expected_contains_footer, update the check
to look for expected_reply_contains and expected_reply_max_sentences keys.
Additionally, add validation logic to count the number of sentences in the
reply_draft and compare it against the expected_reply_max_sentences constraint,
returning an error if the sentence count exceeds the maximum allowed, and
include this validation result in the final error messages along with the other
format checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 37039e58-4cea-4adb-9e7b-d699c1d4917e
📒 Files selected for processing (35)
.claude-plugin/marketplace.jsondocs/index.htmlevals/budget.yamlplugins/utils/.claude-plugin/plugin.jsonplugins/utils/commands/address-reviews.mdplugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/annotations.yamlplugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/input.yamlplugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/annotations.yamlplugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/input.yamlplugins/utils/evals/cases/address-reviews/case-003-categorize-question/annotations.yamlplugins/utils/evals/cases/address-reviews/case-003-categorize-question/input.yamlplugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/annotations.yamlplugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/input.yamlplugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/annotations.yamlplugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/input.yamlplugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/annotations.yamlplugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/input.yamlplugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/annotations.yamlplugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/input.yamlplugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/annotations.yamlplugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/input.yamlplugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/annotations.yamlplugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/input.yamlplugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/annotations.yamlplugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/input.yamlplugins/utils/evals/cases/address-reviews/case-011-reply-format/annotations.yamlplugins/utils/evals/cases/address-reviews/case-011-reply-format/input.yamlplugins/utils/evals/cases/address-reviews/case-012-ci-push-override/annotations.yamlplugins/utils/evals/cases/address-reviews/case-012-ci-push-override/input.yamlplugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/annotations.yamlplugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/input.yamlplugins/utils/evals/cases/address-reviews/case-014-response-imperative-change/annotations.yamlplugins/utils/evals/cases/address-reviews/case-014-response-imperative-change/input.yamlplugins/utils/evals/eval-address-reviews.yamlplugins/utils/scripts/check_replied.py
✅ Files skipped from review due to trivial changes (25)
- plugins/utils/evals/cases/address-reviews/case-003-categorize-question/input.yaml
- plugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-012-ci-push-override/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-003-categorize-question/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/input.yaml
- plugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/input.yaml
- plugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/input.yaml
- plugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/input.yaml
- plugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/input.yaml
- plugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/input.yaml
- plugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/input.yaml
- plugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-011-reply-format/input.yaml
- plugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/input.yaml
- plugins/utils/.claude-plugin/plugin.json
- plugins/utils/commands/address-reviews.md
- .claude-plugin/marketplace.json
- docs/index.html
🚧 Files skipped from review as they are similar to previous changes (2)
- evals/budget.yaml
- plugins/utils/scripts/check_replied.py
bryan-cox
force-pushed
the
fix/address-reviews-dedup-and-evals
branch
from
9738f1b to
7214004
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/reference-analysis.json`:
- Line 2: The category field in reference-analysis.json is incorrectly set to
"CHANGE_REQUEST" when it should be "SUGGESTION" to align with the
case-007-categorize-suggestion test objective. Update the "category" value on
line 2 and line 9 from "CHANGE_REQUEST" to "SUGGESTION" so the test case
properly validates suggestion categorization instead of conflicting with a
different category type.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c1d37ae8-fd3e-4ac5-a31e-320c84edfe43
📒 Files selected for processing (49)
.claude-plugin/marketplace.jsondocs/index.htmlevals/budget.yamlplugins/utils/.claude-plugin/plugin.jsonplugins/utils/commands/address-reviews.mdplugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/annotations.yamlplugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/input.yamlplugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/annotations.yamlplugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/input.yamlplugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-003-categorize-question/annotations.yamlplugins/utils/evals/cases/address-reviews/case-003-categorize-question/input.yamlplugins/utils/evals/cases/address-reviews/case-003-categorize-question/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/annotations.yamlplugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/input.yamlplugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/annotations.yamlplugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/input.yamlplugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/annotations.yamlplugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/input.yamlplugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/annotations.yamlplugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/input.yamlplugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/annotations.yamlplugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/input.yamlplugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/annotations.yamlplugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/input.yamlplugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/annotations.yamlplugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/input.yamlplugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-011-reply-format/annotations.yamlplugins/utils/evals/cases/address-reviews/case-011-reply-format/input.yamlplugins/utils/evals/cases/address-reviews/case-011-reply-format/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-012-ci-push-override/annotations.yamlplugins/utils/evals/cases/address-reviews/case-012-ci-push-override/input.yamlplugins/utils/evals/cases/address-reviews/case-012-ci-push-override/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/annotations.yamlplugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/input.yamlplugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/reference-analysis.jsonplugins/utils/evals/cases/address-reviews/case-014-response-imperative-change/annotations.yamlplugins/utils/evals/cases/address-reviews/case-014-response-imperative-change/input.yamlplugins/utils/evals/cases/address-reviews/case-014-response-imperative-change/reference-analysis.jsonplugins/utils/evals/eval-address-reviews.yamlplugins/utils/scripts/check_replied.py
✅ Files skipped from review due to trivial changes (37)
- plugins/utils/evals/cases/address-reviews/case-012-ci-push-override/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/input.yaml
- plugins/utils/.claude-plugin/plugin.json
- plugins/utils/evals/cases/address-reviews/case-014-response-imperative-change/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-012-ci-push-override/input.yaml
- plugins/utils/evals/cases/address-reviews/case-003-categorize-question/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/input.yaml
- plugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-003-categorize-question/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-003-categorize-question/input.yaml
- plugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-011-reply-format/input.yaml
- plugins/utils/evals/cases/address-reviews/case-011-reply-format/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-001-duplicate-bot-reply/input.yaml
- plugins/utils/evals/cases/address-reviews/case-012-ci-push-override/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-014-response-imperative-change/input.yaml
- plugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/input.yaml
- plugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/reference-analysis.json
- plugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-013-response-question-no-change/input.yaml
- plugins/utils/commands/address-reviews.md
- plugins/utils/evals/cases/address-reviews/case-009-prioritize-mixed/input.yaml
- plugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/input.yaml
- plugins/utils/evals/cases/address-reviews/case-010-filter-coderabbit-kept/annotations.yaml
- docs/index.html
- plugins/utils/evals/cases/address-reviews/case-002-duplicate-extreme/input.yaml
- .claude-plugin/marketplace.json
- plugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-007-categorize-suggestion/annotations.yaml
🚧 Files skipped from review as they are similar to previous changes (8)
- plugins/utils/evals/cases/address-reviews/case-004-categorize-question-2/annotations.yaml
- evals/budget.yaml
- plugins/utils/evals/cases/address-reviews/case-005-categorize-change-request/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-006-categorize-dead-code/input.yaml
- plugins/utils/evals/cases/address-reviews/case-011-reply-format/annotations.yaml
- plugins/utils/evals/cases/address-reviews/case-008-categorize-action-instruction/input.yaml
- plugins/utils/scripts/check_replied.py
- plugins/utils/evals/eval-address-reviews.yaml
bryan-cox
force-pushed
the
fix/address-reviews-dedup-and-evals
branch
8 times, most recently
from
141add0 to
b626d33
Compare
Contributor
Author
|
/test eval-address-reviews |
Contributor
Author
|
/test eval-address-reviews |
openshift-ci
Bot
added
the
needs-rebase
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bryan-cox
force-pushed
the
fix/address-reviews-dedup-and-evals
branch
from
b626d33 to
c750e30
Compare
openshift-ci
Bot
removed
the
needs-rebase
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bryan-cox The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
bryan-cox
force-pushed
the
fix/address-reviews-dedup-and-evals
branch
from
c750e30 to
e90f2bb
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bryan-cox
force-pushed
the
fix/address-reviews-dedup-and-evals
branch
from
e90f2bb to
d26010c
Compare
Contributor
Author
|
/test eval-address-reviews |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-ci
Bot
added
the
needs-rebase
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
github-actionsandgithub-actions[bot]toBOT_SIGNATURESincheck_replied.pyso the skill's dedup check recognizes replies posted under the GHA identityaddress-reviews.mdso the skill respects "Do NOT push" directives from the periodic job harness instead of attempting to push (which conflicts with the harness's separate push phase)Context
The periodic review agent was producing 102+ duplicate replies because:
github-actionswasn't recognized as a bot incheck_replied.py, so the skill-level dedup check didn't catch existing replies posted under that identityThe eval suite validates these fixes using actual review thread data from the duplicate reply storm on PR #8535, covering duplicate prevention, comment categorization, prioritization, bot filtering, reply format, CI push override, and response rules. All 14 tests pass at 100%.
Companion PR: openshift/release for the periodic job's
BOT_ACCOUNTSfix.Test plan
make eval-plugins EVAL_PLUGIN=utils— 14/14 passing (100%)make lint— passes🤖 Generated with Claude Code
Summary by CodeRabbit