AI feedback loop: feedback inbox, ai_instructions fix suggestions, and AI evals - #9801
Draft
nishantmonu51 wants to merge 1 commit into
Draft
AI feedback loop: feedback inbox, ai_instructions fix suggestions, and AI evals#9801nishantmonu51 wants to merge 1 commit into
ai_instructions fix suggestions, and AI evals#9801nishantmonu51 wants to merge 1 commit into
Conversation
…and AI evals Closes the loop between users rating AI answers and admins improving the project's AI: - Persist chat feedback as `ai_feedback` rows (new SQLite migration 0043) with kind, sentiment, categories, comment, predicted attribution and review status. - Add `ListAIFeedback`, `GetAIFeedback` and `UpdateAIFeedbackStatus` to RuntimeService, gated on a new `ManageAIFeedback` permission granted to cloud project admins. - Add `ListProjectAIFeedback`, `GetProjectAIFeedback` and `ResolveProjectAIFeedback` to LocalService so Rill Developer can review feedback from the project's cloud deployment, reporting connectivity problems as states instead of errors. - Add `GenerateAIFeedbackFix`, which proposes an `ai_instructions` rule or measure addition for a feedback item, plus a draft eval case capturing the exchange. - Add a new `eval` resource type: parser, reconciler, runner, LLM judge and structural assertions, triggered on demand via `RefreshTrigger`/`CreateTrigger`. - Add `GenerateAIEvalFix`, which proposes `ai_instructions` changes that fix the latest run's failing cases without regressing the passing ones. - Frontend: feedback inbox with transcript and suggested fixes, eval workspace with runner and fix panels, and an "add to eval" flow from chat. Both behind the `feedback_inbox` and `ai_evals` feature flags (default off). Claude-Session: https://claude.ai/code/session_018c6Uo6WgWZZ2mPKKEK37XK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the loop between users rating AI answers and admins improving the project's AI.
Feedback capture and review
ai_feedbackrows in a new SQLite migration (0043.sql), including sentiment, categories, comment, AI-predicted attribution and review status.ListAIFeedback,GetAIFeedbackandUpdateAIFeedbackStatusruntime APIs, gated on a newManageAIFeedbackpermission that is granted to cloud project admins.ListProjectAIFeedback,GetProjectAIFeedbackandResolveProjectAIFeedbacklocal APIs so Rill Developer can review feedback recorded against the project's cloud deployment. Cloud connectivity problems are returned as states (NOT_LOGGED_IN,NOT_DEPLOYED,NO_PERMISSION,ERROR) rather than errors, so the UI can render precise empty states.Suggested fixes
GenerateAIFeedbackFixproposes a concrete project change for a feedback item: anai_instructionsrule on the project or a metrics view, or a new measure, along with a draft eval case capturing the exchange.AI evals
evalresource type (parser, reconciler, runner, LLM judge and structural assertions on the metrics view, measures and dimensions the agent queried). Evals never run on a schedule; they are triggered on demand throughRefreshTrigger/CreateTrigger, which also supports running a subset of cases and cancelling an in-flight run.GenerateAIEvalFixproposesai_instructionschanges that fix the latest run's failing cases without regressing the passing ones.Frontend
feedback_inboxandai_evalsfeature flags, both default off.Checklist: