Skip to content

AI feedback loop: feedback inbox, ai_instructions fix suggestions, and AI evals - #9801

Draft
nishantmonu51 wants to merge 1 commit into
mainfrom
nishant/ai-feedback-loop
Draft

AI feedback loop: feedback inbox, ai_instructions fix suggestions, and AI evals#9801
nishantmonu51 wants to merge 1 commit into
mainfrom
nishant/ai-feedback-loop

Conversation

@nishantmonu51

Copy link
Copy Markdown
Collaborator

Closes the loop between users rating AI answers and admins improving the project's AI.

Feedback capture and review

  • Chat feedback (ratings and explicit review requests) is persisted as ai_feedback rows in a new SQLite migration (0043.sql), including sentiment, categories, comment, AI-predicted attribution and review status.
  • New ListAIFeedback, GetAIFeedback and UpdateAIFeedbackStatus runtime APIs, gated on a new ManageAIFeedback permission that is granted to cloud project admins.
  • New ListProjectAIFeedback, GetProjectAIFeedback and ResolveProjectAIFeedback local APIs so Rill Developer can review feedback recorded against the project's cloud deployment. Cloud connectivity problems are returned as states (NOT_LOGGED_IN, NOT_DEPLOYED, NO_PERMISSION, ERROR) rather than errors, so the UI can render precise empty states.

Suggested fixes

  • GenerateAIFeedbackFix proposes a concrete project change for a feedback item: an ai_instructions rule on the project or a metrics view, or a new measure, along with a draft eval case capturing the exchange.

AI evals

  • New eval resource type (parser, reconciler, runner, LLM judge and structural assertions on the metrics view, measures and dimensions the agent queried). Evals never run on a schedule; they are triggered on demand through RefreshTrigger/CreateTrigger, which also supports running a subset of cases and cancelling an in-flight run.
  • GenerateAIEvalFix proposes ai_instructions changes that fix the latest run's failing cases without regressing the passing ones.

Frontend

  • Feedback inbox with the full conversation transcript and appliable fix suggestions, an eval workspace with a runner and fix panel, and an "add to eval" flow from chat.
  • Everything is behind the feedback_inbox and ai_evals feature flags, both default off.

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

…and AI evals

Closes the loop between users rating AI answers and admins improving the project's AI:

- Persist chat feedback as `ai_feedback` rows (new SQLite migration 0043) with kind,
  sentiment, categories, comment, predicted attribution and review status.
- Add `ListAIFeedback`, `GetAIFeedback` and `UpdateAIFeedbackStatus` to RuntimeService,
  gated on a new `ManageAIFeedback` permission granted to cloud project admins.
- Add `ListProjectAIFeedback`, `GetProjectAIFeedback` and `ResolveProjectAIFeedback` to
  LocalService so Rill Developer can review feedback from the project's cloud deployment,
  reporting connectivity problems as states instead of errors.
- Add `GenerateAIFeedbackFix`, which proposes an `ai_instructions` rule or measure
  addition for a feedback item, plus a draft eval case capturing the exchange.
- Add a new `eval` resource type: parser, reconciler, runner, LLM judge and structural
  assertions, triggered on demand via `RefreshTrigger`/`CreateTrigger`.
- Add `GenerateAIEvalFix`, which proposes `ai_instructions` changes that fix the latest
  run's failing cases without regressing the passing ones.
- Frontend: feedback inbox with transcript and suggested fixes, eval workspace with
  runner and fix panels, and an "add to eval" flow from chat. Both behind the
  `feedback_inbox` and `ai_evals` feature flags (default off).

Claude-Session: https://claude.ai/code/session_018c6Uo6WgWZZ2mPKKEK37XK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant