Repository navigation
feat(cli): send missing features and host-app friction as ratingless feedback - #5140
Conversation
Edit accuracy: accurate 2059 (base branch 2059), smooth 1423 of thoseThe gate passes. Quarantined, measured but not gated (0) Unstable (1)
|
…tingless feedback
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
jrusso1020
left a comment
There was a problem hiding this comment.
Approve @ effdf97b. A comment with no rating now goes to its own PostHog event and never touches cli_render_feedback, so the rating metric stays clean. The rated path behaves exactly as before.
What I checked
-
The split.
reportRatingruns before the source and telemetry checks and refuses three cases: an unreadable rating (same error as before), neither a rating nor a non-blank comment, and--file-issuewithout a rating.trackFeedbackthen sends a ratingless report tocli_feedback_commentand a rated one tocli_render_feedback. The lint nudge still runs for rated reports only. -
Telemetry gate. Ratingless reports go through the same
shouldTrack()return as rated ones, so an opted-out user sends nothing, as the skill promises. -
Host app.
getDoctorSummaryalready appendsclient=<HYPERFRAMES_CLIENT>(telemetry/feedback.ts:105), so aHOST APP:report names its app both in PostHog (doctor_summary) and in the forwarded env string. -
Merge order. On EF master,
HyperframesFeedbackRequest.ratingis a requiredint. Until EF #55462 lands, a ratingless forward gets a 422, andsubmitFeedbackignores the response, so the report is lost silently and PostHog still has it. Older CLIs always send a rating, so EF can land first too. The "either order" claim holds. I read #55462: it makesratingoptional, skips the bounds check forNone, and posts with no score line. -
Tests: the 4 changed files pass (113 tests). Nine of ten mutations fail at least one test:
- a ratingless report sends no PostHog event (1)
- it goes to the rating event (1)
- the neither-nor check is removed (2)
- a blank comment is let through without trimming (1)
- the
--file-issuecheck is removed (1) rating_scaleis always sent (1)- rating becomes required again (1)
- the comment event carries a rating (1)
- the skill's host-app bullet is deleted (1)
The survivor is deleting the
printFeedbackLintWarningscall. That call wasn't pinned on main either, so this PR didn't lose any coverage.
Nits (none blocking)
- The
HOST APP:packet gets no nudge. The skill asks for a reproduction packet with aHOST APP:report, butlintFeedbackCommenttakes a rating and only runs on rated reports. So an agent that sends a bare symptom gets no "missing REPRO COMMAND" warning, which is exactly the report that needs one. Possible fix: run the repro-marker check on a ratingless comment that starts withHOST APP:. - Reuse:
trackFeedbackCommentrepeats the three optional spreads fromtrackRenderFeedback(doctor_summary,feedback_id,recent_render_ids). One smallfeedbackJoinProps(props)helper used by both keeps the two events' join keys identical when one of them changes.
CI: red only on Studio: timeline viewport gate. Interaction p95 was 66.9 ms, then 59.1 ms, against a 58.3 ms budget, and all 5/5 runs passed on both attempts. This PR touches no studio code, so a rerun should clear it. The rest were green or still running when I checked.
— Rames
jrusso1020
left a comment
There was a problem hiding this comment.
Approve @ 8fb6b132. This head only merges main (2d98d5c1). I diffed the PR's own patch before and after: the same 12 files, and the only changes are a hunk offset in one doc and the regenerated skills-manifest.json hash. The required checks are green at this head. My review at effdf97b stands as written, nits included.
— Rames
What
Agents now report two things the feedback rules never asked about: what a person needed that does not exist yet, and friction in the app that launched the CLI. Both are sent as a comment with no rating, so they never count in the rating metric.
Skill (
hyperframes-cli):npx hyperframes feedback --comment "MISSING FEATURE: <what the person asked for, in their words, with names, clients, figures and paths left out> | WORKAROUND: <what the agent did instead, or none>". The report is about the capability they wanted, never their content.HYPERFRAMES_CLIENTis set (the variable the CLI already reads for the launching app), the CLI is running inside a host app. When that app itself gets in the way (a panel, button, preview or export that misbehaves), the agent sends--comment "HOST APP: <what happened>"with the usual reproduction packet.--search-miss.references/preview-render.mdand the CLI docs no longer call--ratingrequired, and the feedback guide shows the ratingless form.skills-manifest.jsonis regenerated.CLI (
hyperframes feedback):--commentwith no--ratingis now a ratingless report. It goes to its own PostHog event,cli_feedback_comment(comment, doctor summary, feedback id, recent render ids), never tocli_render_feedback, so it cannot enter the rating metric. The--search-missreport already works this way.ratingandrating_scalefor such a report.--ratingwith or without a comment works exactly as before, and an unreadable rating still gets the same error. A report with neither a rating nor a non-blank comment is refused, and so is--file-issuewithout a rating (the issue template is built around a score).Backend dependency
The feedback endpoint still requires an integer rating today. Until it accepts a missing one, the ratingless forward is rejected and dropped like any failed forward: the report still reaches PostHog, but not the feedback channel. A small backend change, in a separate pull request, makes the rating optional and posts such a report without a score. This one is safe to merge in either order.
Checks
--file-issuewithout a rating; the request body leaves outratingandrating_scale; the skill teaches both ratingless commands. Each one fails when the line it covers is broken, and passes 3 runs in a row.