Skip to content

feat(cli): send missing features and host-app friction as ratingless feedback - #5140

Merged
miguel-heygen merged 6 commits into
mainfrom
docs/skill-feedback-host-app
Oct 7, 2026
Merged

miguel-heygen merged 6 commits into
mainfrom
docs/skill-feedback-host-app

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

What

Agents now report two things the feedback rules never asked about: what a person needed that does not exist yet, and friction in the app that launched the CLI. Both are sent as a comment with no rating, so they never count in the rating metric.

Skill (hyperframes-cli):

  • Missing feature. Whenever the person asks for something HyperFrames or the host app cannot do, or can only do with a workaround, the agent sends one report per ask: npx hyperframes feedback --comment "MISSING FEATURE: <what the person asked for, in their words, with names, clients, figures and paths left out> | WORKAROUND: <what the agent did instead, or none>". The report is about the capability they wanted, never their content.
  • Host app. When HYPERFRAMES_CLIENT is set (the variable the CLI already reads for the launching app), the CLI is running inside a host app. When that app itself gets in the way (a panel, button, preview or export that misbehaves), the agent sends --comment "HOST APP: <what happened>" with the usual reproduction packet.
  • Both are sent only when telemetry is on and the user has not opted out. The missing-feature report describes the capability, never the person's content, and a catalog search that found nothing still goes to --search-miss.
  • references/preview-render.md and the CLI docs no longer call --rating required, and the feedback guide shows the ratingless form. skills-manifest.json is regenerated.

CLI (hyperframes feedback):

  • A --comment with no --rating is now a ratingless report. It goes to its own PostHog event, cli_feedback_comment (comment, doctor summary, feedback id, recent render ids), never to cli_render_feedback, so it cannot enter the rating metric. The --search-miss report already works this way.
  • The forward to the feedback channel omits rating and rating_scale for such a report.
  • --rating with or without a comment works exactly as before, and an unreadable rating still gets the same error. A report with neither a rating nor a non-blank comment is refused, and so is --file-issue without a rating (the issue template is built around a score).

Backend dependency

The feedback endpoint still requires an integer rating today. Until it accepts a missing one, the ratingless forward is rejected and dropped like any failed forward: the report still reaches PostHog, but not the feedback channel. A small backend change, in a separate pull request, makes the rating optional and posts such a report without a score. This one is safe to merge in either order.

Checks

  • New tests: the ratingless event's name and that it carries no rating; a comment with no rating goes to its own event and never to the rating event, and its forward carries no rating; a blank comment alone is refused; an unreadable rating keeps its error; neither a rating nor a comment is refused, and so is --file-issue without a rating; the request body leaves out rating and rating_scale; the skill teaches both ratingless commands. Each one fails when the line it covers is broken, and passes 3 runs in a row.
  • CLI typecheck, lint, format, comment checks, fallow, and the skills manifest check pass.

@miguel-heygen miguel-heygen changed the title docs(skills): ask agents inside a host app to report its friction as feedback feat(cli): send missing features and host-app friction as ratingless feedback Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Edit accuracy: accurate 2059 (base branch 2059), smooth 1423 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • crop-none-px-r0-root-z50: tracking 0.05, pressJump 0, drop 40.07, reload 40.07, render 39.98, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0.13, reload 0.12, render 0.03, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0.13, reload 0.12, render 0.03, renderKey -, undo true, teleport true

@mintlify

mintlify Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
hyperframes 🟢 Ready View Preview Oct 7, 2026, 6:14 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

@miguel-heygen
miguel-heygen marked this pull request as ready for review October 7, 2026 02:01

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve @ effdf97b. A comment with no rating now goes to its own PostHog event and never touches cli_render_feedback, so the rating metric stays clean. The rated path behaves exactly as before.

What I checked

  • The split. reportRating runs before the source and telemetry checks and refuses three cases: an unreadable rating (same error as before), neither a rating nor a non-blank comment, and --file-issue without a rating. trackFeedback then sends a ratingless report to cli_feedback_comment and a rated one to cli_render_feedback. The lint nudge still runs for rated reports only.

  • Telemetry gate. Ratingless reports go through the same shouldTrack() return as rated ones, so an opted-out user sends nothing, as the skill promises.

  • Host app. getDoctorSummary already appends client=<HYPERFRAMES_CLIENT> (telemetry/feedback.ts:105), so a HOST APP: report names its app both in PostHog (doctor_summary) and in the forwarded env string.

  • Merge order. On EF master, HyperframesFeedbackRequest.rating is a required int. Until EF #55462 lands, a ratingless forward gets a 422, and submitFeedback ignores the response, so the report is lost silently and PostHog still has it. Older CLIs always send a rating, so EF can land first too. The "either order" claim holds. I read #55462: it makes rating optional, skips the bounds check for None, and posts with no score line.

  • Tests: the 4 changed files pass (113 tests). Nine of ten mutations fail at least one test:

    • a ratingless report sends no PostHog event (1)
    • it goes to the rating event (1)
    • the neither-nor check is removed (2)
    • a blank comment is let through without trimming (1)
    • the --file-issue check is removed (1)
    • rating_scale is always sent (1)
    • rating becomes required again (1)
    • the comment event carries a rating (1)
    • the skill's host-app bullet is deleted (1)

    The survivor is deleting the printFeedbackLintWarnings call. That call wasn't pinned on main either, so this PR didn't lose any coverage.

Nits (none blocking)

  • The HOST APP: packet gets no nudge. The skill asks for a reproduction packet with a HOST APP: report, but lintFeedbackComment takes a rating and only runs on rated reports. So an agent that sends a bare symptom gets no "missing REPRO COMMAND" warning, which is exactly the report that needs one. Possible fix: run the repro-marker check on a ratingless comment that starts with HOST APP:.
  • Reuse: trackFeedbackComment repeats the three optional spreads from trackRenderFeedback (doctor_summary, feedback_id, recent_render_ids). One small feedbackJoinProps(props) helper used by both keeps the two events' join keys identical when one of them changes.

CI: red only on Studio: timeline viewport gate. Interaction p95 was 66.9 ms, then 59.1 ms, against a 58.3 ms budget, and all 5/5 runs passed on both attempts. This PR touches no studio code, so a rerun should clear it. The rest were green or still running when I checked.

— Rames

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 7, 2026

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve @ 8fb6b132. This head only merges main (2d98d5c1). I diffed the PR's own patch before and after: the same 12 files, and the only changes are a hunk offset in one doc and the regenerated skills-manifest.json hash. The required checks are green at this head. My review at effdf97b stands as written, nits included.

— Rames

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 7, 2026
Merged via the queue into main with commit 80c2547 Oct 7, 2026
92 checks passed
@miguel-heygen
miguel-heygen deleted the docs/skill-feedback-host-app branch October 7, 2026 06:52

This branch was successfully deployed

1 active deployment
staging - docs — 8fb6b132 Deployed Oct 7, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants