Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/guides/feedback.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ npx hyperframes feedback --rating 7 --comment "The render finished, but the capt

The rating is from 0 to 10. The comment is optional, but a specific example is usually more useful than a score alone.

A comment with no rating is a report with no score, for something you needed that HyperFrames cannot do yet. It never counts in the rating:

```bash
npx hyperframes feedback --comment "MISSING FEATURE: trim a clip from the timeline | WORKAROUND: none"
```

## Report a reproducible bug

Use `--file-issue` when the project can be shared publicly:
Expand Down
5 changes: 3 additions & 2 deletions docs/packages/cli.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1248,10 +1248,11 @@ the CLI attaches your email (below).
npx hyperframes feedback --rating 10
npx hyperframes feedback --rating 7 --comment "render succeeded but GSAP timeline didn't animate"
npx hyperframes feedback --rating 3 --comment "GSAP timeline froze on seek" --file-issue
npx hyperframes feedback --comment "MISSING FEATURE: trim a clip from the timeline | WORKAROUND: none"
```

`--rating` is required, 0–10. `--comment` adds free text. `--source person`
or `--source agent` says who wrote a rating report (a `--search-miss` report
`--rating` is 0–10. `--comment` adds free text; a `--comment` with no `--rating` is a report with no score (a missing feature, or friction in the app that launched the CLI), kept out of the rating metric. `--source person`
or `--source agent` says who wrote a rating or comment report (a `--search-miss` report
carries neither), so the team can tell a person's feedback from an agent's. `--file-issue`
also opens a GitHub issue, `--dir` picks the project published as its repro
(default: current directory), and `--yes, -y` skips the consent prompt for
Expand Down
9 changes: 9 additions & 0 deletions packages/cli/src/commands/coreSkillContent.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,15 @@ describe("hyperframes-core contract docs", () => {
expect(renderReference).toContain("WORKAROUND:");
});

it("sends missing features and host-app friction as ratingless comments", () => {
const skill = read("skills", "hyperframes-cli", "SKILL.md");

expect(skill).toContain(
'npx hyperframes feedback --comment "MISSING FEATURE: <what the person asked for, in their words, with names, clients, figures and paths left out> | WORKAROUND:',
);
expect(skill).toMatch(/HYPERFRAMES_CLIENT[^\n]*npx hyperframes feedback --comment "HOST APP:/);
});

it("mandates a composition-structure block for visual-defect feedback", () => {
const skill = read("skills", "hyperframes-cli", "SKILL.md");
const renderReference = read("skills", "hyperframes-cli", "references", "preview-render.md");
Expand Down
38 changes: 38 additions & 0 deletions packages/cli/src/commands/feedback.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ const mocks = vi.hoisted(() => ({
}));
vi.mock("../telemetry/client.js", () => ({ shouldTrack: () => true, flush: mocks.flush }));
const trackRenderFeedback = vi.hoisted(() => vi.fn());
const trackFeedbackComment = vi.hoisted(() => vi.fn());
vi.mock("../telemetry/events.js", () => ({
trackCatalogSearchMiss: vi.fn(),
trackFeedbackComment,
trackRenderFeedback,
}));
vi.mock("../telemetry/feedback.js", () => ({ getDoctorSummary: async () => "os=test" }));
Expand Down Expand Up @@ -58,3 +60,39 @@ it("refuses a source that is neither person nor agent, and sends nothing", async
await expect(feedback.run?.({ args: { rating: "6", source: "bot" } } as never)).rejects.toThrow();
expect(mocks.submitFeedback).not.toHaveBeenCalled();
});

it("sends a comment with no rating as its own report, never as a rating", async () => {
vi.spyOn(console, "log").mockImplementation(() => {});
trackRenderFeedback.mockClear();
const comment = "MISSING FEATURE: trim a clip | WORKAROUND: none";
const { default: feedback } = await import("./feedback.js");
await feedback.run?.({ args: { comment, source: "agent" } } as never);
expect(trackRenderFeedback).not.toHaveBeenCalled();
expect(trackFeedbackComment).toHaveBeenLastCalledWith(expect.objectContaining({ comment }));
expect(mocks.submitFeedback).toHaveBeenLastCalledWith(
expect.objectContaining({ rating: undefined, comment }),
);
});

it.each([
["neither a rating nor a comment", {}],
["a blank comment with no rating", { comment: " " }],
["--file-issue with no rating", { comment: "MISSING FEATURE: x", "file-issue": true }],
])("refuses %s, and sends nothing", async (_, args) => {
vi.spyOn(console, "error").mockImplementation(() => {});
mocks.submitFeedback.mockClear();
const { default: feedback } = await import("./feedback.js");
await expect(feedback.run?.({ args } as never)).rejects.toThrow();
expect(mocks.submitFeedback).not.toHaveBeenCalled();
});

it("keeps the rating error for an unreadable rating, so a retry keeps the rating", async () => {
const errors = vi.spyOn(console, "error").mockImplementation(() => {});
const { default: feedback } = await import("./feedback.js");
await expect(
feedback.run?.({ args: { rating: "8/10", comment: "x" } } as never),
).rejects.toThrow();
expect(String(errors.mock.calls.at(-1)?.[0])).toContain(
"Rating must be an integer between 0 and 10",
);
});
85 changes: 62 additions & 23 deletions packages/cli/src/commands/feedback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@ import { defineCommand } from "citty";
import * as clack from "@clack/prompts";
import open from "open";
import type { Example } from "./_examples.js";
import { trackCatalogSearchMiss, trackRenderFeedback } from "../telemetry/events.js";
import {
trackCatalogSearchMiss,
trackFeedbackComment,
trackRenderFeedback,
} from "../telemetry/events.js";
import { shouldTrack } from "../telemetry/client.js";
import { getDoctorSummary } from "../telemetry/feedback.js";
import { readConfig, type RecentRenderRecord } from "../telemetry/config.js";
Expand All @@ -21,6 +25,10 @@ import { lintFeedbackComment, type FeedbackLintInput } from "../utils/feedbackLi
export const examples: Example[] = [
["Submit render feedback", 'hyperframes feedback --rating 8 --comment "fast but font missing"'],
["Quick rating only", "hyperframes feedback --rating 10"],
[
"Report something you needed that is missing (no rating)",
'hyperframes feedback --comment "MISSING FEATURE: trim a clip from the timeline | WORKAROUND: none"',
],
[
"Report a catalog gap after a fruitless search",
'hyperframes feedback --search-miss "typewriter that deletes" --wanted "text that types then backspaces"',
Expand All @@ -38,8 +46,8 @@ function normalizeComment(raw?: string): string | undefined {
/**
* Compact PostHog join keys appended to the environment string that rides
* along with the forwarded report (and therefore lands verbatim in the wild
* feedback channel): `fid` = this submission's PostHog `cli_render_feedback`
* `feedback_id`; `tid` = the install's telemetry distinct_id; `renders` =
* feedback channel): `fid` = the PostHog `feedback_id` (`cli_render_feedback` or
* `cli_feedback_comment`); `tid` = the install's telemetry distinct_id; `renders` =
* recent `render_job_id`s (newest last, `!` suffix = the render failed).
* Together they turn a wild report into an exact telemetry lookup instead of
* a hardware-fingerprint hunt.
Expand Down Expand Up @@ -155,18 +163,62 @@ async function fileGithubIssue(opts: {
await openAndPrintIssue(url);
}

// A comment with no rating is its own report (a missing feature, host-app
// friction): it scores nothing, so it must stay out of the rating metric.
function reportRating(
raw: string | undefined,
comment: string | undefined,
fileIssue: boolean,
): number | undefined {
const rating = raw === undefined ? undefined : parseFeedbackRating(raw);
if (rating === null) {
console.error(c.error("Rating must be an integer between 0 and 10"));
failCommand();
}
if (rating === undefined && !comment?.trim()) {
console.error(c.error("Give a --rating (an integer from 0 to 10), a --comment, or both"));
failCommand();
}
if (rating === undefined && fileIssue) {
console.error(c.error("--file-issue needs a --rating"));
failCommand();
}
return rating;
}

function trackFeedback(report: {
rating: number | undefined;
comment: string | undefined;
doctorSummary: string;
feedbackId: string;
recentRenderIds: string[] | undefined;
}): void {
const { rating, comment, ...keys } = report;
if (rating === undefined) {
if (comment) trackFeedbackComment({ comment, ...keys });
return;
}
// Soft-warn (never blocks) when the comment for a non-clean report is
// missing the mandated reproduction-packet markers. Prints before the
// submission ack so the reporter sees the nudge while their run is fresh.
printFeedbackLintWarnings({ rating, comment });
// The standalone command runs separately from `render`, so it has no real
// elapsed time to report. Omit it rather than recording a fake duration.
trackRenderFeedback({ rating, comment, ...keys });
}

export default defineCommand({
meta: { name: "feedback", description: "Submit feedback about your experience" },
args: {
rating: {
type: "string",
// Required for a rating report, but --search-miss is a different report
// with no rating to give, so the check moved into the run body.
// Optional: --search-miss and a lone --comment are reports with no
// rating to give, so the check lives in the run body.
description: "Likelihood to recommend (0=not likely, 10=extremely likely)",
},
comment: {
type: "string",
description: "Optional details about your experience",
description: "Details about your experience; alone, a report with no rating",
},
source: {
type: "string",
Expand Down Expand Up @@ -220,13 +272,8 @@ export default defineCommand({
return;
}

// `--rating` is no longer required at the arg level, so an absent one
// reaches here as undefined rather than being rejected by the parser.
const rating = args.rating === undefined ? null : parseFeedbackRating(args.rating);
if (rating === null) {
console.error(c.error("Rating must be an integer between 0 and 10"));
failCommand();
}
const comment = normalizeComment(args.comment);
const rating = reportRating(args.rating, comment, args["file-issue"] === true);

const source = parseFeedbackSource(args.source);
if (source === null) {
Expand All @@ -239,7 +286,6 @@ export default defineCommand({
return;
}

const comment = normalizeComment(args.comment);
const doctorSummary = await getDoctorSummary();

// Join keys tying this report to the install's PostHog rows — see
Expand All @@ -254,14 +300,7 @@ export default defineCommand({
});
const envWithJoinKeys = doctorSummary ? `${doctorSummary} ${joinKeys}` : joinKeys;

// Soft-warn (never blocks) when the comment for a non-clean report is
// missing the mandated reproduction-packet markers. Prints before the
// submission ack so the reporter sees the nudge while their run is fresh.
printFeedbackLintWarnings({ rating, comment });

// The standalone command runs separately from `render`, so it has no real
// elapsed time to report. Omit it rather than recording a fake duration.
trackRenderFeedback({
trackFeedback({
rating,
comment,
doctorSummary,
Expand All @@ -281,7 +320,7 @@ export default defineCommand({
email: feedbackEmail(),
});

if (args["file-issue"] === true) {
if (args["file-issue"] === true && rating !== undefined) {
await fileGithubIssue({
rating,
comment,
Expand Down
12 changes: 12 additions & 0 deletions packages/cli/src/telemetry/events.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ const {
trackCliError,
trackFigmaImport,
trackRenderFeedback,
trackFeedbackComment,
trackRenderPreflightRejected,
trackAuthLoginStarted,
trackAuthLoginCompleted,
Expand Down Expand Up @@ -1069,6 +1070,17 @@ describe("render telemetry events", () => {
});
});

describe("trackFeedbackComment", () => {
it("is its own event with no rating, so it never counts in the rating metric", () => {
trackEvent.mockClear();
trackFeedbackComment({ comment: "MISSING FEATURE: trim", feedbackId: "f1" });

const [name, props] = trackEvent.mock.calls[0] as [string, Record<string, unknown>];
expect(name).toBe("cli_feedback_comment");
expect(props).toEqual({ comment: "MISSING FEATURE: trim", feedback_id: "f1" });
});
});

describe("trackRenderFeedback", () => {
beforeEach(() => {
trackEvent.mockClear();
Expand Down
19 changes: 18 additions & 1 deletion packages/cli/src/telemetry/events.ts
Original file line number Diff line number Diff line change
Expand Up @@ -992,6 +992,22 @@ export function trackSkillsInstallSkipped(props: { reason: string }): void {
trackEvent("cli skill install skipped", { reason: props.reason });
}

export function trackFeedbackComment(props: {
comment: string;
doctorSummary?: string;
feedbackId?: string;
recentRenderIds?: string[];
}): void {
trackEvent("cli_feedback_comment", {
comment: props.comment,
...(props.doctorSummary ? { doctor_summary: props.doctorSummary } : {}),
...(props.feedbackId ? { feedback_id: props.feedbackId } : {}),
...(props.recentRenderIds?.length
? { recent_render_ids: props.recentRenderIds.join(",") }
: {}),
});
}

export function trackRenderFeedback(props: {
rating: number;
renderDurationMs?: number;
Expand All @@ -1000,7 +1016,8 @@ export function trackRenderFeedback(props: {
/**
* Join key shared with the forwarded feedback report (Slack/backend): the
* same uuid rides in the report's env string as `fid=…`, so a wild report
* resolves to exactly one PostHog `cli_render_feedback` event and vice versa.
* resolves to exactly one PostHog `cli_render_feedback` or
* `cli_feedback_comment` event and vice versa.
*/
feedbackId?: string;
/** render_job_id values of this install's recent renders (newest last). */
Expand Down
12 changes: 12 additions & 0 deletions packages/cli/src/utils/submitFeedback.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ describe("submitFeedback", () => {
vi.unstubAllGlobals();
});

it("leaves the rating and its scale out of a report that has none", async () => {
const fetchMock = vi.fn<typeof fetch>(async () => new Response(null, { status: 202 }));
vi.stubGlobal("fetch", fetchMock);

await submitFeedback({ comment: "MISSING FEATURE: trim", cliVersion: "1.2.3" });

const body = JSON.parse(String(fetchMock.mock.calls[0]?.[1]?.body));
expect(body).not.toHaveProperty("rating");
expect(body).not.toHaveProperty("rating_scale");
expect(body.comment).toBe("MISSING FEATURE: trim");
});

it("posts feedback to the backend endpoint", async () => {
const fetchMock = vi.fn<typeof fetch>(async () => new Response(null, { status: 202 }));
vi.stubGlobal("fetch", fetchMock);
Expand Down
4 changes: 2 additions & 2 deletions packages/cli/src/utils/submitFeedback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ function cap(value: string | undefined, max: number): string | undefined {
}

export async function submitFeedback(input: {
rating: number;
rating?: number;
comment?: string;
cliVersion: string;
env?: string;
Expand All @@ -28,7 +28,7 @@ export async function submitFeedback(input: {
method: "POST",
body: JSON.stringify({
rating: input.rating,
rating_scale: FEEDBACK_RATING_SCALE,
rating_scale: input.rating === undefined ? undefined : FEEDBACK_RATING_SCALE,
comment: cap(input.comment, MAX_COMMENT),
cli_version: cap(input.cliVersion, MAX_CLI_VERSION),
env: cap(input.env, MAX_ENV),
Expand Down
2 changes: 1 addition & 1 deletion skills-manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"files": 7
},
"hyperframes-cli": {
"hash": "cf031cc9764e451f",
"hash": "599eba83e1bf7245",
"files": 11
},
"hyperframes-core": {
Expand Down
5 changes: 5 additions & 0 deletions skills/hyperframes-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,11 @@ npx hyperframes feedback --rating <0-10> --comment "<specific result or friction

Keep clean-run feedback concise. For any bug or friction, capture a **reproduction packet** before submitting; do not send only a symptom summary. Include the rerunnable command (relative to the project directory — feedback is submitted to a public channel, so do **not** paste absolute paths, home-directory prefixes, or user/machine identifiers), expected versus actual behavior, exact error (also strip absolute paths from stack traces — keep basename + line, drop the leading directory), whether output completed/fell back/failed, workaround, and repro-project status. For a rating ≤ 7 that describes a visual defect (black frame, flicker, corrupt output, wrong frame, blank output, other visual anomaly), also include a `COMPOSITION_STRUCTURE:` block — a privacy-preserving structural anatomy (element census + attribute presence + timeline shape) so maintainers can pattern-match against known bug families without the composition ZIP. Agents auto-fill this via the composition-census helper; the human user does not fill it by hand. If the issue did not reproduce again, say so and still include the last failing command and logs. Use `--file-issue` only with consent: it publishes a minimal reproduction to a public URL. The required packet format and privacy warning live in `references/preview-render.md`.

Two more reports are a `--comment` with no `--rating`, so they never count in the rating metric. Send them unless telemetry is disabled or the user opted out, under the same privacy rules:

- **Missing feature.** Whenever the person asks for something HyperFrames or the host app cannot do, or can only do with a workaround, send one report per ask: `npx hyperframes feedback --comment "MISSING FEATURE: <what the person asked for, in their words, with names, clients, figures and paths left out> | WORKAROUND: <what you did instead, or none>"`. The report is about the capability they wanted, never their content or anything private. A catalog search that found nothing goes to `--search-miss` instead.
- **Host app.** When `HYPERFRAMES_CLIENT` is set, the CLI is running inside a host app. When that app itself gets in the way (a panel, button, preview or export that misbehaves), send `npx hyperframes feedback --comment "HOST APP: <what happened>"` with the reproduction packet.

## Read the matching reference before running a command

The following references and owning skills are mandatory command contracts, not optional background reading. Before running a command in the table, read its matching row.
Expand Down
2 changes: 1 addition & 1 deletion skills/hyperframes-cli/references/preview-render.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ npx hyperframes feedback --rating 10 # clean run, n
npx hyperframes feedback --rating 6 --comment "bg <video> renders grey in multi-scene; worked around with --format png-sequence"
```

`--rating` is an integer from 0-10 (required); `--comment` is free text. Feedback is anonymous and attaches a `doctorSummary` (OS/Node/CPU/mem/ffmpeg) automatically, so don't repeat those fields. A clean run needs only a short result. Before sending any bug, workaround, or confusing behavior, collect this compact reproduction packet:
`--rating` is an integer from 0-10; `--comment` is free text. A `--comment` with no `--rating` is a ratingless report (`MISSING FEATURE:` and `HOST APP:` in `SKILL.md`) that never counts in the rating metric. Feedback is anonymous and attaches a `doctorSummary` (OS/Node/CPU/mem/ffmpeg) automatically, so don't repeat those fields. A clean run needs only a short result. Before sending any bug, workaround, or confusing behavior, collect this compact reproduction packet:

```text
REPRO COMMAND: <HF_*/PRODUCER_* env> npx hyperframes <exact command> # run from the project directory; do NOT paste absolute paths
Expand Down
Loading