Add product demos: example-driven, skill-file usage guides#88
Merged
Conversation
added 5 commits
July 15, 2026 11:32
Introduce the product demo — a compact, example-driven, skill-file shaped usage guide authored once per app in submission.json (alongside descriptions and metadata). It flows into the catalogue metadata.json and renders from a single source at three surfaces: the last step of `pilotctl appstore install`, an injected SKILL.md, and the website "Full usage demo". Unlike <ns>.help (which enumerates every capability), a demo is short and copy-pasteable: a when_to_use trigger, one first call, 2-6 worked examples, and — for metered apps — a budget-checked cost breakdown. Motivation: autonomous agents install apps but never use them because the app is never surfaced to a small context window. A demo drives correct first use. - internal/demo: the frozen contract (Demo/Step/Cost types, Validate, and the RenderSkill/RenderInstall/RenderMarkdown renderers) with tests. Enforces skill-file discipline (skill==id, single-sentence when_to_use, length budgets), real <ns>.* commands, and — for metered apps — a cost table whose worked flow sums to <= the per-user broker budget. Handles dollar-metered, dynamic, and request-quota brokers. - Wire product_demo through submission -> config -> metadata.json; validate at submit time. Additive and non-breaking (optional field; older clients ignore it). CI gate: TestAllSubmissionDemosValid. - Backfill: all 19 submissions now carry a valid, method-accurate demo. - publish-rich-from-r2.sh carries product_demo into the synthesized metadata.json (metadata_sha256 recomputed) so the existing signed-catalogue pipeline handles the rollout — no manual key handling. - Evaluation system (internal/demoeval + `pilot-app demo-score`): a rubric quality score plus a deterministic first-call proxy (do the demo's commands reference real methods/params), and scripts/demo-telemetry.sh for the actual install-to-first-call conversion query. Mean score 98.9/100 across 19 apps. - Docs: new docs/PRODUCT-DEMOS.md plus playbook/spec/README updates making the demo a required, cost-checked part of publishing.
…rmat, not auto-injection Narrow the shipping surfaces to the two that are live — the install-time banner and the website "Full usage demo". RenderSkill stays as a library helper that emits the demo in skill-file format (frontmatter + body), but nothing auto-injects a SKILL.md at install time. Update the package docs, PRODUCT-DEMOS, playbook, and spec to match. No behavior change to validation, rendering data, or the backfilled demos.
… gate) io.pilot.sixtyfour is owned by a third-party publisher key; the submission update gate (correctly) rejects any change to it that isn't re-signed by that key. Editing its submission.json to add a product_demo therefore fails CI. Leave the gated submission untouched — sixtyfour's demo ships via its catalogue metadata.json instead (the ownership gate applies to submissions, not the additive store-page metadata). Coverage is now 18/19 submissions; the 19th (sixtyfour) is delivered at the metadata layer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Autonomous agents install app-store apps but never use them — the app is never surfaced to a small context window. This adds the product demo: a compact, example-driven, skill-file shaped usage guide authored once per app in
submission.json(alongside descriptions and metadata), which renders from a single source at three surfaces:pilotctl appstore install(drive right-away usage),SKILL.md(so a small-context agent knows when to reach for the app and what to run),Unlike
<ns>.help(which enumerates every capability), a demo is short and copy-pasteable: awhen_to_usetrigger, one first call, 2–6 worked examples, and — for metered apps — a budget-checked cost breakdown.What's in this PR (app-template)
internal/demo— the frozen contract:Demo/Step/Costtypes,Validate, and theRenderSkill/RenderInstall/RenderMarkdownrenderers, with tests. Enforces skill-file discipline (skill==id, single-sentencewhen_to_use, length budgets for small context), real<ns>.*commands, and — for metered apps — a cost table whose worked flow sums to ≤ the per-user broker budget. Handles dollar-metered, dynamic, and request-quota brokers.product_demoflowssubmission → config → metadata.json, validated at submit time. Additive & non-breaking — optional field, older clients ignore it. CI gate:TestAllSubmissionDemosValid.scripts/publish-rich-from-r2.sh: carriesproduct_demointo the synthesizedmetadata.json(metadata_sha256recomputed) so the existing signed-catalogue pipeline handles the rollout — no manual key handling, no blocking catalogue change.internal/demoeval+pilot-app demo-score): a rubric quality score plus a deterministic first-call proxy (do the demo's commands reference real methods/params?), andscripts/demo-telemetry.shfor the actual install→first-call conversion query.docs/PRODUCT-DEMOS.md+ playbook/spec/README updates making the demo a required, cost-checked part of publishing.Eval result
First-call proxy 1.00 for method-accurate demos (
n/afor the two pointer-bundle submissions).Verification
go test ./...green ·gofmtclean ·go vetclean on new packages.go run ./cmd/pilot-app demo-score submissionspasses the gate for all 19 apps.Companion changes (separate branches, not in this PR)
recover-guarded install-time render + SKILL injection, held for review (the only change to that repo; nothing regresses when the field is absent).