Skip to content

Add product demos: example-driven, skill-file usage guides#88

Merged
Alexgodoroja merged 5 commits into
mainfrom
feat/product-demos
Jul 15, 2026
Merged

Add product demos: example-driven, skill-file usage guides#88
Alexgodoroja merged 5 commits into
mainfrom
feat/product-demos

Conversation

@Alexgodoroja

Copy link
Copy Markdown
Collaborator

What & why

Autonomous agents install app-store apps but never use them — the app is never surfaced to a small context window. This adds the product demo: a compact, example-driven, skill-file shaped usage guide authored once per app in submission.json (alongside descriptions and metadata), which renders from a single source at three surfaces:

  • the last step of pilotctl appstore install (drive right-away usage),
  • an injected SKILL.md (so a small-context agent knows when to reach for the app and what to run),
  • the website "Full usage demo" section.

Unlike <ns>.help (which enumerates every capability), a demo is short and copy-pasteable: a when_to_use trigger, one first call, 2–6 worked examples, and — for metered apps — a budget-checked cost breakdown.

What's in this PR (app-template)

  • internal/demo — the frozen contract: Demo/Step/Cost types, Validate, and the RenderSkill/RenderInstall/RenderMarkdown renderers, with tests. Enforces skill-file discipline (skill==id, single-sentence when_to_use, length budgets for small context), real <ns>.* commands, and — for metered apps — a cost table whose worked flow sums to ≤ the per-user broker budget. Handles dollar-metered, dynamic, and request-quota brokers.
  • Wiring: product_demo flows submission → config → metadata.json, validated at submit time. Additive & non-breaking — optional field, older clients ignore it. CI gate: TestAllSubmissionDemosValid.
  • Backfill: all 19 submissions now carry a valid, method-accurate demo.
  • scripts/publish-rich-from-r2.sh: carries product_demo into the synthesized metadata.json (metadata_sha256 recomputed) so the existing signed-catalogue pipeline handles the rollout — no manual key handling, no blocking catalogue change.
  • Evaluation system (internal/demoeval + pilot-app demo-score): a rubric quality score plus a deterministic first-call proxy (do the demo's commands reference real methods/params?), and scripts/demo-telemetry.sh for the actual install→first-call conversion query.
  • Docs: new docs/PRODUCT-DEMOS.md + playbook/spec/README updates making the demo a required, cost-checked part of publishing.

Eval result

19 submissions · 19 with demo · 0 without · mean score 98.9 (gate 60)

First-call proxy 1.00 for method-accurate demos (n/a for the two pointer-bundle submissions).

Verification

  • go test ./... green · gofmt clean · go vet clean on new packages.
  • go run ./cmd/pilot-app demo-score submissions passes the gate for all 19 apps.

Companion changes (separate branches, not in this PR)

  • website — "Full usage demo" section (mergeable independently).
  • pilotprotocol — additive, recover-guarded install-time render + SKILL injection, held for review (the only change to that repo; nothing regresses when the field is absent).

Alexgodoroja added 5 commits July 15, 2026 11:32
Introduce the product demo — a compact, example-driven, skill-file shaped
usage guide authored once per app in submission.json (alongside descriptions
and metadata). It flows into the catalogue metadata.json and renders from a
single source at three surfaces: the last step of `pilotctl appstore install`,
an injected SKILL.md, and the website "Full usage demo". Unlike <ns>.help
(which enumerates every capability), a demo is short and copy-pasteable: a
when_to_use trigger, one first call, 2-6 worked examples, and — for metered
apps — a budget-checked cost breakdown.

Motivation: autonomous agents install apps but never use them because the app
is never surfaced to a small context window. A demo drives correct first use.

- internal/demo: the frozen contract (Demo/Step/Cost types, Validate, and the
  RenderSkill/RenderInstall/RenderMarkdown renderers) with tests. Enforces
  skill-file discipline (skill==id, single-sentence when_to_use, length
  budgets), real <ns>.* commands, and — for metered apps — a cost table whose
  worked flow sums to <= the per-user broker budget. Handles dollar-metered,
  dynamic, and request-quota brokers.
- Wire product_demo through submission -> config -> metadata.json; validate at
  submit time. Additive and non-breaking (optional field; older clients ignore
  it). CI gate: TestAllSubmissionDemosValid.
- Backfill: all 19 submissions now carry a valid, method-accurate demo.
- publish-rich-from-r2.sh carries product_demo into the synthesized
  metadata.json (metadata_sha256 recomputed) so the existing signed-catalogue
  pipeline handles the rollout — no manual key handling.
- Evaluation system (internal/demoeval + `pilot-app demo-score`): a rubric
  quality score plus a deterministic first-call proxy (do the demo's commands
  reference real methods/params), and scripts/demo-telemetry.sh for the actual
  install-to-first-call conversion query. Mean score 98.9/100 across 19 apps.
- Docs: new docs/PRODUCT-DEMOS.md plus playbook/spec/README updates making the
  demo a required, cost-checked part of publishing.
…rmat, not auto-injection

Narrow the shipping surfaces to the two that are live — the install-time banner
and the website "Full usage demo". RenderSkill stays as a library helper that
emits the demo in skill-file format (frontmatter + body), but nothing
auto-injects a SKILL.md at install time. Update the package docs, PRODUCT-DEMOS,
playbook, and spec to match. No behavior change to validation, rendering data,
or the backfilled demos.
… gate)

io.pilot.sixtyfour is owned by a third-party publisher key; the submission
update gate (correctly) rejects any change to it that isn't re-signed by that
key. Editing its submission.json to add a product_demo therefore fails CI.
Leave the gated submission untouched — sixtyfour's demo ships via its catalogue
metadata.json instead (the ownership gate applies to submissions, not the
additive store-page metadata). Coverage is now 18/19 submissions; the 19th
(sixtyfour) is delivered at the metadata layer.
@Alexgodoroja
Alexgodoroja merged commit 4bae478 into main Jul 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant