diff --git a/benchmarks/.gitignore b/benchmarks/.gitignore new file mode 100644 index 000000000..c2658d7d1 --- /dev/null +++ b/benchmarks/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/benchmarks/openui-bench/.gitattributes b/benchmarks/openui-bench/.gitattributes new file mode 100644 index 000000000..8f53c213a --- /dev/null +++ b/benchmarks/openui-bench/.gitattributes @@ -0,0 +1,8 @@ +# Benchmark data, not hand-written code: collapse in diffs and keep out of +# language stats. raw/ is verbatim model output, results/ is score.mjs output, +# the two a2ui artifacts are generated by protocols/a2ui/prompt.py. +raw/** linguist-generated=true +results/** linguist-generated=true +protocols/a2ui/catalog-a2ui.json linguist-generated=true +protocols/a2ui/catalog-surface.json linguist-generated=true +protocols/a2ui/system-prompt.txt linguist-generated=true diff --git a/benchmarks/openui-bench/.gitignore b/benchmarks/openui-bench/.gitignore new file mode 100644 index 000000000..34d7cb1f5 --- /dev/null +++ b/benchmarks/openui-bench/.gitignore @@ -0,0 +1,4 @@ +node_modules/ +__pycache__/ +*.pyc +.venv/ diff --git a/benchmarks/openui-bench/README.md b/benchmarks/openui-bench/README.md new file mode 100644 index 000000000..59e2f4d57 --- /dev/null +++ b/benchmarks/openui-bench/README.md @@ -0,0 +1,147 @@ +# openui-bench + +Reliability benchmark for three generative-UI formats over one shared +component catalog: [OpenUI Lang](https://github.com/thesysdev/openui), +Google [A2UI](https://github.com/a2ui-project/a2ui) (v0.9), and Vercel +[json-render](https://github.com/vercel-labs/json-render) (0.19). Results are +published in the [blog post](https://thesys.dev/blog/generative-ui-benchmark); +this directory holds everything needed to check or extend them: the briefs, +the catalog, each format's prompt and validator built from its own SDK, every +raw model output, and the scored verdicts. + +## Headline setup + +- 46 screen briefs in 5 size bands (2 to 18 numbered requirements). None names + a component or layout. +- 70-component catalog derived from OpenUI's public component library: the + open-source chat set, six chat blocks from the same library, and twelve + components from the public shadcn-chat example. `catalog/public-catalog.json` + is the single source all three protocol surfaces derive from. +- 6 models, one seat per company: GPT-5.6 Sol, Claude Opus 4.8, Kimi K3, + Gemini 3.6 Flash, Qwen3.8 2.4T, Muse Spark 1.2 (plus an extra Terra run kept + out of the averages, one OpenAI seat only). +- One uniform condition for every model and format: 4 generations per brief, + temperature 0.7 (Anthropic runs its model default; the API rejects setting + it), reasoning minimal/none, 16,384-token output ceiling. 1,104 scored runs + per format. Gemini's json-render and A2UI legs were generated with 10 + repeats before the 4-rep rule was settled; the scored set is the first 4 + (fixed rule, not outcome-selected) and only those are committed. +- Each format's prompt comes from its own SDK's generator: OpenUI through + `generatePrompt` with its official options (component groups, two additional + rules, two worked examples; see `protocols/openui/prompt.ts`), json-render + through `catalog.prompt()` with three custom rules, A2UI through the agent + SDK's `DirectJsonFormat` generator as-is. +- Each format's validation is its own SDK's shipped code plus one shared + completeness layer (identical rules for all three) described under + "Judgment calls". + +## Layout + +| Path | What it is | +|---|---| +| `briefs/` | The 46 briefs as data (`briefs.mjs`) and the band design (`DESIGN.md`). | +| `catalog/public-catalog.json` | The shared 70-component catalog. | +| `protocols/openui/` | catalog, prompt (lang-core `generatePrompt`), validator (lang-core parser). | +| `protocols/jsonrender/` | catalog (`defineCatalog` with Zod), prompt (`catalog.prompt()`), validator (their stream compiler, `validateSpec`, Zod gate). | +| `protocols/a2ui/` | catalog + prompt generation (official python SDK), scorer (`score.py`), renderer gate (`validator.mjs`, `@a2ui/web_core` MessageProcessor), and the generated `catalog-a2ui.json` / `system-prompt.txt` the runs consumed. | +| `run.mjs` | Generation runner (any OpenAI-compatible provider, Anthropic, Google). | +| `score.mjs` | Offline scorer: replays every raw through the validators, no API keys needed. | +| `tools/` | Token counts, cost estimates, blank-screen floor. | +| `raw/` | Every scored model output, verbatim: `raw/