A repeatable test bench for the CV-to-posting match: does an open model, run locally or hosted in the EU, answer the match questions as well as Jev? Since then it grew into BKS-Lab's general bench for local models: tool families compared per task (tool map) and capability tests C1 to C9 (retrieval, speech, invoice extraction and more, see plan/ROADMAP.md).
This is the one working repository of the bench: tests are designed, run, scored and published here. Until 2026-10-10 the work happened in a private repository (bks-lab/jev-match-bench, now archived as provenance history) and this repository received export commits; see Repository history and leak scan.
Jev's per-question outputs (the result rows under results/*/jev/) are
published with TypeSafe's permission of 2026-10-10
(record), for
reproducing the measurements only, and are not covered by the MIT or the
CC BY 4.0 licence of this repository, see Licence and
LICENSE-JEV-OUTPUTS.
The match on mboiman.github.io/<lang>/match/ asks TypeSafe's Jev many small
typed questions per posting line (is it a requirement, is it mandatory, which
area, which CV entry is the evidence, how well is it covered) and computes the
score in code. TypeSafe hosts its services in the USA (TypeSafe privacy
policy, https://typesafe.ai/legal/privacy-policy, linked from
https://docs.typesafe.ai/legal, read 2026-10-09). For a service that handles
other people's CVs we want to know whether a model we can run ourselves is
good enough, and we want to be able to repeat that test whenever a new model
appears.
Write-up of the results: Jev against local models: decisions on our own hardware on the BKS-Lab blog, and the lab page Decision models with the questions and numbers.
The bench now has three families of test cases. Each has its own cases, requests, reference and method file.
| family | task | data | reference | method |
|---|---|---|---|---|
pv1 (match) |
CV against a posting line: requirement, must, area, evidence, level | invented postings and CVs, de and en | blind model judges, adjudicated | results/pv1/method.md |
route1 (routing) |
put a short text into one of a fixed set of classes | MASSIVE 1.1 test (de, en, 2,974 each) and fastino/fast-decisions dev (en, single-label tasks) | the datasets' gold labels | results/route1/method.md |
ex1 (extraction) |
find the slots of a spoken command | 1,000 MASSIVE 1.1 test utterances per language | MASSIVE slot annotation | results/ex1/method.md |
Results, one line each (run 1 of each arm):
- pv1: Jev leads on axis and level, Qwen3.8-27B on evidence (87.9 % de against Jev 81.0 %), GLiNER2.5 Decide stays at the most-frequent-answer line (best mean of five questions 53.6 % de, baseline 54.3 %, Jev 86.9 %). gliner.md, crosscheck.md
- route1: on MASSIVE Jev is first (scenario 72.8 % de, 74.1 % en), Winnow-12B within 0.3 to 3.5 points, GLiNER 13.2 to 31.8 points behind; on fast-decisions (26 labels or fewer) the English Decide models, Jev and Winnow-12B are level at 63.0 to 64.4 %. findings.md
- ex1: slot F1 Winnow-12B 45.1 % de and 48.9 % en, best GLiNER 22.5 % de and 38.1 % en; Jev not applicable. findings.md
- Machine-readable summary of every measured cell for the tool map: results/toolmap.json.
Jev takes part in pv1 and route1. It has no extraction primitive, so ex1
compares a local LLM and GLiNER only. Sources, hashes and licences of the
route1 and ex1 data: cases/route1/README.md.
The full route1 and ex1 run is one command per machine:
bench/run-route-ex.sh (Jev and Ollama) and bench/run-route-ex.bat
(GLiNER on the GPU machine).
A second family is preregistered and mostly run (seven of nine tests):
capabilities of a local workstation, C1 to C9. These tests do not
compare against Jev. They ask what one in-house PC (RTX 4090, 24 GB) with
open models does on German business tasks: retrieval for RAG, speech to
text, invoice fields from rendered e-invoices, structured output and tool
calling, translation, reading comprehension, long context, throughput and
energy, and personal data detection. Each reports quality with a 95 %
bootstrap interval, speed, peak VRAM, GPU energy and the licence class of
the model, and each must first reproduce a published reference number
before other models are scored. Designs: plan/c0-common.md
and plan/c1.md to plan/c9.md; data, licences, run order and GPU hours:
plan/ROADMAP.md.
Measured so far (one RTX 4090, all sanity gates passed; C6 on its second run, see its findings):
-
C1 retrieval: Qwen3-Embedding-8B is best on MIRACL de (nDCG@10 0.613, 95 % CI 0.576 to 0.651) and separated from every other embedder, but indexes 44 passages per second; bge-m3, e5-large and nomic-v2-moe reach 0.56 to 0.58 at 840 to 1,200 per second and 3 GB. findings.md
-
C2 speech to text: canary-1b-v2 (WER 4.52 %) and whisper-large-v3 (4.70 %) are not separated on FLEURS de; parakeet-v3 (5.16 %) runs 182 times faster than real time. findings.md
-
C3 invoice fields from page images: gemma4-12b and gemma4-26b read 399 and 400 of 408 cells (0.978 and 0.980, not separated) at 2 to 4 s per invoice; OCR plus a text model is worse; the two Qwen3-VL arms did not finish within their time budget. findings.md
-
C4 structured output and tool calling: on BFCL v3 non-live (1,000 items) with native tool calling, qwen3-14b (0.919, 95 % CI 0.901 to 0.935, 10.3 GB) and qwen3.8-27b (0.906) are not separated; the capability map shows qwen3-14b. Putting the functions in the prompt beats native tool calling for five of the seven arms with both modes (qwen3-14b 0.946, gemma4-26b 0.947). findings.md
-
C6 German reading comprehension: the trained reader GELECTRA-large (F1 0.872, 1.7 GB, 28 ms per question) beats the best LLM, gemma4-12b (0.849, p 0.024); gemma4-12b is the recommendation among the LLMs. findings.md
-
C7 long context: at 32k tokens qwen3-14b, qwen3.8-27b and gpt-oss-20b answer all 180 needle prompts; the Gemma and Mistral misses are prompts that Ollama cut because
num_ctxhad no headroom, not lost needles. Lengths above 32k were not tested. findings.md -
C8 throughput and energy: at 16 parallel clients granite4-3b reaches 1,378 tokens per second and 260,731 decisions per hour at 165 J per 1,000 tokens; models whose 16 caches no longer fit next to the weights (for example gemma4-12b: 397.0 tokens per second at 8 clients, 146.2 at 16) fall back. findings.md
-
C5 translation German and English: on FLORES+ devtest (1,012 sentences per direction) gemma4-26b (chrF++ 68.1 de to en, 65.5 en to de) and gemma4-12b (67.9, 65.1) lead and are not separated (p 0.060 and 0.053); the capability map shows gemma4-12b (8.7 GB). The dedicated opus-mt is 3 to 4 points behind but runs at 6 segments per second at batch 1 and 85 at batch 32 on 1.9 GB. The translations are not committed (gated, ShareAlike), only the scores. findings.md
-
Capability map: results/capabilities.json, per-arm chart data in
results/c1/summary.jsontoresults/c8/summary.json(andresults/c5/summary.json,results/c9/summary.json). -
C9 personal data detection in German text (gold noisy): on the 476 German rows of the synthetic Gretel PII finance set, seven arms. The gold audit of 40 documents was done by a language model (Claude Haiku 5.5, spot check by Claude Opus 5.5), not by a human as the design requires; it found 43 of 149 gold spans wrong (gold precision 0.685, recall 0.981), so every number is descriptive and there is no ranking claim. Span F1 on the audited gold: gemma4-12b 0.839 and qwen3.8-27b 0.828 (not separated; on all 476 documents with the original gold 0.627 and 0.644), then gliner-pii-domains 0.738, gliner2.5-multi 0.649, gliner-multi-pii 0.444, Presidio 0.300, regex 0.150. The capability map shows gemma4-12b (9.2 GB). A human audit is still open. findings.md
- T2 stability under shuffled options: when only the option order changes, the local LLMs change 5.7 % (qwen3.8-27b), 7.8 % (winnow-12b) and 9.1 % (gemma4-12b) of their answers, mostly in the 18-option scenario question (10 to 14 %); the same call on a later day gives identical choices (0.0 % flips). GLiNER Decide flips 0 of 5,948 (read the caveat in the report). findings.md
- T9 typos and colloquial wording: typos cost every arm 3.0 to 4.2 points on MASSIVE de (all intervals exclude zero); missing umlauts and colloquial contractions cost 1.8 points or less, and GLiNER flips the most answers (18.2 % under typos). The perturbations are rule-based. findings.md
- T10 throughput with batching: GLiNER Decide goes from 69.8 decisions per second at batch 1 to 399.5 at batch 8 (5.7 times, identical answers); batches 32 and 128 are slower (380.6 and 354.4). findings.md
results/toolmap.json is the machine-readable summary: one row per task,
one cell per tool family (jev, gliner, local-llm). node bench/toolmap.mjs rebuilds it from the score reports, the pv1 references,
the result rows and the register plan/toolmap-plan.yaml. It recounts the
route1 and pv1 accuracies from the rows and stops if they differ from the
reports.
The capability tests C1 to C9 compare no tool families, so they sit apart,
under capabilities: one row per test with a single cell local, measured
from results/capabilities.json (written by bench/score-cap.py) or
planned with its design.
route1 and pv1 cells carry top_family (the family with the highest value in
that row) and mcnemar_vs_top_p: an exact paired McNemar between the cell's
best arm and the top family's best arm on the same items. Cells other than
Jev also carry mcnemar_vs_jev_p. Both compared arms are chosen after the
fact as the best of their family and no p value is corrected for that, so
p >= 0.05 says the difference is not established, not that the two are level.
ex1 cells have no test, and their informative skill is exact_frame_skill
(whole frames against extracting nothing), since F1 against an empty
extraction is 0 by construction.
Every cell has a state:
| state | meaning |
|---|---|
measured |
a value from a run, with best_arm, baseline, skill, version and source_report |
planned |
a dated test design is committed under plan/ (plan field), nothing is measured yet |
open |
a sensible test nobody has designed yet |
not_applicable |
the tool cannot do the task, reason says why (Jev on extraction) |
The register also carries tools: maker, how each tool runs, licence and
sources. A row can exist before any measurement. Planned tests are written
down in plan/ and committed before they run, with data, questions, arms,
reference, metric, baseline and thresholds fixed in advance, so the design
cannot be fitted to the results. The first one is
plan/einv1.md: ten yes/no questions over the free text of
34 KoSIT e-invoices.
- Frozen versions. A bench version (
pv1) never changes once the reference judgment starts. New cases make a new version. - Invented CVs only. Every arm may see every case, including APIs in the USA, and the cases can live in a repository.
- The service's own questions.
bench/match.tsis a vendored copy of the service'ssrc/lib/match.ts(source commit in its header). The bench measures what the service does, including its faults. - Every question on its own. The total score is derived and can hide errors that cancel out.
- The reference is decided blind, before anyone looks at model answers.
For pv1 that is three blind Opus judges, and for German a cross-check by
blind Sonnet and Fable judges with a researched adjudication of every
disagreement (
adjudicated-a, see Reference judgment pv1). A human sample is still pending.
| Path | What |
|---|---|
cases/pv1/ |
postings (en, de), 8 invented CVs (en, de), 10 pairs. See its README |
bench/match.ts |
vendored questions and scoring of the service |
bench/build-requests.mjs |
cases to requests/<pv>/requests.jsonl, the exact requests the service would send |
bench/run-jev.mjs |
Jev arm, TypeSafe API |
bench/run-local.mjs |
local arm through Ollama: options as letters, probabilities from the logprobs of the answer letter |
bench/run-gliner.py |
GLiNER2.5 Decide arm (Python, pip install "gliner2[local]"): one classification task per question, modes prompt, intext, --focus. Result: results/pv1/gliner.md |
bench/normalize.mjs |
one output row per question, the same for every arm |
bench/analyze.mjs |
report: run-to-run flips, option-order effect, agreement with Jev, total score per pair |
bench/significance.mjs |
paired exact McNemar per question, each arm against Jev on run 1, scored against a reference: node bench/significance.mjs --ref claude-a --lang de --md results/pv1/significance-de.md |
results/<pv>/<arm>/ |
one JSONL file per run |
bench/freeze-route-ex.mjs, bench/build-route-ex.mjs |
route1 and ex1: freeze the source data into cases/, then build requests and gold |
bench/run-local-extract.mjs, bench/run-gliner-extract.py |
ex1 arms: local LLM with JSON output, GLiNER2 entity extraction |
bench/score-route.mjs, bench/score-extract.mjs |
route1 and ex1 scorers, reports in results/route1/ and results/ex1/ |
bench/toolmap.mjs, plan/toolmap-plan.yaml |
tool map generator and its register, see Tool map |
bench/c9/, bench/cap_c9.py |
C9 personal data detection: runners, scorer on the Mac (score_c9.py) and the capability row; scores and the gold audit are committed, documents and result rows are not |
bench/c5/, bench/cap_c5.py |
C5 translation: runners, COMET, scorer on the Mac (score_c5.py) and the capability row; only scores are committed |
bench/t2/, bench/t9/, bench/t10/, bench/import-t.py |
T2, T9, T10: runners and scorer; import-t.py keeps compact rows (line, question, pass, choice) in results/t2/, results/t9/, results/t10/ |
plan/ |
dated test designs, committed before the run |
bench/judge-server.mjs, bench/judge.html |
blind reference judgment: node bench/judge-server.mjs, then http://localhost:4350/ (English cases and labels) or http://localhost:4350/?lang=de (German). Listens on 127.0.0.1 only. Never shows a model answer, saves every item at once, resumes where you stopped |
reference/<pv>/ |
blind reference judgment, <judge>.<lang>.jsonl, judges and models in judges.json. pv1: claude-a and claude-b (de), claude-c (en), all claude-opus-5-5, 2026-10-01; sonnet-a, fable-a and adjudicated-a (de), 2026-10-02 |
Needs Node.js 22.18 or later: the .mjs scripts import bench/match.ts
directly, which relies on Node's built-in type stripping. The GLiNER arm
needs Python 3 with pip install "gliner2[local]".
npm install
node bench/build-requests.mjs # cases -> requests
node bench/run-jev.mjs --runs 5 # needs TYPESAFE_API_KEY, see the script header
node bench/run-local.mjs --model qwen3:32b --template chatml --runs 1 # Ollama at OLLAMA_HOST or localhost:11434
python bench/run-gliner.py --model fastino/GLiNER2.5-Decide-1B --arm gliner2.5-decide-1b-focus --focus
node bench/analyze.mjs --md results/pv1/report.mdRun 5 of 5 is the shuffled run: the options of every choice question go out in
a seeded random order. --unit line|project|profile, --only p01,p02 and
--run-ids 1,5 narrow a run.
Jev returns a probability distribution per question. A raw model does not, so
run-local.mjs asks one question per call: the state as named sections, Jev's
instruction word for word, the options as letters A, B, C, and the assistant
turn pre-filled up to Answer: **. The probability of an option is the
probability of its letter as the next token, renormalised over the options.
mass in each row says how much probability the letters had before that; a
low mass means the model wanted to answer something else. Winnow-12B and
Shisa DE-1 ran with --template gemma4 --prefix ''. Shisa DE-1 is trained to
put the letter first. Winnow-12B's own server puts Answer:\n before the
letter (winnow-inference, native/protocol.h), which this template leaves out. The prefix is not neutral: on 15
questions Shisa changed 6 answers between '' and Answer: ** (2026-10-01).
Locally all 348 answers of a second qwen3:32b run on p01 and p04 repeated
exactly (2026-10-01), the probabilities did not (up to 0.19 apart), so
calibration needs repeated runs here
too. Full method, the recomputed numbers and every difference between the
Jev arm and the local arms: results/pv1/method.md.
- The service's line splitter drops one-word lines. In German,
"ISTQB-Zertifizierung" disappears, in English "ISTQB certification" stays,
so
sap-testmanagementhas 13 English and 12 German lines (2026-10-01).
Three blind Opus judges (model claude-opus-5-5, 2026-10-01, recorded in
reference/pv1/judges.json), none saw a model answer: claude-a and
claude-b judged the German lines independently, claude-c the English ones.
Scoring uses claude-a (de) and claude-c (en). claude-b is a comparison
row only. The English lines have a single judge and no such cross-check.
A and B agree on 100 % of is_req and must, 97 % of axis, 95 % of evidence and
level, so the gaps of the models below are real, not judge noise. Caveat: two
judges from the same model family can share a blind spot; a human sample on a
few pairs is still worth doing before a public claim. bench/score.mjs scores
every arm against a reference (results/pv1/score-de.md, score-en.md).
Cross-check (2026-10-02, German only): two more blind judges from other
families, sonnet-a (claude-sonnet-5-5) and fable-a (claude-fable-5-1),
judged the same 636 pairs. The three agree on 90.4 % of them, on all is_req
and must answers. Each of the 61 disagreements went to a fresh researched
adjudicator (Sonnet or Fable); 26 verdicts changed the claude-a choice (13 of
them raise a level by one, 5 replace an entry with none). The result is
reference/pv1/adjudicated-a.de.jsonl (changed rows marked
"adjudicated": true), scored in results/pv1/score-de-adjudicated.md and
significance-de-adjudicated.md. Method, verdict counts and which findings
hold: results/pv1/crosscheck.md. In short: Qwen3.8-27B still invents
evidence least (now p = 0.016 against Jev) and is now first on evidence; Jev
stays first on level and axis, but Winnow-12B is no longer significantly
behind Jev on axis.
The note field in the reference files is the judges' raw output, kept
verbatim. Some judges wrote their notes in German, and some notes quote
third-party web pages with their URLs (see Licence).
Found by the judges: the service cuts CV entries at 600 characters
(plain() in match.ts), three entries end mid-sentence: s-claims-agents in
cv03 (de and en), s-s4-retail-testmgr in cv02 (de), s-trade-lakehouse in cv07
(de), found by scanning requests/pv1/requests.jsonl. The judges flagged only
p04-de-l09. Most is_req
disagreements are the title and frame lines (title, duration, location), which
the judges call no and every model calls yes; that is a definition question.
Which arm ran on which family, run 1 unless noted. "n/a" means the tool has no way to answer that kind of question, "not run" means it could and was not measured yet.
| tool family | arm | pv1 (match) | route1 (routing) | ex1 (extraction) |
|---|---|---|---|---|
| Jev | jev | runs 1 to 5 | run 1 | n/a (choice, score and noul only) |
| local LLM | winnow-12b | yes | yes | yes |
| local LLM | qwen3.8-27b | yes | not run | not run |
| local LLM | qwen3-32b | yes | not run | not run |
| local LLM | shisa-de-1 | yes | not run | not run |
| GLiNER Decide | gliner2.5-multi-decide (prompt, intext, focus) | yes | prompt, intext | n/a (classifier) |
| GLiNER Decide | gliner2.5-decide (prompt, intext, focus) | yes | prompt, intext | n/a (classifier) |
| GLiNER Decide | gliner2.5-decide-1b (prompt, intext, focus) | yes | prompt, intext | n/a (classifier) |
| GLiNER extraction | gliner2.5-multi | n/a | n/a | de, en |
| GLiNER extraction | gliner2-large | n/a | n/a | en only |
focus is a pv1 adaptation (only the state sections a question names) and
does not apply to route1, whose state is one short text. Planned next: the
e-invoice yes/no test in plan/einv1.md, written down before
it runs. The local LLMs other than Winnow-12B were left out of route1 and ex1
on purpose: Winnow-12B was the best local arm in pv1, and every extra arm
makes "best arm of the family" more flattering (see the note in
results/toolmap.json).
References: pv1 de against adjudicated-a (a model judgment cross-checked by
two more model judges) and the first judge claude-a, pv1 en against a
single model judge claude-c. route1 and ex1 against the gold labels of the
public datasets. No human reference sample yet.
The bench used to have two homes: a private working repository
(bks-lab/jev-match-bench) and this public one, filled by export commits
that copied an allow-listed part of the private tree. The split existed
because Jev's per-question outputs could not be published. TypeSafe
permitted that on 2026-10-10, so the split was dropped the same day: the
full working tree (code, plans, queue tooling, results) was added here as
one commit on top of the last export, naming its source commit, and the
private repository was archived. Its history stays there, unchanged, so a
jev-match-bench commit id cited elsewhere (for example on bks-lab.com)
still resolves for BKS-Lab; every export commit in this repository names
the private commit it was made from. The schema strings inside the
generated JSON files (jev-match-bench toolmap v2, jev-match-bench capability summary v1 and so on) keep the old name on purpose: they are
format identifiers that readers check, not the repository name.
What used to be the export's job is now a check on every push and pull
request (.github/workflows/ci.yml):
node bench/leak-scan.mjsscans every tracked file with its configurationbench/leak-scan.json: a positive list of paths (a new kind of file fails until someone adds it with a reason), host, account, tailnet and keychain names of the lab machines, kept in the configuration only as SHA-256 hashes so the scan does not publish what it guards, patterns for private IP ranges, home paths, mail addresses, Windows SIDs and API key shapes, the Jev rule (Jev rows carry only the fields ofjevRows.fields) and the notices (LICENSE-JEV-OUTPUTS at the root, aNOTICE.mdnext to every set of Jev rows). Run it before you push. Further words for your own machine go intoLEAK_SCAN_TERMSinbench/local.env;node bench/leak-scan.mjs --hash <word>prints the line to add one to the committed list.- the queue runner tests, a syntax check of every script, the
reproducibility of
results/toolmap.jsonfrom the committed rows, and the independent recomputesbench/check-cap.py(C2, C3; C1 needs qrels that are not committed) andbench/check-t12.py.
Nothing machine-specific is committed. The queue scripts read the ssh
target of the GPU PC and its Python path from the environment or from
bench/local.env (gitignored; copy bench/local.env.example). The
scheduled task on the PC is a template that bench/queue/install.sh
fills with the account SID it reads over ssh. Job templates use
placeholders ({py}, {data}, {hf}, {out}) that the runner fills
from its own defaults and config.json on the PC.
Inputs that are not committed, and how to get them:
- C1 texts and qrels (MIRACL is ShareAlike and large):
cases/c1/fetch.py, hash-checked againstcases/c1/manifest.json. - C2 audio (FLEURS):
cases/c2/fetch.py, hash-checked againstcases/c2/manifest.json. - C5 texts and the T3 train split: see
plan/c5.mdandplan/t3.md. - Model weights:
bench/c1/prefetch.py,bench/c2/prefetch.py(pinned revisions intoHF_HOME) and the Ollama tags with digests in each plan. - Anything with personal data, or anything a plan marks as not
redistributable, goes under
private/(gitignored) or.cache/, never intocases/. The C9 set is synthetic (Gretel, Apache-2.0) but its documents stay out of git (push protection flagged invented keys in two of them);bench/c9/freeze.pyrebuilds them byte for byte (SHA-256 incases/c9/manifest.json).
GPU jobs run one at a time on one workstation from a queue folder
(D:\jmb-queue, the name predates the rename and is kept so the
scheduled task bench-queue keeps working). The runner
(bench/queue/runner.py, spec in plan/ROADMAP.md)
takes job files in name order; agents only enqueue and fetch:
bench/queue/status.sh # running job, queue, last ten results
bench/queue/enqueue.sh bench/queue/jobs/selftest.json # uploads origin/main as src/<sha>.tar, then the job
bench/queue/fetch.sh <job id> # out dir, log and status into .cache/queue/<id>/
bench/queue/hold.sh set <name> "<reason>" # keep GPU jobs back
bench/queue/install.sh --update # new runner.py; only when the queue is idleEvery job runs the commit it names (repo_ref), extracted from a
git archive of this repository, so the PC needs no clone.
The local arms ran on a single workstation with an RTX 4090. Result rows
record it as local-rtx4090 in the host field.
- Code (everything under
bench/exceptbench/match.ts): MIT, see LICENSE. - Data (cases, requests, reference judgments and results under
cases/,requests/,reference/andresults/): CC BY 4.0, see LICENSE-DATA, except the Jev outputs below. bench/match.tsis a vendored copy from mboiman/mboiman.github.io and stays under CC BY 4.0 with the source credit in its header.- Jev outputs (the result rows under
results/*/jev/) are not covered by the MIT or the CC BY 4.0 licence. They are outputs produced by TypeSafe Jev, published with TypeSafe's permission of 2026-10-10 (record), for reproducing the measurements only. They must not be used to train or distil models or to build a product that competes with TypeSafe. Use is subject to TypeSafe's Acceptable Use Policy and Master Customer Agreement, section 2.3(b). Full notice: LICENSE-JEV-OUTPUTS, and aNOTICE.mdin everyresults/*/jev/directory. - Quotes from third-party sources inside the judges' notes (the
Sources:parts with URLs) belong to their authors and are not covered by the CC BY 4.0 grant. - The route1 and ex1 data are not BKS-Lab's own. The MASSIVE rows
(
cases/route1/massive-*.jsonl,cases/ex1/, the requests, gold rows and result rows built from them) are CC BY 4.0 by Amazon and include text from SLURP, also CC BY 4.0.cases/route1/fastdec-en.jsonland what is built from it are Apache-2.0, published by Fastino, see LICENSES/Apache-2.0.txt. Credits and details: exceptions 4 and 5 of LICENSE-DATA and cases/route1/README.md.