foca-reference-data

The Japanese reference database shipped inside Foca, published here because it has to be.

Four of its five sources are CC BY-SA 4.0, so the database is an adaptation and share-alike applies to it. This repository is where that obligation is discharged: every build the app ships is attached to a release, under the same licence as the data it came from.

You are welcome to use it for anything the licences below permit. It is not a general-purpose Japanese dictionary dump — it is shaped for one app's queries, and the "What's in it" section says how.

Download

Databases are release assets, not files in this repository. A 35 MB binary that changes every content release would make the clone larger forever.

gh release download --repo rio-sanjuan/foca-reference-data --pattern '*.sqlite'

Each release records the upstream versions it was built from, the row counts, and a SHA-256.

What's in it

SQLite, STRICT tables throughout, read-only by design — the app opens it with mode=ro and never writes.

Table	Rows	What
character	10,561	Kanji from KANJIDIC2 (10,384), plus both kana syllabaries enumerated from Unicode (86 hiragana, 91 katakana). Stroke count, grade, radical, frequency.
character_reading	~52,000	On, kun and nanori. Stored twice — かん.じる keeps the okurigana boundary a learner needs, かんじる is what a match compares against.
character_meaning	~25,000	English glosses, in KANJIDIC2's order. Kana carry a Hepburn gloss, except where a kana is structural rather than a syllable — っ is "geminate marker (sokuon)".
word	22,626	JMdict's common tier, with a blended frequency rank.
word_form	~60,000	Every spelling and reading, including search-only forms.
sense	~35,000	Part of speech and glosses, per sense.
word_furigana	~18,000	Which kana belong over which kanji.
sentence	27,677	Tatoeba sentences with a direct English translation.
sentence_token	~250,000	Segmentation with UTF-8 byte offsets, dictionary form, reading, and a JMdict link where one could be made.
cloze	33,787	One row per blankable word in a sentence, with a difficulty score, an i+1 hint, and precomputed distractors.
provenance	5	Source, version, licence and the exact credit line to render.

Two things about that schema are worth knowing before querying it.

Token offsets are UTF-8 bytes. Not characters, not graphemes. SQLite's substr() counts characters and will silently return the wrong span on any sentence mixing kana and ASCII. Slice the bytes.

There is no stroke-order data here, and no flag saying whether any exists. Stroke medians are Arphic PL, which imposes share-alike terms that do not compose with CC BY-SA, so they live in a separate package and nothing crosses over. Earlier releases carried a character.has_strokes boolean; it is gone, because it went stale whenever the templates versioned and this file did not.

link_source says how much to trust a token's dictionary link. 0 means the corpus named the JMdict sequence number outright; 1 means one entry matched; 2 means several did and one was picked by frequency. Only 0 and 1 produce cloze cards.

Licence

The database is CC BY-SA 4.0, inherited from its share-alike sources. Use it, adapt it, ship it commercially — but adaptations you distribute carry the same licence, and the sources below have to be credited.

Source	Licence
KANJIDIC2 — © EDRDG	CC BY-SA 4.0
JMdict — © EDRDG	CC BY-SA 4.0
JmdictFurigana	CC BY-SA 4.0
JLPT vocabulary lists, from Jonathan Waller's lists	CC BY-SA 4.0
Tatoeba	CC BY 2.0 FR

Full text in LICENSE. The same credit lines are in the database's own provenance table, which is what the app renders — attribution that lives only in a README drifts from the data the first time the pipeline changes.

Stroke-order data is not in here. AnimCJK's medians are Arphic Public Licensed, which is a different and incompatible share-alike regime, so they stay in stroke-recognition-templates and only a boolean crosses over.

Caveats

The JLPT levels are unofficial. The Japan Foundation has never published a vocabulary list; everything in circulation descends from Jonathan Waller's lists, which are openly described as an educated guess, and N2/N3 in particular are reconstructions from test-taker recall. Useful for ordering a curriculum, not authoritative.

character.jlpt_legacy is the pre-2010 four-level scale, which is what KANJIDIC2 ships. It is not N5–N1, and the column is named so it cannot be mistaken for it.

Frequency is a blend, and its parts are kept. freq_news comes from JMdict's nfXX buckets, which derive from a 1990s newspaper corpus and over-weight 政府 while under-weighting conversational vocabulary. jlpt pulls a listed word toward the front of its band as a crude correction. freq_rank is the blend the indexes sort on; the components stay in their own columns so that when the ordering looks wrong you can see which signal caused it.

About 40% of the segmentation is inferred. Tatoeba's own hand-checked jpn_indices annotations cover the rest. Where both cover a sentence, the builder measures rather than merges — the most recent build reports 77.5% boundary recall, 93.1% lemma agreement and 90.1% entry agreement against gold. sentence.has_indices says which half a row came from.

Cloze quality scores are guesses. They are a weighted sum of segmentation provenance, sentence length, word frequency, an i+1 shape check and a penalty for translations that give the answer away. The weights have never been validated against a learner.

Rebuilding

The generator is Tools/ReferenceBuilder in the Foca repository, and its --help documents how to fetch every source. It is not open source; this repository publishes the output, which is what the licences require.

Nothing here is hand-edited. If a row looks wrong, it came from upstream or from the builder, and the fix belongs in one of those.