Skip to content

Expose EmbeddingGemma's tokenizer, as CoreMLLLM already does - #171

Open
gen-0s wants to merge 1 commit into
john-rocky:mainfrom
gen-0s:meetingintel/tokenizer-ref
Open

Expose EmbeddingGemma's tokenizer, as CoreMLLLM already does#171
gen-0s wants to merge 1 commit into
john-rocky:mainfrom
gen-0s:meetingintel/tokenizer-ref

Conversation

@gen-0s

@gen-0s gen-0s commented Sep 11, 2026

Copy link
Copy Markdown

EmbeddingGemma.load builds an AutoTokenizer from hf_model and keeps it private. A caller that needs a token count has no way to reach it.

That count is not a nicety for this model: encode truncates at config.maxSeqLen silently —

var ids = tokenizer.encode(text: prefixed)
let L = config.maxSeqLen
if ids.count > L { ids = Array(ids.prefix(L)) }

— no throw, no flag, and the published embeddinggemma-300m-coreml bundle sets max_seq_len to 128. Anything longer is indexed as its first ~128 tokens and reports success. Splitting text into windows before calling encode is the fix, and that needs to know how many tokens a span costs.

Today the only way to get that is to build a second AutoTokenizer over the same folder this instance already read. For EmbeddingGemma's 262,144-entry vocab and 514,906 merges that measures 1.9 s and +92 MB in our app, duplicating an object already resident in the process.

CoreMLLLM in this same package already solves exactly this, at Sources/CoreMLLLM/CoreMLLLM.swift:161-163:

/// Expose the tokenizer so a harness can re-encode / decode arbitrary
/// text without duplicating the swift-transformers wiring.
public var tokenizerRef: any Tokenizer { tokenizer }

This PR is that accessor on EmbeddingGemma, for the same reason. One line plus a doc comment; no behaviour change, no new dependency, nothing else touched.

Happy to also send a follow-up making the truncation visible (returning the token count, or throwing when ids.count > L) if you'd take it — that's the underlying defect, and this accessor only lets callers work around it.

`EmbeddingGemma.load` builds an `AutoTokenizer` from `hf_model` and keeps
it private. A caller that needs a token count — to window text against
`config.maxSeqLen`, which `encode` otherwise truncates to silently — has
no way to reach it, so it builds a second one from the same four files.
For EmbeddingGemma's 262,144-entry vocab and 514,906 merges that costs
about 4.5 s and 92 MB on every load, duplicating an object already
resident in the process.

`CoreMLLLM` in this same package already solves this with `tokenizerRef`
and says so in its doc comment. This is that accessor, on the embedding
class, for the same reason.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant