fix: reuse one tokenizer instead of building one per countTokens call

[kernel-oops]

Fixes #638

@anthropic-ai/tokenizer's countTokens() builds and frees a new WASM Tiktoken on every call. That takes about 70 ms, against about 0.19 ms for encode() on a tokenizer that's reused. syncToolCache counts every uncached tool part on each chat-messages transform, and it re-counts entries once MAX_TOOL_CACHE_SIZE evicts them. On long sessions this blocked the host's event loop for seconds per request, which also stalled every other session on the same OpenCode server.

Change

lib/token-utils.ts now gets the package's getTokenizer, builds the tokenizer lazily on first use and reuses it. It keeps the same NFKC normalisation and "all" special-token handling as countTokens(), so the counts are identical. The existing try/catch in countTokens still falls back to the length estimate if anything throws.

Verification

• New test in tests/token-counting.test.ts: the counts match the package's countTokens() on plain, Unicode (ligatures, emoji), JSON, long and special-token inputs, and 200 calls finish within 2 s. The timing assertion fails on the current code, where 200 calls take about 14 s.
• npm run format:check, npm run typecheck and npm test all pass (120 tests, plus 8 in tests/logger).
• Found with sampling CPU profiles of an OpenCode v2 server during its multi-second stalls, where countTokens was the stack beneath syncToolCache.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RG8UhiogA1Ho2eZRx8tn2y

[Abdullah1738]

+1 here this is a critical issue