Skip to content

Fix broken data parsing: replace NDJSON binary-index with proper parse+Map - #985

Open
hparadiz wants to merge 1 commit into
Kvan7:devfrom
hparadiz:hparadiz/fix-data-parsing
Open

Fix broken data parsing: replace NDJSON binary-index with proper parse+Map#985
hparadiz wants to merge 1 commit into
Kvan7:devfrom
hparadiz:hparadiz/fix-data-parsing

Conversation

@hparadiz

@hparadiz hparadiz commented Jun 19, 2026

Copy link
Copy Markdown

Summary

  • Root cause of SyntaxError: Unexpected non-whitespace character after JSON at position 5: items.ndjson and stats.ndjson have CRLF line endings but the .index.bin files were built against LF-only versions. Byte offsets are off by N−1 for the Nth line, causing JSON.parse to slice from inside the previous line's content (landing on false} at position 5).
  • Prices.ts overviewData.json: The blob is valid JSON that was being parsed with fragile string searches ({"rates":, {"type":", }} boundary heuristic) that depended on undocumented server key ordering. Replaced with a single JSON.parse and a Map index.
  • priceCache: was using object-literal Map keys (reference equality always false — cache never hit). Fixed to string keys.
  • Host.proxy: now retries 3× with status/body diagnostics before throwing.

Changes

renderer/src/assets/data/index.ts

  • Removed dataBinarySearch, ndjsonFindLines, itemNamesFromLines, fnv1a import, and all .index.bin fetches
  • Added parseNdjson<T> (split on \n, JSON.parse each non-empty line — handles CRLF transparently), makeIterator<T>, makeNameGenerator
  • loadItems: one fetch → parse → two Map lookups (ITEM_BY_TRANSLATED, ITEM_BY_REF)
  • loadStats: one fetch → parse → two Map lookups (STAT_BY_REF, STAT_BY_MATCH_STR)
  • All exported function signatures unchanged; no callers updated

renderer/src/web/background/Prices.ts

  • parseXchg, splitJsonBlob, findPriceByQuery string-search approach replaced with JSON.parseOverviewBlobPriceIndex Map
  • Cache key changed from object literal to string

docs/data-parsing.md: full root cause analysis of both issues

Test plan

  • App loads without Uncaught (in promise) SyntaxError in renderer console
  • Price check returns results for items
  • Stat lookups work (map check, item filters)
  • npm test in renderer passes

@hparadiz
hparadiz force-pushed the hparadiz/fix-data-parsing branch 2 times, most recently from 2c95311 to ddb5410 Compare June 19, 2026 06:25
@hparadiz hparadiz mentioned this pull request Jun 19, 2026
@Kvan7

Kvan7 commented Jun 19, 2026

Copy link
Copy Markdown
Owner

Added in the caching fix and made items/stats be LF (always thought it was LF so surprised when it was using windows endings)

@Kvan7

Kvan7 commented Jun 19, 2026

Copy link
Copy Markdown
Owner

For the rest, I'm not quite sure what you mean for parts of it. I haven't personally had any stability issues with it and stability should be garunteed by CI/CD.

has several confirmed bugs that cause runtime crashes

Do you have some real examples?

…rse+Map

Prices.ts (overviewData.json):
- The blob is valid JSON; replaced all manual string searching (parseXchg,
  splitJsonBlob, findPriceByQuery) with a single JSON.parse into OverviewBlob
- Built a PriceIndex Map keyed by ns:name:variant for O(1) lookups
- Fixed priceCache: object-literal Map keys never hit; changed to string keys
- Removed PriceDatabase raw-string store and all start/end/slice logic

assets/data/index.ts (items.ndjson, stats.ndjson):
- Root cause: .ndjson files have CRLF line endings; binary .index.bin files
  were built against LF-only versions; byte offsets off by N-1 for line N,
  causing JSON.parse to slice from inside the previous line (false} at pos 5)
- Replaced dataBinarySearch, ndjsonFindLines, itemNamesFromLines, and all
  .index.bin fetches with parseNdjson<T> (split on newline, JSON.parse each
  line), makeIterator<T> (pattern search on pre-serialized strings), and
  makeNameGenerator
- loadItems: one fetch, parse, build byName/byRefName Maps
- loadStats: one fetch, parse, build byRef/byMatcher Maps
- Removed fnv1a import; .index.bin files are now unused

IPC.ts: Host.proxy retries 3 times with status/body diagnostics before throw

docs/data-parsing.md: full root cause analysis and documentation of both fixes
@hparadiz
hparadiz force-pushed the hparadiz/fix-data-parsing branch from ddb5410 to 6da3280 Compare June 22, 2026 07:26
@hparadiz

Copy link
Copy Markdown
Author

This is basically a rewrite. The thing you're trying to do with the original code is load json and be able to instantly find stuff in it. This is one proposed way to do it. This is simpler in my opinion. The redesign loads the json and then constructs several in-memory Maps. The original code is an attempt to do the same thing but it's brittle in some spots and has clear bugs in others or ways in which extra white spaces in your json can throw off the result. The original code tries to do text searching within the json and then only loads the json based on searching for brackets and commas. The code itself has off by one errors in it and in some cases can be thrown off by extra \r\n. By loading the entire json and building maps you can have instant memory lookups. The Map gets built once during the load and then it's done. But like I haven't fully tested this code and you should for sure just look at it as an option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants