Fix broken data parsing: replace NDJSON binary-index with proper parse+Map - #985
Fix broken data parsing: replace NDJSON binary-index with proper parse+Map#985hparadiz wants to merge 1 commit into
Conversation
2c95311 to
ddb5410
Compare
|
Added in the caching fix and made items/stats be LF (always thought it was LF so surprised when it was using windows endings) |
|
For the rest, I'm not quite sure what you mean for parts of it. I haven't personally had any stability issues with it and stability should be garunteed by CI/CD.
Do you have some real examples? |
…rse+Map Prices.ts (overviewData.json): - The blob is valid JSON; replaced all manual string searching (parseXchg, splitJsonBlob, findPriceByQuery) with a single JSON.parse into OverviewBlob - Built a PriceIndex Map keyed by ns:name:variant for O(1) lookups - Fixed priceCache: object-literal Map keys never hit; changed to string keys - Removed PriceDatabase raw-string store and all start/end/slice logic assets/data/index.ts (items.ndjson, stats.ndjson): - Root cause: .ndjson files have CRLF line endings; binary .index.bin files were built against LF-only versions; byte offsets off by N-1 for line N, causing JSON.parse to slice from inside the previous line (false} at pos 5) - Replaced dataBinarySearch, ndjsonFindLines, itemNamesFromLines, and all .index.bin fetches with parseNdjson<T> (split on newline, JSON.parse each line), makeIterator<T> (pattern search on pre-serialized strings), and makeNameGenerator - loadItems: one fetch, parse, build byName/byRefName Maps - loadStats: one fetch, parse, build byRef/byMatcher Maps - Removed fnv1a import; .index.bin files are now unused IPC.ts: Host.proxy retries 3 times with status/body diagnostics before throw docs/data-parsing.md: full root cause analysis and documentation of both fixes
ddb5410 to
6da3280
Compare
|
This is basically a rewrite. The thing you're trying to do with the original code is load json and be able to instantly find stuff in it. This is one proposed way to do it. This is simpler in my opinion. The redesign loads the json and then constructs several in-memory Maps. The original code is an attempt to do the same thing but it's brittle in some spots and has clear bugs in others or ways in which extra white spaces in your json can throw off the result. The original code tries to do text searching within the json and then only loads the json based on searching for brackets and commas. The code itself has off by one errors in it and in some cases can be thrown off by extra \r\n. By loading the entire json and building maps you can have instant memory lookups. The Map gets built once during the load and then it's done. But like I haven't fully tested this code and you should for sure just look at it as an option. |
Summary
SyntaxError: Unexpected non-whitespace character after JSON at position 5:items.ndjsonandstats.ndjsonhave CRLF line endings but the.index.binfiles were built against LF-only versions. Byte offsets are off by N−1 for the Nth line, causingJSON.parseto slice from inside the previous line's content (landing onfalse}at position 5).Prices.tsoverviewData.json: The blob is valid JSON that was being parsed with fragile string searches ({"rates":,{"type":",}}boundary heuristic) that depended on undocumented server key ordering. Replaced with a singleJSON.parseand aMapindex.priceCache: was using object-literal Map keys (reference equality always false — cache never hit). Fixed to string keys.Host.proxy: now retries 3× with status/body diagnostics before throwing.Changes
renderer/src/assets/data/index.tsdataBinarySearch,ndjsonFindLines,itemNamesFromLines,fnv1aimport, and all.index.binfetchesparseNdjson<T>(split on\n,JSON.parseeach non-empty line — handles CRLF transparently),makeIterator<T>,makeNameGeneratorloadItems: one fetch → parse → twoMaplookups (ITEM_BY_TRANSLATED,ITEM_BY_REF)loadStats: one fetch → parse → twoMaplookups (STAT_BY_REF,STAT_BY_MATCH_STR)renderer/src/web/background/Prices.tsparseXchg,splitJsonBlob,findPriceByQuerystring-search approach replaced withJSON.parse→OverviewBlob→PriceIndexMapdocs/data-parsing.md: full root cause analysis of both issuesTest plan
Uncaught (in promise) SyntaxErrorin renderer consolenpm testin renderer passes