test(tounicode): cover UniGB-UCS2-H without ToUnicode - #425
Open
yzxcj797 wants to merge 5 commits into
Open
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 5 files
Not reviewed (too large): src/adobe_gb1.rs (~28,678 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 3 files (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
…edefined CMaps A Type0 font whose /Encoding names a predefined CMap like GBK-EUC-H (not Identity-H/V) was skipped during ToUnicode CMap discovery, so its text degraded to U+FFFD and could misroute the page to OCR (firecrawl#400). Recognize any predefined encoding CMap shipped in external/bcmaps (name-whitelisted - untrusted /Encoding values are never used as filesystem paths), parse the binary CMap differential operators correctly (signed CID deltas, hex increments, codespace/notdef operand consumption), and map GB1 CIDs to Unicode through a new Adobe-GB1 table generated from cmap-resources UniGB-UCS2 (plain ideographs preferred over Kangxi/compatibility forms), mirroring the existing Adobe-Korea1 module. The regression test builds a synthetic Type0/GBK-EUC-H PDF without ToUnicode and asserts end-to-end extraction of multi-byte GBK codes.
Keep binary CMap code-space ranges instead of collapsing every encoding to its maximum code width. Also reject overflowing differential codes, decode wide zigzag deltas without i32 truncation, and check bundled CMap availability without reading the full file.
Namespace predefined encoding fallback keys by the Type0 font identity components instead of the shared CIDFont object, so GBK-EUC-H and UniGB-UCS2-H mappings can coexist. Keep the end-to-end fixtures focused on extraction and add a direct collision regression test.
yzxcj797
force-pushed
the
fix/364-unigb-ucs2
branch
from
August 19, 2026 01:10
9087b34 to
9301cb0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue #364 reports CJK text becoming U+FFFD for Type0 fonts using
/UniGB-UCS2-H, with no/ToUnicodeand no embedded font program. This PR adds the exact regression requested by that report.The new synthetic PDF uses
/Encoding /UniGB-UCS2-H,/BaseFont /SimSun, Adobe-GB1CIDSystemInfo, no/ToUnicode, and noFontFile*. Its content stream contains UTF-16BE character codes for the issue sample电子发票 G342 郴州西. The test checks both the composed font CMap and full text extraction.No additional runtime change was needed: the predefined-CMap machinery and Adobe-GB1 collection table from #422 already load
UniGB-UCS2-H.bcmap, compose its code-to-CID mappings with the GB1 CID-to-Unicode mapping, and recover the expected text. The bundled CMap filename also matches exactly; there is no filename or parser gap for this encoding.Relationship to #422
This branch is stacked directly on #422 (
5e2852bbd3d1196834f5eb0b7fd754aa51d3cb76) and adds a test-only follow-up commit (5855a66e816bd7a616ca4f5c298455378ea6aca0). It should be reviewed after #422. If #422 merges first, this PR is the issue-#364-specific regression proof; until then, GitHub will also show #422's implementation commits in the stacked diff.Fixes #364
Testing
cargo test --lib— 955 passedcargo test --test integration_tests— 164 passed (including the new UniGB-UCS2-H regression)cargo fmt --check— cleancargo clippy --lib --tests— completed successfully; no new diagnostic is introduced by the changed test code (the repository already has pre-existing warnings)Summary by cubic
Decodes CJK text for Type0 fonts that use predefined encoding CMaps (e.g., GBK-EUC-H, UniGB-UCS2-H) when /ToUnicode is missing. Previously these non‑Identity encodings were skipped and produced U+FFFD (sometimes triggering OCR); now we compose bundled encoding CMaps with the Adobe‑GB1 CID→Unicode mapping and preserve mixed‑width code spaces.
/Encodingnames (never treated as filesystem paths) and builds a namespaced 64‑bit lookup key per Type0+encoding+CIDFont so GBK‑EUC‑H and UniGB‑UCS2‑H can coexist.adobe_gb1CID→Unicode table (prefers plain ideographs); Identity‑H/V still tries embedded TrueType/OpenType first, then collection mapping; non‑Identity predefined encodings compose encoding→CID with GB1 CID→Unicode.get_font_cmap_lookup_key/get_by_cmap_lookup_key./ToUnicode, mixed‑width decoding assertions, and a key‑collision regression.Written for commit 9301cb0. Summary will update on new commits.