fix: full unicode filename handling in 'writeUTF8' and 'sizeUTF8' - #21
Open
mehradotdev wants to merge 1 commit into
Open
fix: full unicode filename handling in 'writeUTF8' and 'sizeUTF8'#21mehradotdev wants to merge 1 commit into
mehradotdev wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: This PR is just for reference. Added a lot of comments to explain the code changes. Feel free to close it. If it doesn't suit your coding style.
Summary
Handle full Unicode code points correctly when encoding ZIP entry names.
This changes
writeUTF8andsizeUTF8so they process full Unicode code points instead of raw UTF-16 code units.It adds a small helper:
and uses it from both UTF-8 writer functions.
Why this is needed?
The current implementation iterates over
str.charCodeAt(ci), which reads UTF-16 code units one at a time. That works for BMP characters, but it breaks for characters represented by surrogate pairs.Examples that currently fail:
rare-kanji-𠮷.txtemoji-😀.txtThose names are encoded as invalid UTF-8 byte sequences, and when parsed back they become mojibake instead of the original filename.
No other ZIP logic is changed.
Benefit
Example behavior
BMP names like
日本語.txtwork as expected even before this change however Non-BMP are broken.Before:
rare-kanji-𠮷.txtround-trip asrare-kanji-í¡í¾·.txtemoji-😀.txtround-trip asemoji-í ½í¸.txtAfter:
rare-kanji-𠮷.txtround-trips correctlyemoji-😀.txtround-trips correctlySuggested validation
Encode and parse archives containing names like:
日本語.txt한국어.txtemoji-😀.txtrare-kanji-𠮷.txtThen verify the parsed keys exactly match the original names.