Repository navigation
types: keep JSON string encoding valid UTF-8 - #505
Merged
Merged
Conversation
Strings holding bytes that do not form a valid UTF-8 sequence were
emitted verbatim by the JSON encoder, so encoding such a value (e.g.
via printf("%J", ...) or by passing objects to print() or writefile())
produced JSON that is not valid UTF-8, which JSON parsers reject
outright.
Validate the bytes as RFC 3629 while encoding: pass valid sequences
through unchanged and replace each invalid byte by the unicode
replacement character. The encoded output is valid UTF-8 no matter
which bytes the original string contained, at the cost of the
information carried by the invalid bytes only.
The plain I/O paths (fs and io read/write, print) are unaffected and
remain byte-transparent, so binary data round-trips through them
losslessly.
Fixes: #501
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The JSON encoder emitted string bytes
>= 0x80verbatim, so encoding astring that contained invalid UTF-8 (e.g. file contents read via
fs.readfile()passed toprintf("%J", …), or toprint()/writefile()as part of an object) produced JSON that is not valid UTF-8, which JSON
parsers reject outright.
Changes
types.c—ucv_to_string_json_encoded():utf8_sequence_length(): no overlongforms, no surrogates, no codepoints above U+10FFFF, no truncation);
\ufffd, so the encoded output isvalid UTF-8 no matter which bytes the original string contained.
This is the single choke point for all JSON output (
%J,print()/writefile()of objects/arrays,__tostring__in JSONcontext).
Behaviour
%Joutput"a" + chr(0xff) + "b""a\ufffdb""caf" + chr(0xc3, 0xa9)"café"(unchanged)chr(0xc0, 0xaf)(overlong)"\ufffd\ufffd"chr(0xed, 0xa0, 0x80)(surrogate)"\ufffd\ufffd\ufffd""a" + chr(0xe2, 0x82)(truncated)"\ufffd\ufffd"One replacement character per invalid byte — no guessing at how many
bytes a truncated sequence would have consumed, so no valid bytes get
swallowed.
Tests
03_stdlib/69_json_encode_utf8— new: encoding behaviour above03_stdlib/01_chr,03_stdlib/27_sprintf— expected output updated(raw high bytes in expected JSON are now
\ufffd)Full custom test suite: 277/277 pass.