Skip to content

Commit cc75830

Browse files
committed
Normalize lone surrogate halves at the clipText boundary
clipText only repaired lone halves its own cut created, so untrusted input that already contained a lone surrogate in the middle passed through and still broke UTF-8 consumers (a Postgres insert rejects it). Each lone half is now normalized to U+FFFD up front — the same treatment String.prototype.toWellFormed() applies — keeping the clip guarantee and closing the no-cut path in one move. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
1 parent 6f972ad commit cc75830

2 files changed

Lines changed: 36 additions & 2 deletions

File tree

‎apps/server/src/text.ts‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,22 @@
44
* character, renders as a replacement character and breaks consumers that
55
* encode the result to UTF-8. When the cut lands between a pair's high and
66
* low half, the high half is dropped with it.
7+
*
8+
* Untrusted input may already contain a lone surrogate half — malformed
9+
* upstream, a bad copy/paste — that no cut of ours produced. Each lone half
10+
* is normalized to U+FFFD before the clip, the same treatment
11+
* `String.prototype.toWellFormed()` applies, so the result is always
12+
* well-formed UTF-16 and survives a UTF-8 encode even when no cut happens.
713
*/
14+
const LONE_SURROGATE_HALF =
15+
/[\uD800-\uDBFF](?![\uDC00-\uDFFF])|[\uDC00-\uDFFF](?<![\uD800-\uDBFF])/gu;
16+
817
export function clipText(text: string, limit: number): string {
9-
const clipped = text.slice(0, limit);
10-
return text.length > limit && /[\uD800-\uDBFF]$/.test(clipped) ? clipped.slice(0, -1) : clipped;
18+
const wellFormed = text.replace(LONE_SURROGATE_HALF, "\uFFFD");
19+
const clipped = wellFormed.slice(0, limit);
20+
return (
21+
wellFormed.length > limit && /[\uD800-\uDBFF]$/.test(clipped)
22+
? clipped.slice(0, -1)
23+
: clipped
24+
);
1125
}

‎tests/clip-text.test.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,23 @@ test("does not trim when the string is exactly the limit", () => {
3939
const text = "\u{1F44D}".repeat(6); // 12 UTF-16 units
4040
assert.equal(clipText(text, 12), text);
4141
});
42+
43+
test("normalizes a lone surrogate already present in the input", () => {
44+
// "ab" + a lone high surrogate + "cd": the input is malformed without any
45+
// cut, so repairing only cut-created halves would let it through.
46+
const clipped = clipText("ab\uD800cd", 100);
47+
assert.equal(clipped, "ab\uFFFDcd");
48+
// the result survives a UTF-8 round trip, e.g. a Postgres insert
49+
assert.equal(Buffer.from(clipped, "utf8").toString("utf8"), clipped);
50+
});
51+
52+
test("normalizes a lone low surrogate in the input", () => {
53+
assert.equal(clipText("ab\uDC01cd", 100), "ab\uFFFDcd");
54+
});
55+
56+
test("normalizes lone halves but keeps a well-formed pair in the same text", () => {
57+
// A lone high surrogate followed by another high surrogate, then a valid
58+
// emoji pair: only the halves that lack a partner are replaced.
59+
const clipped = clipText("a\uD800\uD801b\u{1F44D}c", 100);
60+
assert.equal(clipped, "a\uFFFD\uFFFDb\u{1F44D}c");
61+
});

0 commit comments

Comments
 (0)