feat: use bstr::BString for shell values to fix UTF-8 lossy conversion - #1116
Open
lu-zero wants to merge 6 commits into
Open
feat: use bstr::BString for shell values to fix UTF-8 lossy conversion#1116lu-zero wants to merge 6 commits into
lu-zero wants to merge 6 commits into
Conversation
Public API changes for crate: brush-coreAdded itemsChanged itemsPerformance Benchmark Report
Code Coverage Report: Only Changed Files listed
Minimum allowed coverage is Test Summary: bash-completion test suite
|
Elsie19
suggested changes
May 6, 2026
| let mut s; | ||
| if self.interpret_backslash_escapes { | ||
| s = String::new(); | ||
| s = Vec::new(); |
Contributor
There was a problem hiding this comment.
Suggested change
| s = Vec::new(); | |
| s = Vec::with_capacity(self.args.len()); |
If you do this you can avoid a couple reallocations.
0.19.0 adds unconditional bytes support via BytesMode and the RegexInput trait, which supersedes the fork dependency that was previously required for byte-level matching. Also fix the one breaking change: Captures is now generic over the input type, so name it as Captures<'_, str> in the replace_all closure. Assisted-by: opencode:glm-5.2
Convert ShellValue, ShellValueLiteral, and ArrayLiteral from String to bstr::BString so arbitrary bytes are preserved throughout the shell variable system. This fixes silent data corruption in: - PWD/OLDPWD/HISTFILE for directories with non-UTF-8 paths (Unix) - mapfile reading non-UTF-8 input - From<OsString> for ShellValue (uses into_vec() on Unix) - pathcache entries (uses path_to_bstring on Unix) Added crate-level helpers os_string_to_bstring() and path_to_bstring() that preserve raw bytes on Unix via OsStrExt, with lossy fallback on other platforms. env::get_str() and the expansion pipeline retain String/str at their boundaries (lossy for interpretation/display only); storage is now fully lossless. Closes: reubeno#1115 Co-Authored-by: opencode <opencode@anomaly.co> Assisted-by: opencode:glm-5.2
Enable BytesMode::UnicodeBytes in regex compilation so compiled regexes can match against both &str and &[u8] input. Update glob pathname expansion to match against raw filename bytes via OsStrExt::as_bytes() on Unix, eliminating the to_string_lossy() conversion that previously corrupted non-UTF-8 filenames. For valid UTF-8 input the behavior is unchanged (UnicodeBytes mode preserves Unicode character class semantics while allowing non-UTF-8 byte input). Assisted-by: opencode:glm-5.2
Convert ExpansionPiece (Unsplittable/Splittable) and WordField from
String to bstr::BString so that non-UTF-8 bytes flow through the
expansion pipeline without lossy conversion. This completes the
lossless data path from variable storage through ${var} expansion.
Key changes:
- ExpansionPiece variants now hold BString instead of String
- Added inherent is_empty(), len(), chars() methods on ExpansionPiece
using bstr's ByteSlice trait
- Added bstring_to_string() helper for lossless-then-lossy conversion
at the String boundary (From<ExpansionPiece> for String/PatternPiece)
- polymorphic_subslice uses collect::<String>().into() for char slicing
- ANSI-C quoting ($'...') now preserves raw expanded bytes via
BString::from(expanded) instead of String::from_utf8_lossy
- element_keys()/element_values() flow directly as BString into
ExpansionPiece without .to_string()
Assisted-by: opencode:glm-5.2
Convert PatternPiece (Pattern/Literal) and RegexPiece (Pattern/Literal) from String to bstr::BString so pattern and regex pieces preserve non-UTF-8 bytes losslessly from the expansion pipeline. Key changes: - PatternPiece and RegexPiece variants now hold BString - From<ExpansionPiece> for PatternPiece/RegexPiece is now lossless (BString passed through directly, no bstring_to_string conversion) - escape_literal_regex_piece now accepts &[u8] and escapes non-UTF-8 bytes as \xHH regex hex escapes (e.g. byte 0x80 becomes \x80 in the regex string). This works correctly with BytesMode::UnicodeBytes. - PatternPiece::as_str() uses to_str() with unwrap_or fallback - to_regex_str() in patterns.rs lossy-converts BString pieces to &str for the glob-to-regex converter (known limitation: the glob converter requires valid UTF-8 input) This closes the last lossy boundary in the regex matching path. The only remaining lossy conversion is PatternPiece literal bytes flowing through the glob converter, and env::get_str() for arithmetic (which is moot since non-numeric bytes yield 0 regardless). Assisted-by: opencode:glm-5.2
lu-zero
force-pushed
the
utf8-vs-bytes
branch
2 times, most recently
from
July 29, 2026 14:44
4a426bf to
d2c8229
Compare
The glob-to-regex converter (brush_parser::pattern::pattern_to_regex_str) requires valid UTF-8 input, which was the last lossy boundary for PatternPiece literal bytes. Fix it by encoding non-UTF-8 bytes as Private Use Area characters (U+E000 + byte value) before the converter — these are valid Unicode but not glob metacharacters, so they pass through unchanged. After conversion, replace_pua_with_hex_escapes() converts them to \xHH regex hex escapes that match the original raw bytes under BytesMode::UnicodeBytes. This closes the last lossy boundary in pattern matching. The only remaining lossy conversion is env::get_str() for arithmetic, which is moot (non-numeric bytes yield 0 in both bash and brush). Assisted-by: opencode:glm-5.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ShellValue,ShellValueLiteral, andArrayLiteralfromStringtobstr::BStringso arbitrary bytes are preserved throughout the shell variable systemecho -e,mapfile, ANSI-C quoting ($'...'),${var@E}, PWD/OLDPWD, andFrom<OsString>forShellValueCloses #1115
Details
Bash treats all data (filenames, variable values, command output) as raw byte sequences. Rust's
Stringrequires valid UTF-8, so everyfrom_utf8_lossyorto_string_lossycall was a potential silent data corruption point.The core change replaces
Stringwithbstr::BString(backed byVec<u8>) inShellValue::String,AssociativeArray, andIndexedArray, plus all related types. Key fixes:echo -e '\x80': now writes raw bytes to stdout instead of lossy-convertingmapfile: stores raw bytes asBString$'\x80'): preserves non-UTF-8 bytes${var@E}: same as aboveOsStrExt::as_bytes()on Unix to preserve raw path bytesFrom<OsString>: usesinto_vec()on Unix instead ofto_string_lossy()24 files changed, all tests passing (49 unit + 224 parser + 25 builtins + 1696 compat).
Co-Authored-by: opencode opencode@anomaly.co
Assisted-by: glm-5.1