[myluma]

Adds a bytes field type, then replaces the JSON document and hit encodings
with MessagePack after benchmarking showed the boundary — not the engine — was
where search time was going.

The feature

A bytes field is the byte-array analogue of string: indexed, the whole value
is one term matched byte-for-byte. It's for a binary id or key — a UUID, a hash,
a packed struct — where a String forces you to hex- or base64-encode it
yourself.

let schema = SchemaBuilder()
    .addBytesField("key", stored: true, indexed: true)
    .addTextField("body")
    .build()

try index.add(["key": key, "body": "hello"] as [String: Any])
try index.search(.term("key", key))                    // exact lookup
try writer.deleteDocuments(field: "key", equals: key)  // delete / upsert
hit.data("key")                                        // -> Data?

Values carry any byte pattern — NULs, invalid UTF-8, all 256 byte values. tantivy
has supported this all along (add_bytes_field, Term::from_field_bytes); it was
never wired through the FFI.

Why this turned into an encoding change

Byte values cannot ride the JSON boundary the wrapper used: a JSON string must be
valid UTF-8, and the C strings carrying it are NUL-terminated. The first
implementation sent them as raw pointers alongside the JSON — a blob on results, a
pointer array on documents and queries, and an unconstrained lifetime whose safety
the compiler could not check.

Benchmarking the alternative made that side-channel unnecessary, and turned out to
matter considerably more than the feature that prompted it.

	before	after
search, 1000 hits — decode	19.59 ms	0.92 ms
search, 1000 hits — end to end	21.61 ms	1.65 ms
writes, prose — encode	1.64 µs/doc	0.70 µs/doc
writes, prose — full ingest	4.33 µs/doc	1.78 µs/doc

Neither win is really about JSON being verbose — MessagePack is only 1.05× smaller
on prose. Two measurements drove it:

• On reads, 83% of decode time was Codable, not JSON parsing. FieldValue
  discovered a value's type by attempting each decode in turn, so every string
  threw four DecodingErrors before succeeding. Decode was 87–92% of total
  search time; it is now 46–56%.
• On writes, JSONSerialization cost 3–6× what serde_json spent parsing the same
  content back, and the payload was copied twice more on its way to a
  NUL-terminated C string.

What changed

Documents and hits are MessagePack. Documents are a map of {field: [values]}
decoded against the schema, so each value is read as the field's declared type
rather than guessed from its encoding. Hits are read straight into SearchHits
with no intermediate representation and no Codable. Byte values need no
side-channel — MessagePack has a native byte-string type.

Query trees stay JSON. They're a few hundred bytes sent once per search, and
serde_json::Value's random access suits a recursive tree with optional fields far
better than a streaming reader would. Byte values there are
{"$bytes":"<base64>"} — tagged rather than bare, so a Data aimed at a string
field is reported instead of quietly searching for its base64 text.

JSONSerialization is gone from the library. Query trees and the schema spec
are written from their typed Swift representations with no intermediate
[String: Any]. That deleted a latent trap in SchemaBuilder:

let data = (try? JSONSerialization.data(withJSONObject: spec)) ?? Data("{\"fields\":[]}".utf8)

build() can't throw, so a serialization failure silently produced a schema with
no fields — surfacing as "unknown field" at every later addDocument,
arbitrarily far from the cause. Nothing left can fail. The duplicate string escaper
in IndexWriter is gone too, and both payloads are now byte-stable across runs.

Reviewer notes

Two behaviour changes:

• A whole-valued f64 keeps its type. 4.0 decoded as .int(4) through JSON
  because decimal text can't distinguish it; MessagePack tags the value.
• Mixed-type arrays are rejected by the schema-driven decode in Rust rather than
  a Swift-side check, so the error arrives one layer later.

addDocument(_ value: some Encodable) still goes through JSON, since that's what
JSONEncoder produces; Data properties are base64-encoded and decoded back by the
engine. Correct, but it costs an encode and a decode — Document and the
[String: Any] overload avoid both.

One measurement is unresolved and labelled as such in the benchmark output:
feeding pre-encoded input saves more than the encoding accounts for, reproducibly.
Likeliest cause is merge scheduling, since each timed iteration adds to the same
index. It may mean more is available on writes than the 2.3× measured.

Testing

345 Swift tests, 12 Rust tests. Coverage 85.0% → 89.3% of regions.

The ones worth calling out are the MessagePack boundary cases. The format widens its
length prefixes at 2^5, 2^8 and 2^16, so str32/bin32/array32/map32 and the
64-bit integer forms would otherwise never execute — and a mistake in one is silent
corruption, not a crash. Tested at the codec level on both sides with the tag byte
asserted, because map32 needs a 65,536-field document (tantivy's hard ceiling) and
array32 needs 65,536 values in one field.

Benchmarks ship in the test target, skipped unless enabled:

TANTIVY_BENCH=1 swift test -c release --filter PathBenchmark

⚠️ Breaking: the C ABI changed

A new xcframework release is required before consumers can build.

• tantivy_result_json / _blob / _blob_len → tantivy_result_bytes / _len
• CBytes and CBytesField removed
• tantivy_writer_add_json_bytes → tantivy_writer_add_msgpack
• byte-argument parameters dropped from four functions
• searches return MessagePack

rust/Cargo.toml gains base64 (already in the tree via tantivy) and the
formatting feature on time.