Feature proposals for quereus-crypto-plugin: a canonical multi-field digest helper, a real CID UDF and a MerkleRoot primitive

[risavian]

Version reviewed: 0.13.4
Source: dist/plugin.js (register -> SQL digest/sign/verify/hash_mod/random_bytes), dist/index.js (JS exports)
Type: Feature request / enhancement (no bug claimed against the plugin)
Filed: 2026-06-05

Scope

The SQL digest(data, algorithm?, inputEncoding?, outputEncoding?) UDF is a
single-value hash and behaves correctly: it hashes one blob. How an
application combines multiple logical fields into that blob and whether it
does so injectively is the caller's responsibility, exactly as with any
sha256(blob).

VoteTorrent currently rolls its own multi-field digest downstream
and gets the field-framing wrong; that is being fixed on the VoteTorrent side
(companion report votetorrent-digest-canonicalization.md) and is not asked of
this plugin. "Concatenate then hash" is inherently non-injective; that's a
property of concatenation, not a defect in a hash library.

However this situation can be improved with canonical helper functions.

Selective disclosure related feature could also get a help from the crypto plugin.

The 3 additions below would help the downstreams with similar needs.

---

1. A canonical multi-field / structured digest helper

Several consumers will independently need to hash an ordered tuple of fields,
and each one reinventing the framing is where injectivity bugs creep in
(delimiter joins, null-vs-empty conflation, unencoded arity). The plugin could
optionally offer a vetted canonical encoding so downstreams don't each roll
their own:

• A SQL UDF that accepts multiple data arguments and hashes an injective
  encoding of them e.g. per argument tag(null|value) ‖ varint(len) ‖ bytes
  so distinct tuples never collide and null is distinguishable from empty.
• If the proposed MerkleRoot primitive lands, let's have it too reuse the same framing
  for leaf/node hashing so tree and scalar commitments are consistent.

This is explicitly an enhancement, not a bug fix as the existing single-value
digest is fine and would remain. It is also fine for the plugin to decline
this and keep field-framing entirely caller-side; in that case a short doc note
("digest hashes one blob; callers are responsible for canonical, injective
framing of multi-field inputs") would prevent the footgun.

2. A real CID implementation in UDF, ala IPFS' multihash + CIDv1

Today digest() returns a bare hash (base64url of the raw digest bytes)
with no multibase prefix, multicodec content type, multihash
(hashFnCode ‖ length ‖ digest) framing. Consumers that name their columns
Cid (VoteTorrent has ~29 such columns) are therefore not storing IPFS/IPLD
CIDs: the value would not match a CID any content-addressed store computes for
the same bytes, and it carries no algorithm agility (nothing in the value
identifies sha2-256, so the hash function can't be migrated unambiguously).

Request

Add a SQL UDF (and matching JS export) that produces a self-describing
content identifier:

CIDv1 = multibase( version ‖ multicodec(content-type) ‖ multihash )
        multihash = hashFnCode ‖ digestLength ‖ digestBytes

Desirable shape:

• cid(data, …) / cid_v1(...) wrapping a digest in a multihash + CIDv1, with
  selectable multicodec (e.g. raw, dag-cbor), multihash code
  (sha2-256 / sha2-512 / blake3), and
  multibase (default base32). Input is a single blob, same contract as
  digest framing of multiple fields stays the caller's job.
• Round-trip helpers (cid_decode -> version / codec / hash code / digest) so
  schemas can validate and migrate.

This lets downstreams (per votetorrent/doc/schema-conventions.md, Option B)
turn Cid columns into real, interoperable, upgrade-safe content addresses
instead of bare digests, ideally decided before more signed records exist.

---

3. A MerkleRoot primitive for per-attribute selective disclosure

Why

VoteTorrent voter registration needs per-attribute selective
disclosure: an authority commits to a registrant's selective-disclosure field
set as a single value (Registrant.SelectiveCid, covered by the authority's
signature), then later reveals a subset of those fields to a permitted audience
with an inclusion proof without leaking the values of the undisclosed
fields. A flat Digest(whole set) can't do this (verifying any field needs the
whole pre-image -> all-or-nothing). The Merkle tree
over per-attribute salted leaves: reveals (name, value, salt) + an audit
path per disclosed field; undisclosed fields appear only as opaque sibling
hashes.

Wanting this in the plugin (not purely app-side) is deliberate:

1. So the schema can enforce RegistrantSelective.CidValid: Cid = MerkleRoot(…)
   at insert that a forged/incorrect root becomes impossible to store (the same
   "invalid states impossible by design" posture as the rest of the schema).
2. So the DB, the engine (proof generation), and the recipient (proof
   verification) all share one canonical implementation and can't drift on
   leaf encoding / tree shape.

Requested API shape (your call)

• Generic, minimal: merkle_root(hexLeafHashes_json): plugin owns only the
  tree (the two domain-separated hashes + split rule). App owns leaf
  construction (salt / flatten / encode). Maximally reusable, but the DB
  CidValid CHECK would need leaf hashes materialized, not raw details so it
  can't fully enforce the root from the stored record.
• Selective-specific: selective_merkle_root(selectiveDetails_json): plugin
  owns the entire canonicalization above. Then
  CidValid check (Cid = selective_merkle_root(SelectiveDetails)) is a one-liner
  and the root is fully DB-enforced. Heavier and less generic, but it's the
  version that actually closes the invalid-state hole (the motivation in (1)).
• Companion: merkle_verify_inclusion(root, leafEntry, proof_json): so the
  one canonical impl also backs recipient-side verification (no drift).

Recommendation: the selective-specific shape (or the generic merkle_root
plus a documented canonical leaf-encoding UDF), since DB-side enforcement of the
root from the stored record is the whole point. Either way, the leaf/node framing
should match request 2 so scalar and tree commitments stay consistent.

Related

• Companion votetorrent report (the actual canonicalization bug + fix):
  votetorrent-digest-canonicalization.md
• votetorrent/doc/schema-conventions.md: "Cid columns are digests, not CIDs"
  • the canonicalization sub-section.

[n8allan]

Feedback from my agent. Look good?

#3 — MerkleRoot for selective disclosure: legit and the most valuable, but the spec is the work

This is standard selective-disclosure (same shape as SD-JWT / Certificate Transparency). The need is real and DB-side
enforcement (CHECK Cid = selective_merkle_root(...)) is a genuinely strong "invalid states impossible" posture. But the
hashing is trivial; the canonicalization spec is the entire engineering surface, and it's permanent once records are
signed. Concerns to settle before implementing:

1. Domain separation is mandatory. Leaf vs internal-node hashes must be domain-separated (RFC 6962 style: 0x00‖leaf,
   0x01‖left‖right) or you get second-preimage attacks where an internal node is replayed as a leaf. The reporter knows this
   ("two domain-separated hashes") — we just have to pin the exact constants.
2. Tree shape must be fully canonical. Leaf ordering (sort by attribute name vs insertion order?), odd-node handling
   (promote vs duplicate), unbalanced trees — every rule is a hard contract shared by DB, proof generator, and verifier.
   This is where drift happens, not in the hash.
3. Proof generation needs a home too. The reporter only asks for merkle_verify_inclusion in the plugin. But "one
   canonical impl, no drift" breaks if the engine reimplements audit-path construction. We should at least export a JS
   proof-generation function from the plugin so all three sides share path logic, even if the SQL surface is just root +
   verify.
4. Ship canonical test vectors. Cross-impl drift is the stated risk, so the deliverable isn't just code — it's a frozen
   set of (fields, salts) → root, proof vectors VoteTorrent's engine and recipient can pin against.
5. My recommendation on shape: go with the reporter's lean — selective_merkle_root(detailsJson) (plugin owns full leaf
   canonicalization, so the CHECK is a real one-liner), but implement it as a thin wrapper over a generic
   merkle_root(leafHashesJson). That gets both the enforceable one-liner and a reusable primitive, and the generic version's
   leaf hashing reuses the Build fret test infrastructure #1 JSON recipe.

[risavian]

My agent:

Looks great — this is exactly the right framing: the hash is trivial, the canonicalization is the permanent contract. Agree on all five, especially layering selective_merkle_root over a generic merkle_root (gives us the enforceable one-liner and a reusable primitive), and on giving proof generation a canonical JS home, not just verification.
▎
▎ Three things to fold into the spec before it's frozen:
▎
▎ 1. Salts are part of the security spec, not just the test vectors. Our fields are low-entropy (birthdate, party, precinct), so without a per-leaf salt a holder of root + proof can brute-force undisclosed leaves. The salt scheme (per-attribute nonce, length, derivation/storage) should sit in the canonical leaf definition right next to domain separation.
▎ 2. Lock the responsibility split: selective_merkle_root owns leaf canonicalization (attribute framing + salt + leaf domain tag); generic merkle_root(leafHashesJson) owns only tree construction (node domain tag + odd-node rule). That keeps the CHECK a true one-liner without the generic primitive accidentally becoming "the whole algorithm."
▎ 3. The per-leaf encoding is proposal #1 (tag(null|value)‖varint(len)‖bytes) — suggest resolving #1 as the leaf format of #3 rather than tracking it separately. Separately, any read on #2 (real multihash/multibase CIDs)? Columns named Cid currently store bare digests, so that one stands on its own.
▎

[n8allan]

digest() changes:

For the VoteTorrent maintainer: migrate to @optimystic/quereus-plugin-crypto digest

Why

The schema's Digest(...) is computed by a local UDF in packages/vote-engine/src/database/initialize.ts that does
args.map(String).join('|') then sha256. That framing is not injective, and these digests are signed and gate authorization, so the
collisions are a security bug, not cosmetic:

• Delimiter collision: Digest('a|b','c') == Digest('a','b|c') == Digest('a','b','c'). Any free-text/JSON field (Name, Title,
  Instructions, ImageRef, Relays, Tags, …) can contain |, so a tuple can be re-framed and a signature reused for a different mutation.
• NULL/empty conflation: Digest('x', NULL) == Digest('x', '') (nullable hashed columns are everywhere).
• Type confusion: integer 123 == text '123'.

Plugin digest ≥ 0.14.0 replaces this with a canonical, injective, length-prefixed, type-tagged framing and is registered replicable
(bit-identical across peers — required for signed commitments).

The change (small)

1. Upgrade the dependency to @optimystic/quereus-plugin-crypto@^0.14.0.
2. Delete the local Digest UDF in initialize.ts (the digestSchema/registerFunction(digestSchema) block, ~lines 49–62). Don't replace
   it — the plugin provides digest.
3. Pass config when registering the plugin. At initialize.ts:102:
   await registerPlugin(db, cryptoPlugin, { algorithm: 'sha256', encoding: 'base64url' });
4. (matches the existing Base64url-encoded sha256 column semantics).
5. No schema edits needed. Quereus resolves function names case-insensitively (keys are name.toLowerCase()/numArgs), so every
   Digest(...) call in votetorrent.qsql binds to the plugin's digest automatically. The nested sub-digest pattern (Digest(EffectiveAt,
   ThresholdPolicies) fed into a parent Digest(...)) keeps working and is now injective at both levels; the delete sentinel Digest(Tid,
   old.Id, 'delete') is fine.

⚠️ The one thing that needs a real decision: existing digests change

The new framing produces different hash values than the old |-join for the same inputs. So every previously computed/stored/signed
digest (AdminSigning.Digest, InviteResult.Digest, the AdminSigningDigest index, any signatures over them) will no longer match.

• Pre-production / disposable data: nothing to do — recompute on a fresh DB.
• Production with persisted signatures: you need a migration plan (the old digests can't be recomputed to the new form without
  re-signing). Don't ship the swap against live signed data without one.

Behavioral notes to verify (mostly free wins)

• Types must arrive as their real SQL type. digest type-tags values, so make sure context.Tid reaches SQL as an integer, not a text
  string (text '1' and int 1 now produce different digests — intended). Canonical datetimes (ISO strings) and JSON columns arrive as

Behavioral notes to verify (mostly free wins)

• Types must arrive as their real SQL type. digest type-tags values, so make sure context.Tid reaches SQL as an integer, not a text string (text '1' and int 1 now produce different digests — intended). Canonical datetimes (ISO strings) and JSON columns arrive as TEXT
  and are hashed opaquely — correct.
• NULL is now distinct from '' — desired, but it's part of why all digests change.
• Integer-valued REAL collides with INTEGER (2.0 ⇒ 2); irrelevant unless you hash a REAL column.
• JSON fields must be valid JSON. If any field is a native JS object/array (not a JSON string column), it's canonicalized with sorted keys, and undefined/NaN/bigint/non-plain objects throw rather than silently collapsing. Your JSON columns are TEXT, so this won't
  bite unless you start passing native objects.
• Update test fixtures that assert specific digest/signature values — recompute the expected values.

[n8allan]

CID results:

What was built

A self-describing CIDv1 layer on top of the existing bare-hash digest, delivered as
both JS exports and SQL UDFs. All framing/parsing is delegated to the audited
multiformats library (newly added dep, ^14.0.0); the hashing reuses the plugin's
existing synchronous @noble/hashes functions via resolveHasher.

CIDv1     = multibase( version ‖ multicodec(content-type) ‖ multihash )
multihash = hashFnCode ‖ digestLength ‖ digestBytes

JS API (src/cid.ts, re-exported from index.ts):

cid(data: Uint8Array, codec?='raw', hash?='sha2-256', base?='base32'): string   // hash then frame
cidV1(digest: Uint8Array, hash, codec?='raw', base?='base32'): string           // frame a given digest, no re-hash
cidDecode(cid: string): { version, codec, hashCode, digest: Uint8Array }        // parse / validate

SQL surface (src/plugin.ts, all replicable: true + DETERMINISTIC_FLAGS, same bar as digest):

cid(data BLOB|b64url-TEXT, codec? TEXT, hash? TEXT, base? TEXT) -> TEXT
cid_v1(digest BLOB|b64url-TEXT, hash TEXT, codec? TEXT, base? TEXT) -> TEXT
cid_decode(cid TEXT) -> JSON TEXT  { version, codec, hashCode, digest(base64url) }

Selectable: codec ∈ {raw 0x55, dag-cbor 0x71}; hash ∈ {sha2-256 0x12,
sha2-512 0x13, blake3 0x1e}; base ∈ {base32 (default), base58btc,
base64url, base16}.

Test coverage (test/cid.spec.ts — 19 tests; full suite 80 passing)

[n8allan]

Where we landed for the Merkle Tree request:

Thanks for this — the motivation is spot-on and well-argued. You've correctly diagnosed that a flat digest(whole set) is all-or-nothing, that per-attribute salted leaves are the right shape, and that the DB / engine / recipient must share one canonical implementation so they can't drift. We're going to build it. One adjustment: we're making it smaller than a Merkle tree.

What we're building: a flat salted-leaf set commitment

We're not building a Merkle tree. A tree's only real advantage is O(log n) proof size, and selective-disclosure field sets are small (a handful to a few dozen fields), so the log saving is marginal — while the tree drags in footguns we'd have to hand-roll and pin forever: arity, odd-node handling (the CVE-2012-2459 duplicate-leaf forgery class), mandatory leaf-vs-internal domain separation, and a separate audit-path format.

Instead, each attribute becomes a salted leaf, and the commitment is a digest over the leaf digests:

leafDigest = digest([SD_LEAF_DOMAIN, name, value, salt])
root = digest([SD_SET_DOMAIN, ...leafDigests sorted by bytes])

This is, notably, the same construction the IETF SD-JWT selective-disclosure standard settled on — flat salted hashes of (salt, name, value), not a tree — which is good external evidence the smaller shape is the right one. (We aren't wire-compatible with SD-JWT; we reuse Optimystic's own canonical digest/encodeFields framing, which is what gives us bit-identical results across peers.)

This dissolves your generic-vs-selective dichotomy

You framed a real tension: generic merkle_root(hexLeafHashes) can't fully enforce the root from the stored record, while selective_merkle_root(details) can but isn't reusable. That dichotomy disappears here, because the leaf is built by the existing digest primitive (already canonical, injective, type-tagged, domain-separated, and replicable). So the new primitive stays generic — a salted-set commitment usable anywhere — and is fully DB-enforceable, because every crypto-bearing layer lives in the plugin. It's the same layering the in-flight CID work uses (cid(digest(...))).

Concretely, the shape is:

-- SelectiveDetails stores the [name, value, salt] triples; the CHECK recomputes
-- the root from the stored record, so a forged SelectiveCid can't be inserted.
CHECK (SelectiveCid = cid(set_commit(SelectiveDetails)))

setCommit(leaves) // -> root (the signed/persisted value)
setDisclose(leaves, revealNames) // -> { disclosed triples, opaque hidden digests }
setVerify(root, disclosure) // -> boolean (recipient-side)
SQL surface: set_commit(leaves_json) and set_verify(root, disclosed_json, hidden_json), both replicable/deterministic like digest.

A few specifics worth calling out

• Name is hashed into the leaf, so a disclosed (value, salt) proof can't be replayed against a different attribute slot. Free, given the injective framing.
• Verify is a JS export first, SQL second. Your companion merkle_verify_inclusion was framed as a SQL UDF, but the "permitted audience" is almost never running the DB — so the canonical verifier is a plain exported function the recipient imports (the SQL UDF just
  wraps it). That shared export is the anti-drift mechanism.
• Sort is by leaf-digest bytes, and this is forced — not a preference. The verifier learns only the disclosed names; hidden leaves arrive as opaque digests with no name, so the only ordering key it holds for every leaf is the digest itself. Verification reconstructs
  the entire root, which means the holder can't add, drop, or swap a leaf.
• Duplicate names and empty salts are rejected at commit (invalid-states-impossible posture). Salts come from random_bytes (≥128 bits).
• CID coordination: SelectiveCid is one of the ~29 Cid columns the crypto-cid-v1-content-identifier work covers, which warns that signed records freeze their representation. So the column should hold cid(set_commit(...)) — the self-describing CIDv1 — rather than freezing a bare hash. We've made that ticket a (soft) prereq so the persisted shape is decided consistently.

Net: we keep your motivation and your "one canonical impl, can't drift" goal, drop the tree as the default, compose on digest to get genericity and DB-enforcement at once, and make the recipient-side verifier the primary surface. Tracked and specced for implementation.

[n8allan]

All three landed. One was deliberately built smaller than proposed — reasoning below.

Thanks again for the write-up; the framing that the hash is trivial and the canonical encoding is the permanent contract is what shaped all three.

1. Canonical multi-field digest — built as proposed.
digest() now takes any number of fields directly. Each value is tagged with its type and length before hashing, so two different sets of fields can never produce the same input to the hash. Numbers, text, blobs, booleans, nulls and structured values each have their own tag, and structured values are written out in a fixed key order so the result is byte-identical on every machine.

2. CID — built as proposed.
There are now three functions: cid() hashes content and returns a proper self-describing content address; cid_v1() wraps a digest you already have, requiring you to state which algorithm produced it; and cid_decode() opens one back up into its parts. The encoding and parsing are delegated to the standard multiformats library rather than hand-rolled.

3. Selective disclosure — built, but as a flat set commitment rather than a Merkle tree.

This is the one place we departed from the proposal, and it's worth being explicit about why, since the earlier discussion in this thread had settled on a tree.

The functions are set_commit(), which turns a set of attributes into one signable commitment value, and set_verify(), which checks a partial reveal against that commitment. The behaviour you asked for is all there: an authority commits to a whole set once, later reveals only some attributes, and the recipient can confirm the revealed ones are genuine without learning anything about the withheld ones. It's enforceable directly in a table constraint, which was the main point.

What changed is the internal shape. Each attribute is hashed on its own with its own random salt, and the commitment is a hash over all of those. No tree.

The reason is that a tree's only real advantage is that proofs grow slowly as the set grows, and the sets here are a handful to a few dozen fields — so that advantage is worth almost nothing. Against it, a tree brings several things we'd have to design and get right ourselves and then never change: how to handle an odd number of items at each level (there is a well-known forgery class here — CVE-2012-2459 — that comes from getting exactly this wrong), keeping leaf hashes distinguishable from internal ones, and a separate proof format to specify and version. The flat version has none of that surface. It's also the shape the SD-JWT standard settled on for the same reason, though we're not wire-compatible with it.

To correct something said in this thread earlier: the reusable generic primitive did get built. It just isn't a merkle_root — it's a general salted-set commitment that sits on the same canonical encoding as digest(), and the selective-disclosure behaviour is that primitive used directly rather than a layer on top of a separate tree function.

Two details worth knowing if you build against this:

• The attributes are sorted by their hash values, not by name, and that is forced rather than a stylistic choice. A verifier is only told the names of the attributes that were revealed; the withheld ones arrive as bare hashes with no name attached. Sorting by name would leave the verifier unable to reconstruct the commitment. Please don't "tidy" it into a name sort.
• Salts are mandatory and per-attribute, generated by random_bytes at 128 bits or more. Low-entropy values like dates of birth, booleans or postcodes are trivially guessable from an unsalted hash, and independent salts also stop anyone matching two people by spotting identical hashes.