🌟 scandb content checksums + upload wire surface (ADR: unchanged-scan short-circuit) - #3225
Draft
preslavgerchev wants to merge 13 commits into
Draft
🌟 scandb content checksums + upload wire surface (ADR: unchanged-scan short-circuit)#3225preslavgerchev wants to merge 13 commits into
preslavgerchev wants to merge 13 commits into
Conversation
Per-row digest columns on the four digested tables plus per-kind and asset digests in metadata, computed at scan end over final table state; schema stays 1.0 (additive, announced via digest_algo_version). See the unchanged-scan-short-circuit draft ADR in the server repo. Includes the digestdiff analysis tool. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire surface for content-addressed scan uploads (server ADR #17467): ReportUploadCompletedReq gains bool unchanged (no PUT happened — every kind digest matched the last processed upload), and GetDownloadURL joins PolicyResolver as GetUploadURL's mirror: a signed GET for a server-held artifact, kind-addressed (initially DOWNLOAD_KIND_SCAN_MANIFEST — the trimmed-down scan database of keys + digests). GetUploadURL itself is untouched; no digest travels in any RPC — the digests live in the scan database file. LocalServices answers Unimplemented, same as it does for upload URLs. Dormant until the server side ships. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
Contributor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
…abad94) The digest package now resolves in CI without a local replace. Repin to the mql main lineage once mondoohq/mql merges the llx/digest PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
…R kind fold Repin mql to preslav/llx-scan-digest @ 9d41587 (stdlib hash/fnv instead of the unmaintained cespare/xxhash, AlgoVersion 1). Adopt the final ADR fold rules: the algo version is a fold input of every kind digest, and the asset-level fold is gone - the four kind digests are the only comparison currency, matching what the server recomputes and compares (policy/scancontent). ComputeDigests now returns the four kind digests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
The server announces its scan-content mode as a plain feature name (ScanContentModeShadow / ForceSend / ServerCompare / ClientCompare) in ScanParameters.enabled_features, consumed by the existing withServerFeatures plumbing - no key=value parsing anywhere, and the 'unknown server feature' warning disappears because the names are real mql features now. ComputeDigests runs only when Features.ScanContentDigestsActive() (any mode) - zero digest work for scopes that haven't opted in, closing the C0 gating gap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
topologicalSortQueries iterated the queriesMap - Go map order is random, so the relative order of independent queries flapped between runs, and compile-order-derived artifacts (implicit-property For lists) flapped with it: TestProps_ImplicitPropsOnSharedQueries failed whenever check-2 compiled before check-1. Iterate the input slice instead; the visited set and DFS keep dependency order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
The manifest (row-level diff) is the only comparison currency per the updated ADR: ComputeDigests keeps writing per-row digest columns and now announces only digest_algo_version in metadata - the per-kind metadata digests and the kind fold are gone, and it returns per-table row counts for logging instead of fold values. client_compare later pulls the server-staged manifest and diffs rows, the same comparator the server runs at every ingest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
…atest The client compares content to content against whatever manifest the server currently serves; a delta that raced a concurrent upload fails the server's row-digest verify and falls back to a full upload - one failure class, self-healing, no base pinning anywhere. If the delta extension ever ships and wants a base pin after all, adding a proto3 field back is non-breaking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
…huffle) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST
…ksum Checksum is the platform's word. Renames across the wire surface and code: - scandb `digest` columns -> `checksum`; metadata key digest_algo_version -> checksum_algo_version (schema 1.2, unshipped) - ComputeDigests -> ComputeChecksums (returns ChecksumCounts), OpenForDigest -> OpenForChecksums - the scan gate follows mql: Features.ScanContentChecksumsActive() Hashing now comes exclusively from mql's llx/checksum (data/resource rows, hash writer, canonicalization - explicit, no reflection) and the new policy/scandb/checksum package (score/risk rows, which cannot live in mql without importing cnspec's protos). No local hash code remains. Repin mql to llx/checksum (16904007c). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The writer half of the wire contract had no tests. Pins: every checksum column lands bit-equal to the canonical packages' output for the same row (the values the server's extract-vs-recompute parity gate compares), the metadata key announces the algorithm, the pass is deterministic, it folds final upserted state, and a read-only store refuses it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manifest is the comparison artifact both sides share: the server builds and stages it today, the client pulls and reads it in client_compare. Declaring its schema and metadata keys here makes the file format the client's contract; the server consumes this declaration instead of owning a copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
preslavgerchev
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The cnspec half of the unchanged-scan short-circuit ADR (mondoohq/server#18038, issue server#17467), in one PR:
Scan-content checksums (C0 emission, spiked and measured)
checksum INTEGERcolumns ondata/scores/scored_risk_factors/resourcesand thechecksum_algo_versionmetadata key. There is no kind- or asset-level fold anywhere — row equality (the manifest diff: server-side at every ingest, the client's manifest pull inclient_comparelater) is the only comparison currency.ComputeChecksums(viaOpenForChecksums) runs one pass afterFinalize(measured: 15.6 ms on a real 2.1 MB / 312-row scan database; ~75% SQLite I/O, the multiset array canonicalization is ~1%), writing the checksum columns and announcing the algo version; it returns per-table row counts. Canonicalization lives in mql'sllx/checksumfor llx-typed rows (structural, never proto bytes; multiset arrays;AlgoVersion = "1") andpolicy/scandb/checksumfor score/risk rows, gated on the server-activatedScanContentMode*features viaFeatures.ScanContentChecksumsActive()— zero checksum work for scopes that haven't opted in.digestdiffdev tool for diffing/benching two scan databases (can be dropped in review if unwanted).Wire surface (dormant until C1)
ReportUploadCompletedReq.unchanged— "no PUT happened: the scan is row-identical to the staged manifest"; statistics still ridedetailsin both outcomes.GetDownloadURLRPC —GetUploadURL's mirror: a signed GET for a server-held artifact, kind-addressed (DOWNLOAD_KIND_SCAN_MANIFEST: the staged manifest — the trimmed-down scan database of keys + checksums the server already diffs at every ingest; the client diffs the same object). Response:download_url+unavailable— deliberately no base-identity field: the comparison policy is always-latest, and a delta that raced a concurrent upload fails the server's row-checksum verify and falls back to a full upload.GetUploadURLitself is untouched — no checksum travels in any RPC; checksums live in the file.LocalServices.GetDownloadURLanswersUnimplemented, mirroring its upload-URL stance; the platform server implements it as a deliberate decline (unavailable, no URL) untilclient_compareships (mondoohq/server#18380).Nothing calls the new surface yet — this lands the proto so the C1 compare-and-skip client can follow independently.
Testing
go generate ./policyregenerated (pb/ranger/vtproto);go build ./...green. Checksum determinism was proven against real scans: score/risk rows bit-identical across five consecutive local scans on the released os provider (after cnspec#3179 + mql#9652 + mql#9713 fixed the ordering/parse bugs the checksums themselves surfaced). The server side additionally pins cross-repo parity in its test suite: this writer's columns vs the server's independent recompute over the same scandb, bit-identical.🤖 Generated with Claude Code
https://claude.ai/code/session_018HHX819u6q9PngxJfpCaST