Repository navigation
Seventh review: DuckDB 2.0 fixes, faster nullable and list paths, 86 live-engine tests - #147
Merged
Merged
Conversation
…uards, faster validity and list paths, 86 live-engine tests Findings and measurements are in AUDIT.md section 13; the user-visible changes are in CHANGELOG.md [Unreleased]. Added - TypeId::TimestampTzNs (DuckDB 2.0's type 42) with VectorReader/Writer and Value support, without calling the two slots 2.0.0 appended, so a 546-slot build still loads into 1.5.6 and 2.0.0. - ListVector/MapVector::get_entry_checked; DescriptionYml::opt_in_platforms. - benches/engine (SDK functions beside hand-written twins, timed inside a real DuckDB) and scripts/check-api-coverage.py with its allowlist, run by the abi-table CI job. - 98 more live-engine tests in tests/ffi_roundtrip (394 in the binary, 296 before), 86 of them for public functions no test executed. Fixed - A VARIANT in a signature crashed DuckDB 2.0.0; LogicalType::try_new now refuses it. - A zero-parameter aggregate counted no rows on 2.0; registering one is now an error that names the workaround. Behaviour change for 1.x users. - get_entry on a NULL row returned stale offsets; the book's read loops and two test helpers walked them. - append_metadata rewrote its output in place, crashing a host that had the old build mapped; it now writes a sibling file and renames it. - Scaffold: no extension_config.cmake (nothing reads it), license in the generated Cargo.toml, musl rustflags; validator: platform group names and opt_in_platforms the build ignores now warn. Performance (benches/engine/compare.py, DuckDB 1.5.6 and 2.0.0, one thread) - is_valid and set_null test and clear the validity bit directly, as_str skips full UTF-8 validation for ASCII, ListBuilder keeps its child writer between rows. Where the runs separate, the median ratio is 1.1x to 7.1x (both engines); non-NULL columns are unchanged. Tables and spreads: AUDIT.md 13.7. CI, scripts and docs - bench-compile and abi-table jobs extended; dependabot config and the MSRV-vs-DuckDB-CI and CI-aggregate scripts updated (AUDIT.md 13.2); README, book, CONTRIBUTING and the issue and PR templates brought in line with the code. Verified on this tree: fmt, clippy (default, duckdb-1-5-4 and three live-test configurations), 992 / 1,114 tests (default / duckdb-1-5-4), live-engine suites on DuckDB 1.5.6 (1,586), 2.0.0 (1,586) and 1.4.4 (1,318), Miri on vector:: (95), MSRV 1.86, cargo package --locked. Not run: ASAN/LSAN, cargo-deny, macOS/Windows/Wasm builds, any workflow in GitHub Actions (AUDIT.md 13.8). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
LESSONS.md and the book's pitfalls page gain the three engine behaviours the previous commit guards against: a VARIANT in a signature crashes DuckDB (P13), an aggregate with no parameter counts no rows on DuckDB 2.0 (P14), and the entry of a NULL LIST or MAP row holds stale bytes (P15). The count of known pitfalls (19 L entries plus 15 P entries) is now 34 in the README, FAQ, introduction, contributing guides and FFI reference. CHANGELOG.md gets a line and the book's copy is regenerated. The PR template asks for exactly this when a new FFI pitfall is found; the previous commit left it undone. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
scalar_bind_data_copy::get_argument_hands_out_each_argument_as_an_owned_expression registers a bind callback that calls the raw duckdb_scalar_function_bind_get_argument, but was gated with inspects_arguments_or_refuses, which asserts that the *guarded* ScalarBindInfo::argument refuses with "before v1.5.5". The raw call has no guard, so on v1.5.0 and v1.5.3 (the CI jobs that build with duckdb-1-5 and duckdb-1-5-3) the probe query succeeded and the assertion failed. The test now skips engines before v1.5.5 through a new engine_copies_arguments_safely helper, which inspects_arguments_or_refuses also uses, so its other callers behave as before. The defect was in the test only. The local gate had no engine between 1.4.4 and 1.5.6 with a duckdb-1-5 feature. Reproduced on v1.5.0 and v1.5.3 before the change (1 failed each) and green after: v1.5.0 1,491, v1.5.3 1,496, v1.5.4 1,586, v1.5.5 1,586, v1.4.5 1,318 passed; v1.5.6 ffi_roundtrip 394. AUDIT.md 13.8 records this as the fifth test defect. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
…lippy lint CI's first run on this branch found two more defects. LogicalType::try_new(TypeId::TimestampTzNs) on an engine older than 2.0.0 sent type id 42 to duckdb_create_logical_type. A release engine answers INVALID and the call returned the documented error, but a DuckDB built with assertions (the bundled-test source build) reaches D_ASSERT(0) in LogicalTypeIdFromC and aborts the process, which hides every test after it. precheck now refuses the type before sending the id when abi::engine_version() is a release older than 2.0.0; a development version string, which parse_version refuses to read, is let through. Every Value path that needs the type goes through try_new, so Value::timestamp_tz_ns and as_timestamp_tz_ns get the same answer. The live test of the pre-2.0 behaviour now asserts the message. Rust 1.99.0's clippy (CI's stable; the local toolchain was 1.97.0) flags assert!(x.is_empty()) as assert_is_empty in two new append_metadata tests; both are assert_eq! against an empty Vec now. Verified: DuckDB 1.5.6 built from source with assertions, CI's two commands of test-bundled (--features bundled-test, then --all-features): 1,318 and 1,586 passed, 0 failed (the first aborted after 1,053 tests before the change); clippy 1.99.0 clean in every configuration CI lints; default and duckdb-1-5-4 tests 992 and 1,114, doctests 211; live DuckDB 2.0.0 1,586. AUDIT.md 13.8 records the three CI findings of this branch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
…invariants test-bundled-duckdb-2-0 (DuckDB 2.0 built from the libduckdb-sys 1.20000.1 source, with assertions) failed five tests on its first run, a job this branch added and had not run. timestamp_tz_ns::a_scalar_function_over_the_type_round_trips_nanoseconds_and_nulls casts TIMESTAMP_NS to TIMESTAMPTZ_NS, a cast the ICU extension supplies. The source build has no ICU until the first reference to the TimeZone setting makes it download and load it (duckdb_extensions() lists core_functions alone before and core_functions plus icu after; the cast fails before and works after). The test now sets the time zone, as its siblings do. The other four assert an exact number of binds or copies and saw 5 for 2, 3 reports for 1 and 2 copies for 0: lifecycle::identical_calls_share_bind_data_unless_the_function_is_volatile and table_cast::fold_errors::a_failed_fold_reports_the_message_and_type_not_json (both older than this branch) and two in scalar_bind_data_copy. A DuckDB built with assertions verifies the plan (Planner::Optimize runs LogicalOperator::Verify under #ifdef DEBUG, which copies every planned expression); the release 2.0.0 library gives the exact counts. The assertions now state what the tests are about: at least one bind per call, every report carries the same type and message, every bind sees the three arguments in order, and the plain-projection copy count is not asserted. The end-of-run drops == binds + copies stays exact. Verified: the job's command on the source build, 1,062 library and 394 ffi_roundtrip tests passed (5 failed before); release libraries 1.5.6 and 2.0.0 1,586 each, 1.5.0 1,491; clippy 1.99.0 clean on the live-test configurations. AUDIT.md 13.8 records the finding and what was and was not isolated. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
copy_info_errors::each_phase_reads_the_state_the_earlier_phases_left reports the file name of the COPY destination from the global-init callback. It split the path at '/', which leaves a Windows path whole, so on the windows-latest bundled-test job the note held C:\Users\...\out.state and the assertion on "file=out.state" failed. With the type-id abort gone, this was the one failure left on that job (391 passed, 1 failed in the --all-features step). Path::file_name splits on the platform's separator. Checked on Linux: the two copy_info_errors tests pass against libduckdb 1.5.6 and clippy 1.99.0 is clean on the live-test configuration. The Windows result is CI's; no Windows machine was available. AUDIT.md 13.8 records the finding. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
…s_timestamp_tz_ns CI's incremental mutation job (cargo mutants --lib) left 7 mutants alive, all in code this branch added and all in logic that only a live-engine test reached, which that job does not run: the `||`, two `==` and the unary `-` that decide whether Value::timestamp_tz_ns appends "+00", and the `<` of the 2.0.0 comparison in LogicalType::precheck's new TIMESTAMPTZ_NS guard. Both decisions are small pure functions now, utc_text and predates_timestamp_tz_ns, with unit tests on the boundaries: finite instants and their neighbours either side of each infinity; releases just below and above 2.0.0 on each of major, minor and patch, and version strings parse_version cannot read. Behaviour is unchanged. Verified: cargo-mutants 27.1.0 (the CI pin) with CI's flags on the two functions, 15 mutants: 14 caught, 1 unviable, 0 missed. Clippy 1.99.0 clean, fmt clean, rustdoc clean, unit tests 994 default and 1,116 with duckdb-1-5-4, live DuckDB 1.5.6 1,588. AUDIT.md 13.8 records the finding and the new totals. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
On the windows-latest bundled-test job, writer_api::timestamp_tz_reads_and_writes_the_instant died with "Extension Autoloading Error ... 'icu': Could not move file: Access is denied". A source-built DuckDB has no ICU and installs it the first time a query needs a time zone. The tests run on many threads, each with its own database, and every install writes to one extension directory, so on Windows two installs collide on the final file move. The suite already had the race (earlier Windows runs passed it by luck); this branch made it likelier by adding tests that need a time zone. Fixture::open now runs INSTALL icu once per process (std::sync::Once) before any other fixture goes on, and ignores a failure, so the others find a finished install and only load it. On a library that carries ICU the statement is a no-op. Not reproduced: the race is Windows-specific. Checked on Linux that nothing else changes: release libraries 1.5.6 and 2.0.0 (1,588 each), 1.4.4 (1,320), 1.5.0 (1,493); the 1.5.6 source build from an empty home directory installs ICU during the run and passes both of CI's commands (1,320 and 1,588); clippy 1.99.0 clean on the live-test configurations. Whether it removes the Windows failure is CI's answer. AUDIT.md 13.8 records the finding. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EAuNUCqy8XuoN3zKvJT2NX
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Output of an adversarial review of 0.19.0 (
9c2dc76), with the fixes made in the same change. The full write-up isAUDIT.mdsection 13; the user-visible changes are under[Unreleased]inCHANGELOG.md.VARIANTin a function signature segfaulted DuckDB 2.0.0 (LogicalType::try_newnow refuses it). An aggregate with no parameter counted no rows on 2.0 (SELECT f() FROM tover 100 rows: 100 on 1.5.6, 0 on 2.0.0), so registering one is now an error that names the workaround.get_entryon a NULL list/map row returned stale offsets;ListVector::get_entry_checkedandMapVector::get_entry_checkedare new.append_metadatarewrote its output in place and crashed a host that had the old build mapped; it now writes a sibling file and renames it.TIMESTAMPTZ_NS(DuckDB 2.0's type 42) is now aTypeIdwith vector andValuesupport. It deliberately avoids the two C functions 2.0.0 appended, so a 546-slot build still loads into 1.5.6 and 2.0.0. Creating the type before 2.0 is refused before the id is sent, because a DuckDB built with assertions aborts the process on a type id it does not know.is_valid/set_nulltest the validity bit directly,as_strskips full UTF-8 validation for ASCII,ListBuilderkeeps its child writer between rows. Measured with the newbenches/engine/compare.pyinside a real DuckDB (1.5.6 and 2.0.0, one thread, release build): where the runs separate, the median ratio is 1.1x to 7.1x; non-NULL columns are unchanged. Tables and spreads are inAUDIT.md13.7.tests/ffi_roundtrip(394 in the binary, 296 before), 86 of them for public functions no test executed, and 2 unit tests. Newscripts/check-api-coverage.pyclassifies every C API slot as used or deliberately unused, and runs in theabi-tablejob.extension_config.cmakeand sets the license and musl flags; thedescription.ymlvalidator warns about platform groups and ignoredopt_in_platforms;bench-compile,abi-tableand a new DuckDB 2.0 source-build job (informational) inci.yml; pitfalls P13 to P15 added toLESSONS.mdand the book.Behaviour change for reviewers to weigh: code that registered a zero-argument aggregate for DuckDB 1.x now fails to register.
Type of change
Checklist
Code quality
cargo test --all-targetspasses (994 default, 1,116 withduckdb-1-5-4), andcargo test --doc --features duckdb-1-5-4(211 passed)test-bundledcommands, 1,320 and 1,588) and DuckDB 2.0 built from source (--libandffi_roundtrip, 1,062 and 394). In CI all of these jobs pass on the final head, along with the older-engine matrix (1.4.4 to 1.5.5)cargo clippy --all-targets -- -D warningspasses: default,duckdb-1-5-4and the live-test configurations, on both the local 1.97 and CI's 1.99.0cargo fmtapplied (no diff)cargo doc --no-depsbuilds without warningsunsafeblocks have a// SAFETY:comment (undocumented_unsafe_blocksis on and clippy is clean under-D warnings)src/value/temporal_checks.rs411 to 502, and four new live-test files of 553 to 774 lines, one per area of the public API they exercise)Testing
cargo mutantsshows zero surviving mutants: CI's incremental mutation workflow passes on the final head (run 180) and on1e50fe3(run 179). Its first run left 7 alive, all in newsrc/logic that only a live-engine test reached; fixed in1e50fe3by making the two decisions pure functions with boundary unit tests (cargo-mutants27.1.0 on them: 15 mutants, 14 caught, 1 unviable, 0 missed)Documentation
CHANGELOG.mdupdated under[Unreleased], andscripts/sync-book-changelog.pyrun (--checkpasses)book/src/) updatedLESSONS.mdandbook/src/reference/pitfalls.md(P13 to P15; the count of known pitfalls is now 34)Safety (for changes touching
unsafe)ffi_roundtrip) pass in CI, which is evidence, not an audit#[deny(unsafe_op_in_unsafe_fn)]remains satisfied (compiles)CI history on this branch
The first push failed jobs in both the push and pull-request runs, from causes I had not run locally. Each was reproduced where possible, fixed, and recorded in
AUDIT.md13.8:6c53d11: a new test calling the rawduckdb_scalar_function_bind_get_argumentwas gated on the guarded wrapper's refusal, so it failed on DuckDB 1.5.0 and 1.5.3 (test defect).6b06ad2: Rust 1.99.0's clippy (CI'sstable; I had 1.97.0) flaggedassert!(x.is_empty())in two new tests (test defect). AndLogicalType::try_new(TypeId::TimestampTzNs)sent type id 42 to engines older than 2.0, which a DuckDB built with assertions aborts on (defect insrc/, fixed by refusing before the id is sent).c7ed006: the newtest-bundled-duckdb-2-0job, which I added without running it, failed five tests: one neededSET TimeZone(ICU is not in a source build), four asserted exact bind/copy counts that only hold on a release build (two of those tests predate this PR; relaxed to the invariants they check).b79835d: a COPY test split a path at/, which fails on Windows (test defect).1e50fe3: the mutation job left 7 mutants alive (see above).248e3c6: on Windows, tests that need a time zone raced to install ICU into one directory (Could not move file: Access is denied). A race the suite already had and this branch made likelier.Fixture::opennow installs ICU once per process.On the final head
248e3c6, CI's push run passes all 56 jobs, the pull-request run, Mutation Testing and Coverage all pass. The Windows race could not be reproduced locally; one green run is weak evidence for a race that also passed once before the fix, so treat item 6 as fixed by construction, not by observation.Not run
ListBuilder(the pointer paths are covered by Miri, ASAN, LSAN and the mutation proofs, not by valgrind).benchmarks.yml(runs only on pushes tomain) and thedependabot.ymlchange, which a PR does not exercise. The benchmark numbers inAUDIT.md13.7 are local, frombenches/engine/compare.pyon a shared 4-vCPU VM.Decisions left to the owner
Listed in
AUDIT.md13.3: thelibduckdb-sys <2cap, trusted publishing, ABI policy for 2.0.x, duplicate push and PR CI runs, a yank or RUSTSEC advisory for 0.15 and earlier, SPDX headers, mutation-gate exclusions, the size ofAUDIT.mdandCHANGELOG.md, API changes to make before 1.0,AppendErrorhaving a feature-dependent type, anddefault-features.🤖 Generated with Claude Code