Skip to content

testing.md

Latest commit

 

History

History
192 lines (159 loc) · 9.45 KB

testing.md

File metadata and controls

192 lines (159 loc) · 9.45 KB
Raw

Testing

Choose tests by the behavior you changed, not just the nearest filename. Setup prepares the default suite; selected integrations own additional prerequisites. Verification layers explain what each kind of evidence proves.

Choosing tests

Change Starting checks
Ingest / expression IR tests/ingest.rs; snippet roundhouse-ast --round-trip
Analysis / diagnostics tests/analyze.rs, tests/real_blog.rs
Lowering Owning regression suite; inspect the lowered shape with dump_ir
Ruby emission tests/lowered_ruby_emit.rs; emitted execution for semantics
Framework Ruby tests/runtime_ruby_unit.rs, tests/runtime_src_integration.rs, selected framework_tests_<target>
New supported construct (removed error) tests/emit_and_run.rs: zero errors and correct emitted Ruby execution
A target's emitted project <target>_toolchain with --ignored; comparison against Rails when behavior changes
CI implementation The owning CI contract tests
Documentation references tests/docs_references.rs

A diagnostic-count test does not prove runtime support. Fixture lanes cannot exercise constructs absent from their inputs; add a targeted emitted regression. Framework suites may cover only a named subset (Rust CI uses the inflector subset); a suite's green result is not a claim about everything in that target.

Fixtures

  • fixtures/tiny-blog/ and fixtures/tiny-api/ are checked in. Use them for small shape/absent-feature gates; extend them deliberately.
  • fixtures/real-blog is generated by bin/rh fixture using scripts/create-blog.
  • fixtures/store is generated by (cd fixtures && ../scripts/create-store store); its generator runs the Rails guide tests too.
  • External corpora have their own pins and setup. For example, Writebook is an inventory, not runtime conformance.

The generated trees are ignored by Git; a fresh clone has neither. Accessors in src/fixtures.rs name the missing tree and command. bin/rh verify lists missing fixtures but never installs or generates them. Missing Ruby, gems, or SDKs can cause a harness to skip or fail; read the executed test output rather than assuming exit zero means coverage.

Complete local verification

  1. Install cargo-nextest once:

    cargo install cargo-nextest --locked

  2. Run the complete default suite:

    bin/rh verify --all

This command runs library and binary tests plus every integration suite with Nextest. It then runs cargo test --locked --doc, because Nextest does not run documentation tests. Ignored tests remain ignored. It does not use a result cache, and it bypasses Nextest's default filter to prevent omitted tests.

The full run compiles integration sources into one executable. build.rs discovers every tests/*.rs suite; no second list needs maintenance. Only full verification enables this target. Normal Cargo commands and bin/rh verify --test NAME keep the individual targets.

Rust tests that start their own executable use the name! helper for --exact selection. It preserves the test name in both layouts. tests/test_subprocess.rs checks that the child executes its test body, not just that the child exits successfully.

--all cannot be combined with --test, --toolchain, --ignored, --lib, or --contract. NEXTEST_TEST_THREADS controls test processes; --jobs controls Cargo workers. Full verification defaults incremental compilation on and preserves an explicit CARGO_INCREMENTAL value. --all --plan remains read-only.

An ARM64 macOS run produced these measurements with the optimized test profile:

Measurement Separate integration targets Combined integration target
Test executables 867 9
Nextest-reported test execution 382.188 s 262.323 s

The complete combined command took 265.837 s with a warm build, including discovery and documentation checks. The first combined build in this work took 188.84 s. That build cost is not included in the warm-run time. Switching between full and focused modes can rebuild the library and test target. The final run passed 5,985 tests. All 6,334 existing test identities and their ignored status matched the separate targets; one new regression test checks child selection. The same 350 tests remained ignored.

The macOS verification used TMPDIR=/private/tmp. The default /var/... temporary path caused existing assertions against canonical /private/var/... paths to fail.

Focused verification

Without --all, bin/rh verify runs the integration suites you select, sequentially and fail-fast. The library suite runs only with --lib, after those suites. --contract runs ingests_without_errors and type_analysis_coverage in tests/real_blog.rs, then every_runtime_method_body_is_fully_typed in tests/runtime_src_integration.rs, and does not build the other integration binaries. A --contract --exact filter that matches no test fails that check. Neither mode is a hosted CI emulator or merge approval. A green focused run does not replace the commit bar below.

bin/rh verify --plan --base main --test ingest
bin/rh verify --test ingest --test real_blog
bin/rh verify --contract
bin/rh verify --lib --test ingest
bin/rh verify --test framework_tests_ruby --ignored
bin/rh verify --toolchain ruby --json > /tmp/verification.json
  • Repeat --test or --toolchain to select more suites. Default --test excludes ignored tests; an ignored-only suite otherwise executes nothing. --ignored runs only ignored tests in selected --test suites, not the library suite or the contract filters. --toolchain already selects ignored tests in <target>_toolchain.
  • --plan is read-only: no Cargo, installs, tests, or verification lock. With no --test, --toolchain, --contract, or --lib, it is only a hosted-coverage preview and reports no local checks. A run without --plan, --all, or one of those selectors exits with an error.
  • --base REF compares the current working tree with that exact commit (default HEAD), including non-ignored untracked files. It neither fetches nor finds a merge base. The hosted selection preview is informational, not GitHub's actual PR merge-tree/draft/label context.
  • Execution uses locked Cargo commands. Every suite this loop launches keeps --test-threads=1, because suites share SQLite or the process-global ingest::survey. Cargo workers default to CARGO_BUILD_JOBS or the machine's processor count. --jobs overrides workers. Explicit Cargo environment settings are preserved. Verify does not set CARGO_INCREMENTAL; Cargo's own dev/test default applies, and a caller-supplied value is passed through. Hosted unit shards do not use this script.
  • The worktree lock coordinates verifier callers only. Do not run unrelated Cargo builds/tests concurrently in that checkout.
  • --json separates executed, failed, and not-run checks; child output goes to stderr. It includes filesystem-space snapshots, with advisory low-space warnings. It is not a reusable receipt or content fingerprint.

For all options, use bin/rh verify --help. The implementation and verifier contract tests are bin/rh and tests/rh_verify_test.rb.

Direct Cargo checks

cargo test --locked --lib
cargo test --locked --test ingest
cargo test --locked --test framework_tests_ruby -- --ignored --nocapture
cargo test --locked --test rust_toolchain -- --ignored --nocapture
cargo test --locked                 # default suite before commit
cargo test --locked --all-targets   # milestones

CI repeats the CRuby gates (framework_tests_ruby, uuid_key_ruby, tiny_api cruby_gate_) on the next MRI line, env.MRI_RUBY_NEXT, in compare-ruby-next. To reproduce that locally, put that Ruby first on PATH and install the scaffold bundle for it too (bundle resolves ruby from PATH), then run the same commands.

Focused checks do not replace the default-suite commit bar. Native toolchain and framework integrations are generally ignored so local default tests do not require every SDK. They are not all separate GitHub jobs; see CI.

The test profile uses opt-level = 1 to reduce compiler pipeline costs across the integration suites. It keeps debug assertions, overflow checks, and file/line backtraces. No test selection or ignored-test rule changes. The first build costs more; later runs reuse the optimized artifacts.

On an ARM64 macOS checkout, three runs of each built test binary gave these median times. These measurements exclude compilation:

Suite Unoptimized Test profile
ingest 0.394 s 0.142 s
analyze 4.015 s 1.330 s
real_blog 0.872 s 0.499 s
runtime_src_integration 0.702 s 0.394 s
emit_and_run 127.186 s 88.074 s

For debugger locals, use CARGO_PROFILE_TEST_DEBUG=2. To step through unoptimized test code, also use CARGO_PROFILE_TEST_OPT_LEVEL=0. Suites that assert file/line backtraces need symbols. Neither the verifier nor a passing subset cleans old build artifacts. If space is exhausted, stop builds and inspect the dedicated target directory before removing disposable outputs. Never strip live programs or clean a cache shared with another checkout.