Repository navigation

T1.20 docs: the locked gate run recorded #457

Workflow file for this run

	# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
	# All rights reserved.
	# SPDX-License-Identifier: Apache-2.0

	# Every gate a change has to pass, run where a contributor can see it.
	#
	# One gate is not here: the contract check reads the canonical OpenAPI spec
	# from where it is maintained, which a fork cannot reach. The generated
	# bindings are pinned by SHA-256 in PROVENANCE.json either way, and a drifted
	# copy fails the build it ships in.
	name: CI

	on:
	pull_request:
	push:
	# The Rust core migration's epic branch and its phase branches run the
	# same gates as main; the reference and archive branches deliberately
	# do not, including a lane package's archive branch,
	# rust-core-phase-1-archive-*, which keeps an abandoned tip (E43, CI2).
	branches:
	- main
	- rust-core
	- 'rust-core-phase-*'
	- '!rust-core-phase-1-archive-*'

	permissions:
	contents: read

	# A newer push cancels a topic branch's run in progress. On the phase
	# branch and the lane branches a run in progress is not cancelled (E43): a
	# landing's runs there are recorded after the fast-forward, so the next
	# landing or the lane's next package must not cancel them. A run still
	# queued there can be: GitHub keeps one pending run per group, and a newer
	# push supersedes it. No landing depends on that run alone; land.py verify
	# reports a cancelled run, and passes on another green run of the same tree.
	concurrency:
	group: ci-${{ github.ref }}
	cancel-in-progress: ${{ !(github.ref_name == 'rust-core-phase-1' \|\| startsWith(github.ref_name, 'rust-core-phase-1-lane-')) }}

	env:
	# Whether this run saves its caches: the Rust builds and pip's downloads.
	# GitHub restores a push's caches only from its own branch and the default
	# branch, so a cache a topic branch saves serves that branch alone, and
	# those filled the repository's 10 GiB. Only the long-lived branches save,
	# and the lane branches: each parallel lane of packages pushes to one lane
	# branch it reuses, so a package's first run restores the builds the
	# lane's previous package saved (decision E43). Every run restores what
	# its own branch saved; a run that saves nothing still runs every check.
	SAVE_CACHES: ${{ github.event_name == 'push' && (contains(fromJSON('["main", "rust-core", "rust-core-phase-1"]'), github.ref_name) \|\| startsWith(github.ref_name, 'rust-core-phase-1-lane-')) }}
	# The manylinux2014 image both wheel builds run in, pinned by digest.
	MANYLINUX: quay.io/pypa/manylinux2014_x86_64:2026.09.14-1@sha256:21c37461985655aaa25ed3a923b28e6c9d4dd9e10edc0281eb50385184bddd31
	# The binding's tests run at the E16 floor; DuckDB, at the reference's
	# version, is for the DuckDB provider's tests there (T1.11b).
	BINDING_FLOOR: pyarrow==14.0.1 pandas==2.2.0 numpy==1.26.4 duckdb==1.5.6

	jobs:
	lint:
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- run: pip install --upgrade pip 'ruff==0.16.*'
	- run: ruff check .
	- run: ruff format --check .

	hygiene:
	runs-on: ubuntu-latest
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- run: python scripts/check_repo_hygiene.py all

	# The Rust core (docs/design/rust-core/phase-1/design.md, section CI). Each
	# job installs the toolchain rust/rust-toolchain.toml pins, and caches the
	# build keyed on Cargo.lock. Third-party actions are pinned by commit, as
	# the organisation's allow list requires.
	rust-lint:
	runs-on: ubuntu-latest
	defaults:
	run:
	working-directory: rust
	steps:
	- uses: actions/checkout@v7
	with:
	fetch-depth: 0
	- run: rustup toolchain install
	- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
	with:
	workspaces: rust
	save-if: ${{ env.SAVE_CACHES == 'true' }}
	- run: cargo fmt --all --check
	# sdm-testkit builds the bundled DuckDB, so its clippy runs in rust-test,
	# which builds it anyway (H3); the L0 self-tests' clippy fixture needs
	# only jiff, and stays here.
	- run: cargo clippy --workspace --exclude sdm-testkit --all-targets --locked -- -D warnings
	# The binding with its test-only operations, as its tests build it.
	- run: cargo clippy -p sdm-py --features test-ops --locked -- -D warnings
	# The L0 guards (T1.10c): the crate graph (l0_deps.py), with the
	# bridge freeze (bridge_freeze.py, E50), whose baseline ratchets
	# against the merge base with origin/rust-core-phase-1, which the full
	# checkout above fetches; and the vocabulary, SQL-fragment, skip and
	# ignore, clippy-ban and step-variant scans (l0_scan.py). The bans
	# themselves are enforced by the clippy runs above, from
	# rust/clippy.toml.
	- name: L0 graph, bans and scans
	run: \|
	python3 scripts/rust_core/l0_deps.py
	python3 scripts/rust_core/l0_scan.py
	working-directory: .
	# Each guard passes on the tree and fails on a fixture seeded with the
	# fault it exists for, and clippy refuses each banned item in a fixture
	# crate under rust/clippy.toml.
	- name: L0 self-tests
	run: python3 scripts/rust_core/selftest.py
	working-directory: .
	# land.py's checks, each passing a sound landing and refusing a seeded
	# fault, against a scripted git and gh; lane.py's lane rules and
	# land.py's on a lane, against real temporary git repositories and a
	# fake gh; and prune_caches.py's stale-cache rules; with no network
	# (decision E43, CI2).
	- name: land.py, lane.py and prune_caches.py self-tests
	run: \|
	python3 scripts/rust_core/land_selftest.py
	python3 scripts/rust_core/lane_selftest.py
	python3 scripts/rust_core/prune_caches.py --selftest
	working-directory: .
	# The contract schema: regenerated from the types and compared with the
	# committed file, locked to the current version; no entry merged into
	# rust-core rewritten, and at most one working entry, which is why this
	# job checks out every branch (for the merge base with rust-core).
	- run: cargo test -p sdm-contract --locked --test schema_lock
	- run: python3 scripts/rust_core/contract_lock.py
	working-directory: .
	# The client wheel's notices for the Rust crates it links, regenerated
	# from Cargo.lock and compared with the committed file.
	- run: python3 scripts/rust_core/rust_notices.py --check
	working-directory: .

	rust-deny:
	runs-on: ubuntu-latest
	defaults:
	run:
	working-directory: rust
	steps:
	- uses: actions/checkout@v7
	- run: rustup toolchain install
	- uses: taiki-e/install-action@9114bf4d891761788c546334fd37538eae1bf8b3 # v2.87.16
	with:
	tool: cargo-deny
	- run: cargo deny --locked check

	rust-test:
	runs-on: ubuntu-latest
	defaults:
	run:
	working-directory: rust
	steps:
	# Every branch, for the gate's checkpoint (the merge base with the
	# phase branch) and its worktree.
	- uses: actions/checkout@v7
	with:
	fetch-depth: 0
	- run: rustup toolchain install
	- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
	with:
	workspaces: rust
	save-if: ${{ env.SAVE_CACHES == 'true' }}
	# The test host's ICU, DuckDB's signed extension for v1.5.6, which it
	# installs the first time and checks against its recorded digest before
	# every load (decision E37): cached, so a run downloads it once.
	# Saved, like every cache here, only where SAVE_CACHES says.
	- id: duckdb-extensions
	uses: actions/cache/restore@v6
	with:
	path: ~/.duckdb/extensions/v1.5.6
	key: duckdb-extensions-v1.5.6-${{ runner.os }}-${{ runner.arch }}
	# The test kit's clippy, here on the DuckDB build this job makes (H3).
	- run: cargo clippy -p sdm-testkit --all-targets --locked -- -D warnings
	- run: cargo test --workspace --locked
	# The pure-Rust host over the contract cases, its observations compared
	# with the cases' expectations and reference observations (T1.10;
	# T1.18 moves this to the checkpoint's comparator), from T1.11e under
	# contract/compare-policy.yaml, which pyyaml reads: its statements and
	# their results against the reference's (H9), its call against the
	# reference's request, and base.regression reaching its result. The
	# test host links no Python. Its DuckDB loads ICU explicitly, installing
	# DuckDB's signed extension for its version the first time, from the
	# cache above when it holds it (decision E37).
	- run: cargo run -p sdm-testkit --bin sdm-contract-run --locked -- --out ../rust-host.jsonl ../contract/cases
	- run: \|
	cargo tree -e normal -p sdm-testkit --locked > ../testkit-tree.txt
	! grep -w pyo3 ../testkit-tree.txt
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	# On the phase branch and its lanes the gate step below runs this
	# comparison with the checkpoint's comparator, each baselined case's
	# outcome against its effective expectation (an approved change
	# included), so this one runs only elsewhere (T1.14); whether the
	# gate runs is the gate step's to decide, failing closed.
	- run: \|
	pip install jsonschema pyyaml
	where=$(python scripts/rust_core/ratchet.py where) \|\| exit 1
	if [ -n "$where" ]; then
	echo 'the gate step compares rust-host on this branch'
	exit 0
	fi
	python scripts/rust_core/compare.py --cases contract/cases --reference contract/reference/*.jsonl rust-host.jsonl
	working-directory: .
	# The gate (T1.14): the candidate judged by its checkpoint's harness,
	# from the checkpoint's own worktree (design, "Reference observations
	# and the gate"; H2): rust-host's observations made with the
	# checkpoint's case loader and observation writer, the checkpoint's
	# ratchet (protected paths, reference immutability, cases, register,
	# relaxations, KNOWN_DIFFERING, baselines and availability), its
	# comparator and its L0 scripts. On a push to the phase branch the
	# checkpoint is the push's previous tip, and the push lands exactly
	# one package; on a lane or topic branch, the merge base with the
	# phase branch's tip, a landing commit judged against its own
	# recorded base. Before T1.14 lands, the bootstrap: the candidate's
	# harness runs.
	# Whether the gate runs fails closed (T1.18), decided in the step
	# itself so a local replay runs it too (replay_jobs.py takes no
	# conditions): on a push to the phase branch or any branch under it
	# the gate must run, from GitHub's own event and ref, not from the
	# candidate's ratchet.py, whose `where` then only names the base; an
	# error or an empty answer there fails. Elsewhere there is no ratchet.
	# tests/test_contract_ci_gate.py runs this step's text.
	- name: gate (rust-host)
	run: \|
	where=$(python scripts/rust_core/ratchet.py where) \|\| { echo 'gate: ratchet.py where failed'; exit 1; }
	case "${GITHUB_EVENT_NAME:-}:${GITHUB_REF_NAME:-}" in
	push:rust-core-phase-1 \| push:rust-core-phase-1-*)
	[ -n "$where" ] \|\| { echo "gate: ratchet.py where named no base on $GITHUB_REF_NAME, where the gate must run"; exit 1; } ;;
	esac
	[ -n "$where" ] \|\| { echo 'gate: no ratchet off the phase branch'; exit 0; }
	scripts/rust_core/gate.sh $where --run rust-host
	working-directory: .
	- if: ${{ !cancelled() && env.SAVE_CACHES == 'true' && steps.duckdb-extensions.outputs.cache-hit != 'true' }}
	uses: actions/cache/save@v6
	with:
	path: ~/.duckdb/extensions/v1.5.6
	key: ${{ steps.duckdb-extensions.outputs.cache-primary-key }}

	# The client wheel, which carries the core as sdm_client._native (design,
	# "Packaging"). maturin, at the version native_wheel.py pins, builds it in
	# a manylinux2014 container run with docker rather than through an
	# action: the x86_64 cp310-abi3 wheel that ships. native_wheel.py records
	# the build's target flags and fails on target-cpu, and reads the wheel
	# with auditwheel show. Every job that installs the client takes it from
	# this job's artifact, and this job builds and uploads nothing else, so
	# those jobs wait for the shipped wheel only (CI1): the test-only build
	# and the binding's tests run beside it, in native-test-ops and binding.
	native-wheel:
	runs-on: ubuntu-latest
	env:
	NATIVE_ARGS: --out native
	steps:
	- uses: actions/checkout@v7
	- run: rustup toolchain install
	working-directory: rust
	- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
	with:
	workspaces: rust
	save-if: ${{ env.SAVE_CACHES == 'true' }}
	# As the runner's user, so the cache can read rust/target afterwards,
	# in a home the runner made, so that user can write it, holding the
	# runner's crate registry, which the cache restores. NATIVE_ARGS says
	# which build: native-test-ops runs this same step for its own.
	- &manylinux-build
	name: Build the wheel in manylinux2014
	run: \|
	home="$RUNNER_TEMP/manylinux-home"
	mkdir -p "$HOME/.cargo/registry" "$HOME/.cargo/git" "$home/.cargo/registry" "$home/.cargo/git"
	docker run --rm --user "$(id -u):$(id -g)" \
	-e HOME=/tmp/home -e CARGO_HOME=/tmp/home/.cargo -e RUSTUP_HOME=/tmp/home/.rustup \
	-e NATIVE_ARGS \
	-v "$home:/tmp/home" -v "$PWD:/io" -w /io \
	-v "$HOME/.cargo/registry:/tmp/home/.cargo/registry" \
	-v "$HOME/.cargo/git:/tmp/home/.cargo/git" \
	"$MANYLINUX" bash -euo pipefail -c '
	# rustup-init 1.29.0, checked against its published SHA-256.
	curl --proto "=https" --tlsv1.2 -sSf -o /tmp/home/rustup-init \
	https://static.rust-lang.org/rustup/archive/1.29.0/x86_64-unknown-linux-gnu/rustup-init
	echo "4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10 /tmp/home/rustup-init" \
	\| sha256sum -c -
	chmod +x /tmp/home/rustup-init
	/tmp/home/rustup-init -y --profile minimal --default-toolchain none --no-modify-path
	export PATH="$CARGO_HOME/bin:$PATH"
	(cd rust && rustup toolchain install)
	/opt/python/cp312-cp312/bin/python -m venv /tmp/home/maturin
	/tmp/home/maturin/bin/pip install --quiet maturin==1.15.0
	export PATH="/tmp/home/maturin/bin:$PATH"
	python scripts/rust_core/native_wheel.py $NATIVE_ARGS --manylinux
	'
	- uses: actions/upload-artifact@v7
	with:
	name: native-wheel
	path: native/

	# The binding with its test-only operations, built beside native-wheel by
	# the same step and tested at the E16 floor: every file of the binding's
	# tests, and the Python host's tests that need those operations, which
	# the test job does not collect (E23). Its wheel is its own artifact,
	# which wheel-smoke installs in its clean containers and no job that
	# builds or ships takes (E23).
	native-test-ops:
	runs-on: ubuntu-latest
	env:
	NATIVE_ARGS: --out native-test-ops --test-ops
	steps:
	- uses: actions/checkout@v7
	- run: rustup toolchain install
	working-directory: rust
	- uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
	with:
	workspaces: rust
	save-if: ${{ env.SAVE_CACHES == 'true' }}
	- *manylinux-build
	- uses: actions/setup-python@v7
	with:
	python-version: '3.10'
	# The connectors are for the canary test's journey over DuckDB
	# (T1.11d), which tests_test_ops runs here at the floor.
	- name: The binding's tests on the test-ops build
	run: \|
	python -m venv /tmp/test-ops
	/tmp/test-ops/bin/pip install --quiet pytest $BINDING_FLOOR \
	./packages/structured-data-models-api ./packages/structured-data-models-pql \
	./packages/structured-data-models-connectors \
	"$(ls native-test-ops/*.whl)"
	/tmp/test-ops/bin/python -m pytest rust/bindings/sdm-py/tests -q \
	-p no:cacheprovider --native-build test-ops
	/tmp/test-ops/bin/python -m pytest packages/structured-data-models-client/tests_test_ops -q \
	-p no:cacheprovider
	- uses: actions/upload-artifact@v7
	with:
	name: native-wheel-test-ops
	path: native-test-ops/

	# The binding's tests on the wheel that ships, at the E16 floor: the file
	# any build passes.
	binding:
	runs-on: ubuntu-latest
	needs: native-wheel
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: '3.10'
	- uses: actions/download-artifact@v8
	with:
	name: native-wheel
	path: native
	- name: The binding's tests on the shipped build
	run: \|
	python -m venv /tmp/production
	/tmp/production/bin/pip install --quiet pytest $BINDING_FLOOR \
	./packages/structured-data-models-api ./packages/structured-data-models-pql \
	"$(ls native/*.whl)"
	/tmp/production/bin/python -m pytest rust/bindings/sdm-py/tests/test_binding.py -q \
	-p no:cacheprovider --native-build production

	# The wheels as a user installs them (T1.17, decision E16), each lane in
	# a clean container, both gating: the floor (Python 3.10, pyarrow 14.0.1,
	# pandas 2.2.0, NumPy 1.26.4) and the newest (Python 3.14 with the newest
	# pandas 3, NumPy 2 and pyarrow, resolved on every run). wheel_smoke.py
	# runs itself in the lane's Python image, pinned by digest, the checkout
	# mounted read-only; it installs the shipped wheel and the test-ops one,
	# as these jobs built them, each in an environment of its own holding
	# only wheels, and runs the binding's tests on them from outside the
	# checkout (shared and released buffers, admitted types, outputs after
	# collection, error and destruction, busy and cancel). Each lane's
	# resolved environments, reports and environment.json are its artifact,
	# uploaded whether it passed or not.
	wheel-smoke:
	name: wheel-smoke (${{ matrix.lane }})
	runs-on: ubuntu-latest
	needs: [native-wheel, native-test-ops]
	strategy:
	fail-fast: false
	matrix:
	lane: [floor, newest]
	steps:
	- uses: actions/checkout@v7
	- uses: actions/download-artifact@v8
	with:
	name: native-wheel
	path: native
	- uses: actions/download-artifact@v8
	with:
	name: native-wheel-test-ops
	path: native-test-ops
	- name: The lane's clean container
	run: \|
	python3 scripts/rust_core/wheel_smoke.py --lane ${{ matrix.lane }} --container \
	--wheel "$(ls native/.whl)" --test-ops-wheel "$(ls native-test-ops/.whl)" \
	--out wheel-smoke
	- if: ${{ !cancelled() }}
	uses: actions/upload-artifact@v7
	with:
	name: wheel-smoke-${{ matrix.lane }}
	path: wheel-smoke/

	typecheck:
	runs-on: ubuntu-latest
	needs: native-wheel
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- uses: actions/download-artifact@v8
	with:
	name: native-wheel
	path: native
	# The engine needs its runtime dependencies present to be checked at all,
	# and a toolchain to install. It ships py.typed exactly like the other
	# four, so leaving it out would make the gate narrower than the promise.
	# The image's compilers and CMake are that toolchain; the runner image
	# carries them, so nothing is installed (CI1).
	- run: for tool in gcc g++ make cmake; do command -v "$tool"; done
	- run: pip install --upgrade pip mypy
	# The client from the native wheel; its stub, _native.pyi, ships in it.
	- run: \|
	pip install -e './packages/structured-data-models-api' -e './packages/structured-data-models-pql' \
	"$(ls native/*.whl)" \
	-e './packages/structured-data-models-connectors[sqlite,duckdb,postgres,snowflake,databricks]' \
	-e './packages/structured-data-models-relational-engine'
	# Every session a source takes too, so each driver's types are its own
	# rather than Any. pyspark-client is pyspark's Python without its jars.
	- run: pip install snowflake-snowpark-python pyspark-client
	- run: \|
	mypy packages/structured-data-models-api/src packages/structured-data-models-pql/src packages/structured-data-models-client/src \
	packages/structured-data-models-connectors/src packages/structured-data-models-relational-engine/src
	# What a caller's type checker infers for a source, and what it refuses.
	- run: mypy packages/structured-data-models-connectors/tests/typing/check_sources.py

	# Every interpreter the four pure-python distributions classify. The
	# internal pipeline runs one of them, so a defect specific to any other
	# reached a release through a promise nothing checked.
	test:
	runs-on: ubuntu-latest
	needs: native-wheel
	strategy:
	fail-fast: false
	matrix:
	python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']
	# A real server for the PostgreSQL reads and the PostgreSQL sampler, which
	# only a PostgreSQL can say it accepts. SDM_REQUIRE_POSTGRES turns a
	# missing DSN or driver into a failure, where a skip would read as a pass.
	services:
	postgres:
	# Docker's official image, from its public ECR mirror: Docker Hub's
	# anonymous pull limit failed every attempt of a run. Pinned by the
	# digest CI pulled for the tag postgres:17.
	image: public.ecr.aws/docker/library/postgres@sha256:2d2b8998d31037bf721cfdf764d76ba74171b4fab3431b7f72c27c56ddbdf9e3
	env:
	POSTGRES_DB: sdm_test
	POSTGRES_PASSWORD: postgres
	ports:
	- 5432:5432
	options: >-
	--health-cmd pg_isready --health-interval 5s
	--health-timeout 5s --health-retries 10
	env:
	SDM_POSTGRES_TEST_DSN: postgresql://postgres:postgres@localhost:5432/sdm_test
	SDM_REQUIRE_POSTGRES: '1'
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: ${{ matrix.python-version }}
	- uses: actions/download-artifact@v8
	with:
	name: native-wheel
	path: native
	# pip's downloads and built wheels, keyed on the week and on every
	# distribution's manifest and restored from the newest of this job's
	# otherwise. pip still resolves each requirement against the index, so
	# the cache only saves downloads and changes no version (CI1).
	- id: week
	run: echo "week=$(date -u +%G-%V)" >> "$GITHUB_OUTPUT"
	- id: pip-cache
	uses: actions/cache/restore@v6
	with:
	path: ~/.cache/pip
	key: pip-${{ runner.os }}-${{ github.job }}-${{ matrix.python-version }}-${{ steps.week.outputs.week }}-${{ hashFiles('packages/*/pyproject.toml') }}
	restore-keys: pip-${{ runner.os }}-${{ github.job }}-${{ matrix.python-version }}-
	# databricks is here for its driver, not for a server: those tests mock
	# the connection. Without the driver, importing the backend raises and
	# every test covering it skips. The client is the native wheel's, the
	# one abi3 build every interpreter here installs.
	- run: \|
	pip install --upgrade pip 'ruff==0.16.*' "tomli; python_version < '3.11'"
	pip install -e './packages/structured-data-models-api[test]' -e './packages/structured-data-models-pql[test]' \
	"$(ls native/*.whl)[test]" \
	-e './packages/structured-data-models-connectors[sqlite,duckdb,postgres,databricks,test]'
	- run: \|
	python -m pytest packages/structured-data-models-api/tests packages/structured-data-models-pql/tests \
	packages/structured-data-models-client/tests packages/structured-data-models-connectors/tests tests -q \
	-m 'not live_nim and not live_warehouse'
	- if: ${{ !cancelled() && env.SAVE_CACHES == 'true' && steps.pip-cache.outputs.cache-hit != 'true' }}
	uses: actions/cache/save@v6
	with:
	path: ~/.cache/pip
	key: ${{ steps.pip-cache.outputs.cache-primary-key }}

	engine:
	runs-on: ubuntu-latest
	needs: native-wheel
	# A real server for the PostgreSQL reads and the PostgreSQL sampler, which
	# only a PostgreSQL can say it accepts. SDM_REQUIRE_POSTGRES turns a
	# missing DSN or driver into a failure, where a skip would read as a pass.
	services:
	postgres:
	# Docker's official image, from its public ECR mirror: Docker Hub's
	# anonymous pull limit failed every attempt of a run. Pinned by the
	# digest CI pulled for the tag postgres:17.
	image: public.ecr.aws/docker/library/postgres@sha256:2d2b8998d31037bf721cfdf764d76ba74171b4fab3431b7f72c27c56ddbdf9e3
	env:
	POSTGRES_DB: sdm_test
	POSTGRES_PASSWORD: postgres
	ports:
	- 5432:5432
	options: >-
	--health-cmd pg_isready --health-interval 5s
	--health-timeout 5s --health-retries 10
	env:
	SDM_POSTGRES_TEST_DSN: postgresql://postgres:postgres@localhost:5432/sdm_test
	SDM_REQUIRE_POSTGRES: '1'
	steps:
	# Every branch, for the gate's checkpoint and its worktree.
	- uses: actions/checkout@v7
	with:
	fetch-depth: 0
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- uses: actions/download-artifact@v8
	with:
	name: native-wheel
	path: native
	# The image's compilers and CMake, which a source build uses; the
	# runner image carries them, so nothing is installed (CI1).
	- run: for tool in gcc g++ make cmake; do command -v "$tool"; done
	# pip's downloads and built wheels, keyed on the week and on every
	# distribution's manifest and restored from the newest of this job's
	# otherwise. pip still resolves each requirement against the index, so
	# the cache only saves downloads and changes no version (CI1).
	- id: week
	run: echo "week=$(date -u +%G-%V)" >> "$GITHUB_OUTPUT"
	- id: pip-cache
	uses: actions/cache/restore@v6
	with:
	path: ~/.cache/pip
	key: pip-${{ runner.os }}-${{ github.job }}-3.12-${{ steps.week.outputs.week }}-${{ hashFiles('packages/*/pyproject.toml') }}
	restore-keys: pip-${{ runner.os }}-${{ github.job }}-3.12-
	# ruff is here for the root suite, not for linting: the test that
	# checks the engine ignore-list note shells out to it and skips
	# without it, which would make that check vacuous here.
	# The Databricks and Snowflake drivers are for the offline tests of those
	# backends, which skip whole modules without them; pyyaml reads the
	# semantic-view fixtures, and `explain` carries the SDK the explanation
	# tests fake around. jsonschema validates the contract's cases and
	# observations, which the root suite checks.
	- run: \|
	pip install --upgrade pip 'ruff==0.16.*'
	pip install -e './packages/structured-data-models-api' -e './packages/structured-data-models-pql' \
	-e './packages/structured-data-models-connectors[sqlite,duckdb,databricks,snowflake,postgres]' \
	pyyaml jsonschema
	pip install -e './packages/structured-data-models-relational-engine[test,explain]'
	# The same marker filter as the internal pipeline. The live NIM and live
	# warehouse tests need credentials this job does not have, so they are
	# deselected by name rather than left to skip. A skip that remains is a
	# test that checked its own environment and stood down, and is reported.
	- run: \|
	cd packages/structured-data-models-relational-engine
	python -m pytest tests -q -m 'not live_nim and not live_warehouse'
	# The client's suite again, with an engine present. The test job installs
	# none, so every client test that needs one skips there and runs nowhere.
	- run: \|
	pip install "$(ls native/*.whl)[test]"
	python -m pytest packages/structured-data-models-client/tests -q \
	-m 'not live_nim and not live_warehouse'
	# And the root suite, which holds the relational wire snapshots. They
	# only exercise anything with an engine installed, so running them in
	# the test job would prove nothing.
	# The contract cases and the runner's controls (contract/tests) run
	# in contract-python, from T1.18.
	- run: python -m pytest tests -q
	- if: ${{ !cancelled() && env.SAVE_CACHES == 'true' && steps.pip-cache.outputs.cache-hit != 'true' }}
	uses: actions/cache/save@v6
	with:
	path: ~/.cache/pip
	key: ${{ steps.pip-cache.outputs.cache-primary-key }}

	# The L2 corpus on the Python runners per runtime cell (H1; design, "CI";
	# T1.18): each cell pinned as run_python.py's CELLS names it, python-core
	# and python-old observed with the checkpoint's runner and judged by the
	# checkpoint's ratchet and comparator through gate.sh, which also
	# validates the register; rust-test runs the L0 scripts. In a cell other
	# than the reference's the comparator holds python-core to python-old on
	# host fields and reports the reference's (H1). Each observation must name
	# the job's cell, so a pin that drifts fails. The runner's controls and
	# the route tests (contract/tests) run once, in ref-pandas3, with their
	# comparisons on the same effective expectations as the gate. One job per
	# cell, sharing its steps: replay_jobs.py substitutes no matrix value but
	# the Python version.
	contract-python-ref-pandas3:
	name: contract-python (ref-pandas3)
	runs-on: ubuntu-latest
	needs: native-wheel
	env:
	CELL: ref-pandas3
	CELL_PINS: pandas==3.0.6 pyarrow==25.0.1 numpy==2.5.3
	steps:
	- &contract-checkout
	uses: actions/checkout@v7
	with:
	# Every branch, for the gate's checkpoint and its worktree.
	fetch-depth: 0
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- &contract-wheel
	uses: actions/download-artifact@v8
	with:
	name: native-wheel
	path: native
	# The cell's pins as constraints, so nothing installed after moves
	# them; DuckDB at the reference's version; the engine, which a case's
	# materialize needs, built from source with the runner image's
	# compilers and CMake.
	- &contract-install
	name: Install the cell
	run: \|
	printf '%s\n' $CELL_PINS duckdb==1.5.6 > "$RUNNER_TEMP/cell.txt"
	pip install --upgrade pip
	pip install -c "$RUNNER_TEMP/cell.txt" $CELL_PINS duckdb==1.5.6 pyyaml jsonschema pytest \
	./packages/structured-data-models-api ./packages/structured-data-models-pql \
	'./packages/structured-data-models-connectors[duckdb]' \
	./packages/structured-data-models-relational-engine "$(ls native/*.whl)[test]"
	- name: The runner's controls and the route tests
	run: python -m pytest contract/tests -q
	# As in rust-test, whether the gate runs fails closed (T1.18); off
	# the phase branch, the candidate's own runners and comparator run.
	# Every cell gates, the floor's too (T1.18's own run alone reported
	# it, E65(c); the waiver went at T1.25b, every checkpoint descending
	# from T1.18's landing). tests/test_contract_ci_gate.py runs this
	# step's text.
	- &contract-gate
	name: gate (python-core, python-old)
	env:
	PYTHON: python
	run: \|
	set -o pipefail
	where=$(python scripts/rust_core/ratchet.py where) \|\| { echo 'gate: ratchet.py where failed'; exit 1; }
	case "${GITHUB_EVENT_NAME:-}:${GITHUB_REF_NAME:-}" in
	push:rust-core-phase-1 \| push:rust-core-phase-1-*)
	[ -n "$where" ] \|\| { echo "gate: ratchet.py where named no base on $GITHUB_REF_NAME, where the gate must run"; exit 1; } ;;
	esac
	mkdir -p gate-observations
	status=0
	if [ -z "$where" ]; then
	echo 'gate: no ratchet off the phase branch: the candidate runners and comparator'
	for runner in python-core python-old; do
	python -I contract/runners/run_python.py --mode $runner \
	--out gate-observations/$runner.jsonl contract/cases \|\| exit 1
	done
	python scripts/rust_core/compare.py --cases contract/cases \
	--reference contract/reference/.jsonl gate-observations/python-.jsonl \|\| status=$?
	else
	scripts/rust_core/gate.sh $where --run python-core,python-old --no-l0 \
	--out gate-observations \|\| status=$?
	fi
	python - <<'EOF' \|\| exit 1
	import json, os, pathlib, sys
	cell = os.environ['CELL']
	for runner in ('python-core', 'python-old'):
	path = pathlib.Path('gate-observations') / f'{runner}.jsonl'
	lines = path.read_text().splitlines() if path.is_file() else []
	try:
	seen = {json.loads(line)['identity']['cell'] for line in lines}
	except (ValueError, RecursionError, KeyError, TypeError):
	sys.exit(f'{runner}: an observation names no cell')
	if seen != {cell}:
	sys.exit(f'{runner}: observed in {sorted(map(str, seen))}, not {cell}')
	print(f'every observation in {cell}')
	EOF
	exit $status

	contract-python-dbr17-pandas2:
	name: contract-python (dbr17-pandas2)
	runs-on: ubuntu-latest
	needs: native-wheel
	env:
	CELL: dbr17-pandas2
	CELL_PINS: pandas==2.2.3 pyarrow==19.0.1 numpy==2.1.3
	steps:
	- *contract-checkout
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- *contract-wheel
	- *contract-install
	- *contract-gate

	contract-python-floor-pandas2-2:
	name: contract-python (floor-pandas2.2)
	runs-on: ubuntu-latest
	needs: native-wheel
	env:
	CELL: floor-pandas2.2
	CELL_PINS: pandas==2.2.0 pyarrow==14.0.1 numpy==1.26.4
	steps:
	- *contract-checkout
	- uses: actions/setup-python@v7
	with:
	python-version: '3.10'
	- *contract-wheel
	- *contract-install
	- *contract-gate

	build:
	runs-on: ubuntu-latest
	needs: native-wheel
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	# The image's compilers and CMake, which a source build uses; the
	# runner image carries them, so nothing is installed (CI1).
	- run: for tool in gcc g++ make cmake; do command -v "$tool"; done
	- run: pip install --upgrade pip build
	# Each distribution builds in an isolated environment holding only the
	# build backend. That catches what no test can, because it fails before a
	# test could run: a version resolved by importing the package root cannot
	# produce a wheel at all. The client is the exception: it carries the
	# core, so it is the wheel native-wheel built, taken as it is.
	- uses: actions/download-artifact@v8
	with:
	name: native-wheel
	path: native
	- run: \|
	for package in structured-data-models-api structured-data-models-pql structured-data-models-connectors structured-data-models-relational-engine; do
	echo "building $package"
	( cd "packages/$package" && python -m build --wheel --outdir "$GITHUB_WORKSPACE/dist" . )
	done
	cp native/*.whl dist/
	# Every distribution carries the one version they are released together
	# at, read from what was just built rather than written here.
	- run: \|
	python - <<'EOF'
	import pathlib, re, sys
	names = sorted(p.name for p in pathlib.Path('dist').glob('*.whl'))
	versions = {re.match(r'[^-]+-([^-]+)-', n).group(1) for n in names}
	print('\n'.join(names))
	if len(names) != 5:
	sys.exit(f'expected 5 wheels, built {len(names)}')
	if len(versions) != 1:
	sys.exit(f'distributions disagree on the version: {sorted(versions)}')
	pathlib.Path('dist/VERSION').write_text(versions.pop())
	EOF
	# Building each one proves nothing about installing them together. A pin
	# only one distribution carries makes the set unsatisfiable, and the
	# resolver is the only thing that says so.
	- run: \|
	VERSION=$(cat dist/VERSION)
	pip install --dry-run --find-links "$GITHUB_WORKSPACE/dist" \
	"structured-data-models-client[relational,sqlite,duckdb]==$VERSION"
	# The wheels only: dist/ also carries VERSION for the jobs downstream,
	# and twine refuses anything that is not a distribution.
	- run: pip install --upgrade twine && twine check dist/*.whl
	- uses: actions/upload-artifact@v7
	with:
	name: wheels
	path: dist/

	# The suite has only ever run against the source tree. What users install is
	# a wheel, and the difference is where packaging bugs live: a module left out
	# of `packages.find`, a `py.typed` that never shipped. This runs from a
	# directory where the source tree is not importable.
	installed-wheel:
	runs-on: ubuntu-latest
	needs: build
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- uses: actions/download-artifact@v8
	with:
	name: wheels
	path: dist
	- run: \|
	pip install --upgrade pip pytest requests-mock
	VERSION=$(cat dist/VERSION)
	pip install --find-links "$GITHUB_WORKSPACE/dist" \
	"structured-data-models-client[sqlite,duckdb]==$VERSION"
	- name: The installed distribution is what gets imported, and it is typed
	working-directory: /tmp
	run: \|
	python - <<'EOF'
	import pathlib
	import predictive_query, sdm_api, sdm_client, sdm_connectors

	for module in (sdm_api, sdm_client, sdm_connectors, predictive_query):
	location = pathlib.Path(module.__file__).resolve()
	assert 'site-packages' in location.parts, (
	f'{module.__name__} was imported from {location}, not from '
	f'the installed wheel, so this job is testing the checkout'
	)
	marker = location.parent / 'py.typed'
	assert marker.is_file(), (
	f'{module.__name__} ships no py.typed in the built wheel, so '
	f'a downstream type checker ignores its annotations entirely'
	)
	print('four distributions imported from site-packages, all typed')

	# The core is in the installed client, not merely in the checkout;
	# found without importing it, which only a core route does.
	import importlib.util
	spec = importlib.util.find_spec('sdm_client._native')
	assert spec and 'site-packages' in spec.origin, spec
	print(f'the core ships in the client: {pathlib.Path(spec.origin).name}')
	EOF
	- working-directory: /tmp
	run: \|
	python -m pytest "$GITHUB_WORKSPACE/packages/structured-data-models-client/tests" -q \
	-m 'not live_nim and not live_warehouse' -p no:cacheprovider

	# The dependency inventory a consumer needs before accepting this into their
	# own supply chain, covering every connector extra that resolves without a
	# native toolchain.
	sbom:
	runs-on: ubuntu-latest
	needs: build
	steps:
	- uses: actions/checkout@v7
	- uses: actions/setup-python@v7
	with:
	python-version: '3.12'
	- uses: actions/download-artifact@v8
	with:
	name: wheels
	path: dist
	- run: \|
	pip install --upgrade pip cyclonedx-bom
	VERSION=$(cat dist/VERSION)
	pip install --find-links "$GITHUB_WORKSPACE/dist" \
	"structured-data-models-client[sqlite,duckdb,postgres,databricks,snowflake,s3]==$VERSION"
	cyclonedx-py environment --of JSON --validate -o dist/sbom.cyclonedx.json
	python -c "import json,sys; d=json.load(open('dist/sbom.cyclonedx.json')); names={c['name'] for c in d.get('components', [])}; missing={'structured-data-models-client','structured-data-models-api','structured-data-models-pql','structured-data-models-connectors'} - names; print(f'{len(names)} components'); sys.exit(f'SBOM missing {sorted(missing)}' if missing else 0)"
	- uses: actions/upload-artifact@v7
	with:
	name: sbom
	path: dist/sbom.cyclonedx.json

Navigation Menu

Repository navigation

T1.20 docs: the locked gate run recorded #457

Workflow file

T1.20 docs: the locked gate run recorded #457

Uh oh!

Workflow file for this run