[pierrebrunelle]

Summary & Motivation

Adds dagster-pixeltable, a resource for running Dagster jobs against existing Pixeltable tables. Dagster schedules runs, orders work across systems, and records run history. Pixeltable stores the data and maintains computed columns, views, and indexes.

• PixeltableResource(expected_home=...): Pixeltable keeps one instance per process, chosen by PIXELTABLE_HOME. Setup rejects a relative or ~ home, and a home that differs from expected_home or from an earlier SDK initialization in the process. It then calls pxt.init(). get_table(path) opens an existing local table, view, or snapshot. It rejects hosted paths (pxt://..., https://pixeltable.com/t/...) and works only between setup and teardown. Assets write with the Pixeltable SDK; the resource adds no wrappers.
• examples/basic: a table declared as a TableModel and deployed with pxt schema update. The asset upserts records by primary key with batch_update(..., if_not_exists="insert") and returns a MaterializeResult with the table name, row count, column schema, table id, and version. An asset check verifies the computed columns.
• examples/video: a pxt.Video table with a frame view. The asset appends with insert, because batch_update cannot write media columns.

Scope of 0.1.0: the in-process executor on one host. Both examples set executor=in_process_executor. There is no IO manager, partition support, multiprocess support, or Component.

Dependencies

• requires-python = ">=3.11": Pixeltable requires 3.11.
• dagster>=1.9.4: Pixeltable requires pydantic>=2.11. Dagster 1.8.5 to 1.9.3 cap pydantic below 2.10, and 1.8.0 to 1.8.4 predate the TableMetadataSet fix that led to that cap.
• pixeltable>=0.7.15, pydantic>=2.11 (direct, for PrivateAttr). No upper bounds and no extras, like the other libraries here.

Notes for reviewers

• The tests use a real embedded Postgres server that Pixeltable starts in a temporary home, plus the pxt daemon on a free port. Session teardown stops both. Docker and credentials are not needed. The suite takes about 35 s.
• examples/video/clip.mp4 (9.9 KB) comes from Pixeltable's Apache-2.0 test data.
• ty check . exits 0 with 3 unsupported-base warnings in the example app.py files, which come from Pixeltable's TableModel metaclass.
• Release tag: dagster_pixeltable-0.1.0. Publishing uses this repository's token, so we would like to agree on who cuts releases. We would also like to know whether a Pixeltable maintainer can be added on PyPI to ship fixes. Pixeltable-side maintainer: @pierrebrunelle.

How I Tested These Changes

Base b93b3eae05b747a57453241ef160dd01df389238, head 7a2c51f2d9f819977b15f79246545f0965aeab29.

OS	Python	dagster	pixeltable	pydantic	Result
macOS 26.6.1 arm64	3.12.7	1.13.26	0.7.16	2.14.1	ruff, ty, pytest: 35 passed
macOS 26.6.1 arm64	3.11.13	1.9.4	0.7.15	2.11.0	pytest: 35 passed (--resolution lowest-direct)
Debian 12 aarch64 (Docker)	3.12.12	1.13.26	0.7.16	2.14.1	ruff, ty, pytest: 35 passed
Debian 12 x86_64 (Docker, emulated)	3.12.12	1.13.26	0.7.16	2.14.1	ty, pytest: 35 passed

The Linux runs used the shared workflow's steps: uv python install 3.12, uv sync --all-extras, ruff check, ruff format --check, ty check ., pytest.

The tests cover:

• Lifecycle: import and construction leave Pixeltable uninitialized. Setup works against the expected home, with two resources and repeated runs. Teardown ends table access after both successful and failed assets.
• Home checks: setup rejects a mismatch with an earlier initialization, conflicting expected homes, and relative or ~ homes, including a cached relative home that would otherwise resolve to the expected one.
• Table access: a missing table is not created, and all hosted path forms are rejected before the SDK is called. Version references open read-only handles.
• Basic example: replay keeps one row per id, and an empty batch skips the write. Invalid ids and SDK validation errors leave the table unchanged. A missing primary key fails the run. A stored computed error commits the row and fails the check. The tests also cover the metadata and the executor setting.
• README commands: pxt schema update, then dagster asset materialize twice, then a readback from a fresh process.
• Video example: each run appends the clip and its frames, and batch_update rejects media even for new rows.

Also:

• Wheel: built the wheel and sdist; the wheel contains only dagster_pixeltable/. Installed it into a fresh venv without extras. From a copy of examples/ outside the checkout, ran pxt schema update, ran dagster asset materialize twice (3 rows, 0 computed errors), and ran the video example.
• dagster dev (with dagster-webserver 1.13.26): materialized ingest_documents from the UI. The run succeeded, and the asset page showed row count 3 and the column schema.
• Pyright (supplementary): reports 4 reportPrivateImportUsage errors in the package, for pxt.home, pxt.init, pxt.get_table, and pxt.Table. Pixeltable builds __all__ at runtime, so Pyright cannot see its exports. This is a Pixeltable typing issue, not specific to this package.

Changelog

libraries/dagster-pixeltable/CHANGELOG.md has an [Unreleased] entry.