Skip to content

feat(dagster-openlineage): emit jobType and per-asset ownership facets - #353

Open
dan-newton-sparta wants to merge 1 commit into
dagster-io:mainfrom
dan-newton-sparta:feat/openlineage-job-facets
Open

dan-newton-sparta wants to merge 1 commit into
dagster-io:mainfrom
dan-newton-sparta:feat/openlineage-job-facets

Conversation

@dan-newton-sparta

@dan-newton-sparta dan-newton-sparta commented Sep 11, 2026 •

Copy link
Copy Markdown

Attach a jobType facet and an ownership facet to every emitted job.

Summary & Motivation

Emitted jobs carried neither a jobType nor an ownership facet, so a lineage backend could not tell that a pipeline was a Dagster job, nor who owned it.

This adds:

  • a jobType facet (integration=DAGSTER, processingType=BATCH) on every job, so backends attribute the pipeline to Dagster rather than a generic fallback; and
  • an ownership facet. The asset-emitting methods take an owners argument (the asset's own Dagster owners — team:<team> / email), so different assets can be owned by different teams. Jobs with no per-asset owners fall back to a default team from OPENLINEAGE_TEAM or the team= argument; with neither, no ownership facet is emitted.

This lets a backend such as OpenMetadata route the pipeline to its Dagster service and set the pipeline owner from the team. The sensor wires each asset's native owners into the owners argument in a follow-up (see Related changes). Backward compatible: no owners and no default team reproduces the previous output.

How I Tested These Changes

  • Unit tests: jobType present on every job; per-asset owners override the default team; default-team fallback via team= and OPENLINEAGE_TEAM; arg-over-env precedence; no owners → no facet; owners threaded onto a real materialization's job.
  • uv run pytest green; ruff check, ruff format --check, and ty check . clean.
  • Tested locally end to end: ingested the emitted OpenLineage events into OpenMetadata and used it to visualise the resulting lineage.

Changelog

Added a ### Features entry under ## [Unreleased] in CHANGELOG.md.

Related changes

Part of a set of small improvements to Dagster → OpenLineage. The lineage-fidelity changes are each independent and mergeable on their own; the ownership pair is stacked (#354 builds on #353).

Lineage fidelity:

Ownership:

Attach a jobType facet (integration=DAGSTER, processingType=BATCH) to every
job, so lineage backends can attribute the pipeline to Dagster rather than a
generic fallback.

Attach an ownership facet built from each asset's own Dagster owners
(team:<team> / email), threaded through the asset-emitting methods, so
different assets can be owned by different teams. Any job with no per-asset
owners — pipeline/step events, or assets that declare none — falls back to a
default team from OPENLINEAGE_TEAM or the team= adapter argument. No per-asset
owners and no default team means no ownership facet, so existing deployments
are unaffected. jobType is built once and applied to every job.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant