Repository navigation
feat(dagster-openlineage): emit jobType and per-asset ownership facets - #353
Open
dan-newton-sparta wants to merge 1 commit into
Open
dan-newton-sparta wants to merge 1 commit into
dan-newton-sparta wants to merge 1 commit into
Conversation
Attach a jobType facet (integration=DAGSTER, processingType=BATCH) to every job, so lineage backends can attribute the pipeline to Dagster rather than a generic fallback. Attach an ownership facet built from each asset's own Dagster owners (team:<team> / email), threaded through the asset-emitting methods, so different assets can be owned by different teams. Any job with no per-asset owners — pipeline/step events, or assets that declare none — falls back to a default team from OPENLINEAGE_TEAM or the team= adapter argument. No per-asset owners and no default team means no ownership facet, so existing deployments are unaffected. jobType is built once and applied to every job.
This was referenced Sep 11, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Attach a
jobTypefacet and anownershipfacet to every emitted job.Summary & Motivation
Emitted jobs carried neither a
jobTypenor anownershipfacet, so a lineage backend could not tell that a pipeline was a Dagster job, nor who owned it.This adds:
jobTypefacet (integration=DAGSTER,processingType=BATCH) on every job, so backends attribute the pipeline to Dagster rather than a generic fallback; andownershipfacet. The asset-emitting methods take anownersargument (the asset's own Dagsterowners—team:<team>/ email), so different assets can be owned by different teams. Jobs with no per-asset owners fall back to a default team fromOPENLINEAGE_TEAMor theteam=argument; with neither, no ownership facet is emitted.This lets a backend such as OpenMetadata route the pipeline to its Dagster service and set the pipeline owner from the team. The sensor wires each asset's native
ownersinto theownersargument in a follow-up (see Related changes). Backward compatible: no owners and no default team reproduces the previous output.How I Tested These Changes
team=andOPENLINEAGE_TEAM; arg-over-env precedence; no owners → no facet; owners threaded onto a real materialization's job.uv run pytestgreen;ruff check,ruff format --check, andty check .clean.Changelog
Added a
### Featuresentry under## [Unreleased]inCHANGELOG.md.Related changes
Part of a set of small improvements to Dagster → OpenLineage. The lineage-fidelity changes are each independent and mergeable on their own; the ownership pair is stacked (#354 builds on #353).
Lineage fidelity:
exclude_asset_keysto the storage wrapperOwnership: