Repository navigation
Databricks: document and validate explicit Unity assets across SQL and job operators #74196
Description
Activity
Hi @zozo123 ! I would love to work on this issue.
I see that it is a follow-up to #74191. Since it focuses on backend logic, documentation, and parameter validation for explicit inlets/outlets on theDatabricksSQLStatementsOperator,DatabricksRunNowOperator, andDatabricksSubmitRunOperator, I can handle this
Should i Start working on this ? Thanks!Reacted by Yossi Eliaz@kadubhumika would be great. thank you for offering <3
Reacted by BHUMIKA KADU✨@zozo123 I’m ready to start working on this issue. I’ll cover the validation, docs, and tests for DatabricksSQLStatementsOperator, DatabricksRunNowOperator, and DatabricksSubmitRunOperator.
Before I start coding, I want to clarify how to handle asset events when wait_for_termination=False (fire-and-forget mode). Since Airflow marks the task as successful immediately after submitting the job, which path should we take?
• Option A (Suppress the Signal): Block the outlet asset event from firing completely since we cannot confirm if the remote table actually refreshed.
• Option B (Document the Behavior): Allow the normal task-success event to fire, but explicitly document that it represents a successful submission, not a completed table refresh.
Which semantic approach fits best with your vision? Thanks!@zozo123 .
Reacted by Yossi EliazThanks @kadubhumika — and sorry for the wait. Really appreciate you asking this before coding.
Please go with B, gently: keep caller-provided outlets as set (including
[]), and let Airflow’s normal task-success outlet behavior stand whenwait_for_termination=False. In that mode the event means the Databricks run was submitted, not that a Unity/Delta table refreshed — so please document that clearly, and add a friendly init warning if outlets are set with fire-and-forget.Please don’t have the provider emit or upgrade any verified Unity/table-refresh asset event in fire-and-forget. Verified outputs only after the remote run actually succeeds (sync and deferrable completion). Authors who need “table ready” scheduling are better served by
wait_for_termination=Trueor a sensor (see #74225).Starting with
DatabricksSQLStatementsOperatoris perfect; we can mirror to RunNow/SubmitRun once that looks good. Happy to review the first cut — shout if anything is unclear.Reacted by BHUMIKA KADU✨@zozo123 Thanks, I'll go with B, starting with DatabricksSQLStatementsOperator Thank you!
Hi @zozo123 , the implementation for #74196 is ready for review.
I’ve added the explicit Databricks asset examples and documentation for SQL, RunNow, and SubmitRun operators, along with the warning forwait_for_termination=False, outlet-preservation behavior, and focused tests.
The focused unit tests and Ruff checks pass locally, and the PR CI is currently running.
Could you please review it when you get a chance? If anything needs to be changed, please let me know and I’ll address it.
Thanks!
Description
Extend the Unity asset examples beyond COPY INTO, beginning with explicit inlets/outlets on SQL and job operators. Evaluate typed input_tables/output_tables convenience only if examples demonstrate an ergonomic or validation gap beyond existing BaseOperator arguments.
Use case/motivation
SQL, jobs and pipelines may write known Unity tables even when automatic SQL lineage is unavailable. Authors need consistent asset identities and success semantics without maintaining a separate inferred SQL parser.
Delivery and acceptance
Related issues
Follow-up to #74191. Its current docs already recommend explicit outlets for SQL/job operators; this request starts from that supported capability, rather than assuming it is absent. Pipeline-specific extensions can follow demonstrated demand.