Skip to content

Databricks: add Delta table version sensor (#74195) - #74550

Open
garvit-arora wants to merge 1 commit into
apache:mainfrom
garvit-arora:feat/74195-delta-version-sensor
Open

garvit-arora wants to merge 1 commit into
apache:mainfrom
garvit-arora:feat/74195-delta-version-sensor

Conversation

@garvit-arora

Copy link
Copy Markdown

What this changes

Adds the first, standalone slice of #74195: DatabricksDeltaTableVersionSensor and its deferrable trigger. The sensor observes DESCRIBE HISTORY, enrolls a baseline or waits for an explicit target, emits version and table-identity provenance, and supports the provider's existing Unity Catalog asset URI.

This branch incorporates the public implementation foundation from #74225 with attribution in the commit, then addresses its latest blocking review findings:

  • mode="reschedule" now requires baseline_version or target_version, so process-local enrollment cannot restart forever.
  • Rendered target_version values are converted to integers before comparison.
  • Recent history is scanned so OPTIMIZE, VACUUM START, and VACUUM END do not cause a data-change wakeup. The policy and scan depth are configurable.
  • Documentation explains SQL warehouse polling cost, including that deferrable execution frees an Airflow worker but still runs warehouse queries.

The external producer/checkpointed delivery slice remains out of scope for this PR.

Validation

  • python -m compileall completed for the changed sensor, trigger, and tests.
  • git diff --check passed.
  • The focused pytest command could not start in this Windows environment because the repository's all-extras development environment builds pykerberos, which requires unavailable gssapi/gssapi.h headers. No test failure was reached.

AI use

I used an AI coding assistant for implementation support and reviewed the resulting changes.

Co-authored-by: Aditya Goyal <adityagoyal@Adityas-MacBook-Air-4.local>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant