Skip to content

Issues with SQL streaming patterns code snippets #583

Description

@cyril-brk

Some of the code snippets for SQL streaming in declarative pipelines in https://github.com/databricks-solutions/ai-dev-kit/blob/main/databricks-skills/databricks-spark-declarative-pipelines/references/sql/3-streaming-patterns.md are not correct

  1. Deduplication Pattern snippet
Bronze: Ingest all (may contain duplicates)
CREATE OR REFRESH STREAMING TABLE bronze_events AS
SELECT *, current_timestamp() AS _ingested_at
FROM STREAM read_files(...);

-- Silver: Deduplicate by event_id
CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS
SELECT
  event_id, user_id, event_type, event_timestamp, _ingested_at
FROM (
  SELECT
    *,
    ROW_NUMBER() OVER (PARTITION BY event_id ORDER BY event_timestamp) AS rn
  FROM STREAM bronze_events
)
WHERE rn = 1;

These types of window functions do not work with streaming tables.

  1. Time window deduplication snippet:
CREATE OR REFRESH STREAMING TABLE silver_events_dedup AS
SELECT
  event_id, user_id, event_type, event_timestamp,
  MIN(_ingested_at) AS first_seen_at
FROM STREAM bronze_events
GROUP BY
  event_id, user_id, event_type, event_timestamp,
  window(event_timestamp, '1 hour')
HAVING COUNT(*) >= 1;

The having clause should be just be "=1" instead of ">=1"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions