Skip to content

Commit 155d89a

Browse files
committed
Add shared-stream producer and trigger to Apache Kafka provider
Event-driven triggers and asset watchers that watch the same Kafka topics each open their own consumer in the triggerer, which does not scale as event-driven schedules grow. Backing them with the shared-stream producer-side ack channel (#67523) lets siblings on the same topics and connection share one consumer, while offset commits remain gated on trigger-event persistence so a triggerer crash cannot drop a message the broker already considers delivered (at-least-once). Signed-off-by: PoAn Yang <payang@apache.org>
1 parent 9eea9a0 commit 155d89a

6 files changed

Lines changed: 1109 additions & 0 deletions

File tree

‎providers/apache/kafka/provider.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ triggers:
110110
python-modules:
111111
- airflow.providers.apache.kafka.triggers.await_message
112112
- airflow.providers.apache.kafka.triggers.msg_queue
113+
- airflow.providers.apache.kafka.triggers.shared_stream
113114

114115
connection-types:
115116
- hook-class-name: airflow.providers.apache.kafka.hooks.base.KafkaBaseHook

‎providers/apache/kafka/src/airflow/providers/apache/kafka/get_provider_info.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ def get_provider_info():
8383
"python-modules": [
8484
"airflow.providers.apache.kafka.triggers.await_message",
8585
"airflow.providers.apache.kafka.triggers.msg_queue",
86+
"airflow.providers.apache.kafka.triggers.shared_stream",
8687
],
8788
}
8889
],

0 commit comments

Comments
 (0)