Skip to content

Commit c2a34f6

Browse files
committed
Add shared-stream producer and trigger to Apache Kafka provider
Event-driven triggers and asset watchers that watch the same Kafka topics each open their own consumer in the triggerer, which does not scale as event-driven schedules grow. Backing them with the shared-stream producer-side ack channel (#67523) lets siblings on the same topics and connection share one consumer, while offset commits remain gated on trigger-event persistence so a triggerer crash cannot drop a message the broker already considers delivered (at-least-once). Signed-off-by: PoAn Yang <payang@apache.org>
1 parent fc6c8b1 commit c2a34f6

6 files changed

Lines changed: 1108 additions & 0 deletions

File tree

‎providers/apache/kafka/provider.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,7 @@ triggers:
113113
python-modules:
114114
- airflow.providers.apache.kafka.triggers.await_message
115115
- airflow.providers.apache.kafka.triggers.msg_queue
116+
- airflow.providers.apache.kafka.triggers.shared_stream
116117

117118
connection-types:
118119
- hook-class-name: airflow.providers.apache.kafka.hooks.base.KafkaBaseHook

‎providers/apache/kafka/src/airflow/providers/apache/kafka/get_provider_info.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ def get_provider_info():
8383
"python-modules": [
8484
"airflow.providers.apache.kafka.triggers.await_message",
8585
"airflow.providers.apache.kafka.triggers.msg_queue",
86+
"airflow.providers.apache.kafka.triggers.shared_stream",
8687
],
8788
}
8889
],

0 commit comments

Comments
 (0)