Repository navigation
Commit 155d89a
committed
Add shared-stream producer and trigger to Apache Kafka provider
Event-driven triggers and asset watchers that watch the same Kafka
topics each open their own consumer in the triggerer, which does not
scale as event-driven schedules grow. Backing them with the
shared-stream producer-side ack channel (#67523) lets siblings on the
same topics and connection share one consumer, while offset commits
remain gated on trigger-event persistence so a triggerer crash cannot
drop a message the broker already considers delivered (at-least-once).
Signed-off-by: PoAn Yang <payang@apache.org>1 parent 9eea9a0 commit 155d89a
6 files changed
Lines changed: 1109 additions & 0 deletions
File tree
- providers/apache/kafka
- src/airflow/providers/apache/kafka
- triggers
- tests
- integration/apache/kafka/triggers
- unit/apache/kafka/triggers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
| 113 | + | |
113 | 114 | | |
114 | 115 | | |
115 | 116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| 86 | + | |
86 | 87 | | |
87 | 88 | | |
88 | 89 | | |
| |||
0 commit comments