Repository navigation
Commit c2a34f6
committed
Add shared-stream producer and trigger to Apache Kafka provider
Event-driven triggers and asset watchers that watch the same Kafka
topics each open their own consumer in the triggerer, which does not
scale as event-driven schedules grow. Backing them with the
shared-stream producer-side ack channel (#67523) lets siblings on the
same topics and connection share one consumer, while offset commits
remain gated on trigger-event persistence so a triggerer crash cannot
drop a message the broker already considers delivered (at-least-once).
Signed-off-by: PoAn Yang <payang@apache.org>1 parent fc6c8b1 commit c2a34f6
6 files changed
Lines changed: 1108 additions & 0 deletions
File tree
- providers/apache/kafka
- src/airflow/providers/apache/kafka
- triggers
- tests
- integration/apache/kafka/triggers
- unit/apache/kafka/triggers
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
| 116 | + | |
116 | 117 | | |
117 | 118 | | |
118 | 119 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
| 86 | + | |
86 | 87 | | |
87 | 88 | | |
88 | 89 | | |
| |||
0 commit comments