Repository navigation
Commit cb9baf8
committed
Report a Lang-SDK parse past its import timeout as an import error
The parse child resolves the get_dagbag_import_timeout policy for the
file and sends the timeout with the runtime's schema version. The policy
thus runs where it does for a Python Dag file, and a failure is only
that file's import error. The timeout counts from the start of the
parse. Past it, with no result yet, the runtime is killed and the file
gets an import error.
LangSDKDagFileProcessorProcess.run applies the same timeout, so it no
longer takes one. Until the child reports it, dag_file_processor_timeout
bounds the parse, as it does in the Dag processor. At the deadline run
stops waiting even if an exited runtime left a process that holds its
output open.1 parent 789dc06 commit cb9baf8
5 files changed
Lines changed: 174 additions & 87 deletions
File tree
- airflow-core
- src/airflow/dag_processing
- tests/unit/dag_processing
- task-sdk
- src/airflow/sdk/coordinators
- tests/task_sdk/coordinators
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| |||
78 | 80 | | |
79 | 81 | | |
80 | 82 | | |
81 | | - | |
| 83 | + | |
82 | 84 | | |
83 | 85 | | |
| 86 | + | |
| 87 | + | |
84 | 88 | | |
85 | 89 | | |
86 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
87 | 99 | | |
88 | 100 | | |
89 | 101 | | |
| |||
101 | 113 | | |
102 | 114 | | |
103 | 115 | | |
104 | | - | |
| 116 | + | |
105 | 117 | | |
106 | 118 | | |
| 119 | + | |
| 120 | + | |
107 | 121 | | |
108 | 122 | | |
109 | 123 | | |
| |||
149 | 163 | | |
150 | 164 | | |
151 | 165 | | |
| 166 | + | |
152 | 167 | | |
153 | 168 | | |
154 | 169 | | |
| |||
212 | 227 | | |
213 | 228 | | |
214 | 229 | | |
215 | | - | |
216 | 230 | | |
217 | 231 | | |
218 | 232 | | |
219 | 233 | | |
220 | 234 | | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
225 | 238 | | |
| 239 | + | |
226 | 240 | | |
227 | 241 | | |
228 | 242 | | |
| |||
235 | 249 | | |
236 | 250 | | |
237 | 251 | | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
246 | 259 | | |
247 | 260 | | |
248 | 261 | | |
| |||
376 | 389 | | |
377 | 390 | | |
378 | 391 | | |
| 392 | + | |
379 | 393 | | |
380 | 394 | | |
381 | 395 | | |
| |||
438 | 452 | | |
439 | 453 | | |
440 | 454 | | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
441 | 462 | | |
442 | 463 | | |
443 | 464 | | |
| |||
449 | 470 | | |
450 | 471 | | |
451 | 472 | | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
452 | 480 | | |
453 | 481 | | |
454 | 482 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
39 | 40 | | |
40 | 41 | | |
41 | 42 | | |
| |||
332 | 333 | | |
333 | 334 | | |
334 | 335 | | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
335 | 357 | | |
336 | 358 | | |
337 | 359 | | |
| |||
357 | 379 | | |
358 | 380 | | |
359 | 381 | | |
360 | | - | |
| 382 | + | |
361 | 383 | | |
362 | | - | |
| 384 | + | |
363 | 385 | | |
364 | 386 | | |
365 | 387 | | |
366 | | - | |
367 | 388 | | |
368 | 389 | | |
369 | 390 | | |
| |||
398 | 419 | | |
399 | 420 | | |
400 | 421 | | |
401 | | - | |
| 422 | + | |
| 423 | + | |
402 | 424 | | |
403 | | - | |
404 | 425 | | |
405 | 426 | | |
406 | 427 | | |
| |||
418 | 439 | | |
419 | 440 | | |
420 | 441 | | |
421 | | - | |
422 | 442 | | |
423 | | - | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | | - | |
430 | | - | |
| 443 | + | |
431 | 444 | | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
432 | 448 | | |
433 | 449 | | |
434 | 450 | | |
435 | 451 | | |
436 | 452 | | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
437 | 497 | | |
438 | 498 | | |
439 | 499 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1768 | 1768 | | |
1769 | 1769 | | |
1770 | 1770 | | |
| 1771 | + | |
| 1772 | + | |
| 1773 | + | |
| 1774 | + | |
| 1775 | + | |
| 1776 | + | |
| 1777 | + | |
| 1778 | + | |
| 1779 | + | |
| 1780 | + | |
| 1781 | + | |
| 1782 | + | |
| 1783 | + | |
| 1784 | + | |
| 1785 | + | |
| 1786 | + | |
| 1787 | + | |
| 1788 | + | |
| 1789 | + | |
| 1790 | + | |
| 1791 | + | |
| 1792 | + | |
| 1793 | + | |
| 1794 | + | |
| 1795 | + | |
| 1796 | + | |
| 1797 | + | |
| 1798 | + | |
| 1799 | + | |
| 1800 | + | |
| 1801 | + | |
| 1802 | + | |
| 1803 | + | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
| 1814 | + | |
| 1815 | + | |
| 1816 | + | |
| 1817 | + | |
| 1818 | + | |
1771 | 1819 | | |
1772 | 1820 | | |
1773 | 1821 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
27 | 26 | | |
28 | 27 | | |
29 | 28 | | |
| |||
45 | 44 | | |
46 | 45 | | |
47 | 46 | | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | 47 | | |
62 | 48 | | |
63 | 49 | | |
| |||
104 | 90 | | |
105 | 91 | | |
106 | 92 | | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
124 | 102 | | |
125 | 103 | | |
126 | 104 | | |
| |||
0 commit comments