Better integration with IDE debuggers to test and debug DAGs/tasks

[GuillaumeDesforges]

When I want to debug a DAG and its tasks, I find airflow dags test and airflow tasks test to be very helpful.
There is the interesting -m option to debug on uncaught exception which is nice.

However, without an IDE debugger it is harder to use.

Since I'm on VSCode, I end up adding

import debugpy

debugpy.listen(("localhost", 5678))
debugpy.wait_for_client()

in my Python tasks, then use my IDE to connect.

It works pretty well, but it is not handy at all. I have to add the code snippet and remove it every time I do a debugging session.

I'm not sure how this could be improved, but that would be a great plus for Data Engineers.

[turbaszek]

Have you tried using debug executor?
https://airflow.apache.org/docs/apache-airflow/stable/executor/debug.html

[GuillaumeDesforges]

Hi @turbaszek, DebugExecutor looks indeed like the way to go, but it's a bit tedious to select the right DAG (I generate them dynamically using a db) and the execution date to debug without committing it to my Git repo.
Other issue I have, I can't select a specific task to run and debug.
Would you have suggestions on how to improve that compared to the doc ?

[potiuk]

@GuillaumeDesforges . Could you please elaborate on what should be your 'ideal' workflow for debugging then ? I'd love if you describe how you would like to perform your Debug sessions, what commands you would like to execute etc.

Maybe what you really need is another flag (say --debug) on those two commands airflow dags test , airflow tasks test that would start debugger and wait for it to connect from VSCode (without you having to add those lines ?

Would that work?

[GuillaumeDesforges]

Maybe what you really need is another flag (say --debug) on those two commands airflow dags test , airflow tasks test that would start debugger and wait for it to connect from VSCode (without you having to add those lines ?

That does sound very interesting (to me at least)

[potiuk]

Would you care to create an issue and describe your use-case @GuillaumeDesforges ?

[GuillaumeDesforges]

Done #15119
Thanks