Tasks stuck in "Queued" (LocalExecutor)

[FredericoCoelhoNunes]

Hi everyone,

I would love to hear your advice regarding the best way to set up the LocalExecutor.
Up to this point I have been using the SequentialExecutor and the DAG in question runs just fine. However, I have recently started needing to run tasks in parallel, so I have launched a local PostgreSQL database and switched to the local executor.

However, the same DAG now doesn't even start running - the first tasks are stuck in the "Queued" state for a minute or so, and then are set to "Failed". Here is a log file (dag/task names were modified):

[2021-02-15 10:06:49,568] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: my_task 2021-02-15T10:03:48.794562+00:00 [queued]>
[2021-02-15 10:07:11,664] {taskinstance.py:826} INFO - Dependencies all met for <TaskInstance: my_task 2021-02-15T10:03:48.794562+00:00 [queued]>
[2021-02-15 10:07:11,670] {taskinstance.py:1017} INFO - 
--------------------------------------------------------------------------------
[2021-02-15 10:07:11,670] {taskinstance.py:1018} INFO - Starting attempt 1 of 1
[2021-02-15 10:07:11,670] {taskinstance.py:1019} INFO - 
--------------------------------------------------------------------------------
[2021-02-15 10:07:11,700] {taskinstance.py:1038} INFO - Executing <Task(OracleDropTablesOperator): my_task> on 2021-02-15T10:03:48.794562+00:00
[2021-02-15 10:07:11,706] {standard_task_runner.py:51} INFO - Started process 29714 to run task
[2021-02-15 10:07:11,710] {standard_task_runner.py:75} INFO - Running: ['airflow', 'tasks', 'run', 'my_dag', 'my_task', '2021-02-15T10:03:48.794562+00:00', '--job-id', '8', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/dags/my_dag.py', '--cfg-path', '/tmp/tmptjqg05f0']
[2021-02-15 10:07:11,711] {standard_task_runner.py:76} INFO - Job 8: Subtask my_task
[2021-02-15 10:08:17,930] {logging_mixin.py:103} INFO - Running <TaskInstance: my_task 2021-02-15T10:03:48.794562+00:00 [running]> on host my_host
[2021-02-15 10:08:22,985] {local_task_job.py:169} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2021-02-15 10:08:22,990] {process_utils.py:95} INFO - Sending Signals.SIGTERM to GPID 29714
[2021-02-15 10:08:23,004] {process_utils.py:61} INFO - Process psutil.Process(pid=29714, status='terminated', exitcode=<Negsignal.SIGTERM: -15>, started='10:07:10') (29714) terminated with exit code Negsignal.SIGTERM
[2021-02-15 10:08:23,004] {local_task_job.py:118} INFO - Task exited with return code Negsignal.SIGTERM

At first I thought it might be a database connection issue, but airflow db check only takes ~10 seconds and returns a success message. And this database is being run locally so I don't see why it would be significantly slower than the default sqlite database.

I have no clue how to go about debugging this! So any tips would be much appreciated :)

Cheers

[FredericoCoelhoNunes]

I wasn't able to run any of my DAGs, so I decided to test with the Airflow "tutorial" DAG (https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html), with the SequentialExecutor.
Apparently the problem is in the connection speed to the backend, because the tasks did run, but each task spends approximately 3 mins in queued and takes 4 minutes to run:

I don't know why this is happening, because this is a locally hosted Postgres database - if anyone has any insights, I'd love to hear them!