Should mysql transaction isolation level set to "read commited"?

[doowhtron]

After my airflow upgraded to 2.0.0(and then 2.0.1) and scheduler expanded to 3 nodes, something weird happened：

1. dagruns were success but the task instances were not scheduled at all
2. task failed with a null hostname #13692
3. Task is set "upstream_failed" while upstream tasks are success #13671

These phenomena never happened when there was only one scheduler node.

And I found that after task instances of a new dagrun were created by a scheduler node, they were not found in another scheduler node's task_instance_scheduling_decisions function.

Then I checked the mysql configurations and found transaction isolation was set to be Repeatable read by default.

After I set transaction isolation to be read commited, everything seems to be good now. But I still wonder are there any side effects?

[turbaszek]

This is indeed interesting. Possibly @ashb or @kaxil can have some more insights about this behavior?

[doowhtron]

Thanks for your reply. I read the docs and cannot find the instructions about the transaction isolation of the database.

I wonder if it is a key configuration for users.

[ashb]

So PostgreSQLs default is Read Commited, so perhaps.

That said, absence of evidence is not evidence of absence, so setting the isolation level may have just hidden the problem for now rather than fixing it 100%