Skip to content

Airflow 1.10.14 - celery worker doesn't want to connect to redis. #14100

Description

@Rudasek86

Apache Airflow version: 1.10.14

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment: Red Hat Enterprise Linux 8.2 (VMware)

  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release): Red Hat Enterprise Linux 8.2
  • Kernel (e.g. uname -a): 4.18.0-193.19.1.el8_2.x86_64 Improving the search functionality in the graph view #1 SMP Wed Aug 26 15:29:02 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: Airflow 1.10.14, redis.x86_64 (version 5.0.3.2), redis client installed with airflow (redis 3.5.3)
  • Others:

What happened:

We have upgraded version of airflow from 1.10.2 to 1.10.14 and there is a problem to connect to redis by celery worker.
This is really wierd problem because when we run a airflow celery worker (without -D parameter) it's connecting without problem. In Flower tool we can see redis connection is online, but when we try do it with -D, airflow celery worker -D there are errors like below.

[2021-02-04 17:33:00,442: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 107 connecting to redis:6379. Operation on closed file..
Trying again in 12.00 seconds... (6/100)
Before upgrade on version 1.10.2, we didin't have a problems like that. Our configuration has been working with settings
broker_url = redis://localhost:6379/0

Of course our redis is working on default port on this same machine
netstat-ntlp

tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN -

Now we have a settings IN airflow cfg we have setting like below:

broker_url = redis://localhost/0
We have tried also settings like below:

broker_url = redis://localhost:6379/0
broker_url = redis://127.0.0.1/0
broker_url = redis://127.0.0.1:6379/0
broker_url = redis://redis:6379/0
broker_url = redis://redis/0

What you expected to happen:

We would like to connect to redis on airflow celery deamon functionality.

We do not really know, redis server version is this same, when airflow 1.10.2 was connecting sucessfully.
Later Version of airflow and redis client has been upgraded.

How to reproduce it:

Anything else we need to know:

Activity

  1. boring-cyborg commented on Feb 5, 2021

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template!

  2. added
    priority:mediumBug that should be fixed before next release but would not block a release
    on Feb 6, 2021
  3. locked and limited conversation to collaborators on Oct 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:corekind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions