Airflow worker pods starts very slow because of K8s init container

[jagusztinl]

Apache Airflow version:

2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1-5-g76a04fc", GitCommit:"c66c03f3012a10f16eb86fdce6330433adf6c9ee", GitTreeState:"clean", BuildDate:"2021-02-13T03:54:59Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3", GitCommit:"9cdca7b", GitTreeState:"clean", BuildDate:"2020-10-02T23:12:03Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  Azure+Openshift

• OS (e.g. from /etc/os-release):
  NAME="Red Hat Enterprise Linux"
  VERSION="8.3 (Ootpa)"

• Kernel (e.g. uname -a):
  xlocalhost.localdomain 4.18.0-240.el8.x86_64 Improving the search functionality in the graph view #1 SMP Wed Sep 23 05:13:10 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• HELM

• Others:

What happened:

helm install airflow airflow --set executor=KubernetesExecutor.....

What you expected to happen:

oc get pods:
examplebranchoperatorbranching.405a5270d25b4783a219c48e2f9fc267 0/1 Init:1/2 0 101s

worker pods are starting very slow.
investigated the logs, the k8s init container takes a lot of time to start:
�[37mnami �[39m �[32mINFO �[39m Initializing airflow-worker
�[36mairflow�[39m �[32mINFO �[39m ==> No injected configuration files found. Creating default config file...
�[36mairflow�[39m �[32mINFO �[39m ==> Configuring Airflow database...
�[36mairflow�[39m �[32mINFO �[39m ==> Setting Airflow Fernet key...
�[36mairflow�[39m �[32mINFO �[39m ==> Deploying Airflow from scratch...
�[36mpostgre�[39m �[32mINFO �[39m Trying to connect to PostgreSQL server
�[36mpostgre�[39m �[32mINFO �[39m Found PostgreSQL server listening at airflow-postgresql:5432
�[36mpostgre�[39m �[32mINFO �[39m PostgreSQL server listening and working at airflow-postgresql:5432
�[36mairflow�[39m �[32mINFO �[39m ==> Waiting for Airflow Webserver host...
�[36mairflow�[39m �[32mINFO �[39m
�[36mairflow�[39m �[32mINFO �[39m ########################################################################
�[36mairflow�[39m �[32mINFO �[39m Installation parameters for airflow-worker:
�[36mairflow�[39m �[32mINFO �[39m Webserver URL: airflow:8080
�[36mairflow�[39m �[32mINFO �[39m This installation requires no credentials.
�[36mairflow�[39m �[32mINFO �[39m ########################################################################
�[36mairflow�[39m �[32mINFO �[39m
�[37mnami �[39m �[32mINFO �[39m airflow-worker successfully initialized

How to make it faster?

How to reproduce it:

Use Airflow with kubernetes executor and check worker pod starting time.

Anything else we need to know:

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[jagusztinl]

I have been found that the init container stops here for 30 secs:
�[37mnami �[39m �[32mINFO �[39m Initializing airflow-worker
�[36mairflow�[39m �[32mINFO �[39m ==> No injected configuration files found. Creating default config file...

How to make this faster?