How to disable the airflow scheduler while keeping the webserver on

[vibhavbol]

I am running airflow using AWS ECS. For testing something, I need to disable the airflow scheduler. But I am not able to do so. I have tried several things:

1. Kill the scheduler from docker by a kill command:
   For this we require the pid for airflow scheduler which I am not able to find. My airflow docker contains only following files/folders - airflow-webserver.pid, airflow.cfg, dags, logs, plugins, unittests.cfg, webserver_config.py
   There is no airflow-scheduler.pid file.
   there is no result for command jobs.
   (So the basic problem with killing the scheduler is that there is no way to get scheduler pid)
2. I also tried making changes to entrypoint and CMD. Originally I was using the same entry point and CMD as there is in the standard airflow Dockerfile
   ENTRYPOINT ["/usr/bin/dumb-init", "--", "/entrypoint"]
CMD ["webserver"]
   Using these commands simultaneously starts the scheduler and webserver I figured. So I tried to replace it with something like
   ENTRYPOINT ["/bin/bash", "-c"]
CMD ["airflow webserver &"]
   in order to only start the webserver and not the scheduler. But it doesn't work (not even webserver is started)
   After changing it to
   ENTRYPOINT ["/bin/bash", "-c"]
CMD ["airflow webserver "]
   Both scheduler and webserver are running again.

So basically I need a way to create a condition such that only web server is running and the scheduler is not running. Please let me know if anyone can find any solution.

[turbaszek]

What image do you use?

[vibhavbol]

airflow_image

I basically build the airflow docker using command
docker build --rm -f Dockerfile-airflow -t airflow_image .
This is when I am running locally. For AWS ECS, we use the airflow docker build on circleCi

The Dockerfile-airflow is based on the latest dockerfile in airflow 2.0 on github

[turbaszek]

To run only webserver run

docker run apache/airflow webserver 
# Or in your case
docker run airflow_image webserver

to run scheduler and webserver I would run first scheduler in detached mode (using docker -d flag) and then start webserver.

[vibhavbol]

When we deploy the airflow docker build on AWS ECS, it starts the docker run by itself. We don't execute any command for the docker run. I was able to achieve not running scheduler on my local machine by doing a bash into airflow docker and starting the webserver only. But It is not possible on AWS as I have no control over the docker run command when airflow docker is deployed. Furthermore, I cannot do a bash into airflow docker and then start the webserver like in the case when I'm running locally. Because the machine does not reach steady-state unless the webserver is running. Is there some kind of solution that would involve changes in the docker file? Or some method to kill the running scheduler?

[turbaszek]

I'm not sure why scheduler should be started out of the box. Are you sure there's no other process running scheduler on EC2?

What happens when you do docker run airflow_image webserver locally? Are there both webserver and scheduler running?