Scheduler has to be restarted every few days

[hackmad]

This seemed to have occurred on October 8th and then on October 23rd, 2018. We start seeing messages like this in the logs:

When investigating the ECS instance, seems like s3fs has also failed on one of the instances:

$ mount -l | grep airflow
s3fs on /mnt/prod-airflow-ecs type fuse.s3fs (rw,noatime,user_id=0,group_id=0,allow_other)
$ sudo ls /mnt/prod-airflow-ecs/
ls: cannot access /mnt/prod-airflow-ecs/: Transport endpoint is not connected

The second instance is fine:

[ec2-user@ip-10-0-10-132 ~]$ mount -l | grep airflow
s3fs on /mnt/prod-airflow-ecs type fuse.s3fs (rw,noatime,user_id=0,group_id=0,allow_other)
[ec2-user@ip-10-0-10-132 ~]$ sudo ls /mnt/prod-airflow-ecs/
dags  plugins

Currently we don't have monitoring to detect this and the only recourse is to stop the Scheduler task in ECS and it bounces back and runs the scheduled jobs it missed. If the scheduler task restart doesn't work, terminating the ECS instances helps but it does take time to failover and restart all the instances.

The first occurrence on October 8th was due to issues with the docker volume mount but it wasn't investigated in-depth at the time.

[hackmad]

One solution proposed by @colin-streicher is to get rid of s3fs and the docker volume mount and replace it with a sidecar that uses git-sync to the repo containing the DAGs and plugins.