Airflow Schedule Insights

Airflow Schedule Insights is an Apache Airflow 3.1 plugin that visualizes how DAG schedules are distributed across the day and helps teams identify quieter scheduling windows.

It acts as a schedule observability dashboard and a first step toward a Smart Scheduling Assistant for Airflow.

---

Demo

Features

• Schedule Heatmap
  • Weekday × hour distribution of projected DAG runs

• Scheduler Load Curve
  • Understand hourly scheduler pressure

• Hourly Distribution

  • See total projected runs per hour

  

• Daily Scheduler Load

  • Highlights free hours

  • Recommends low‑load windows

    

• Planning Window Selector

  • 7 / 14 / 30 day projections

    

---

How it Works

The plugin analyzes DAG schedules and projects their future execution windows.

Backend Flow

1. Load DAG definitions using DagBag
2. Read each DAG's timetable
3. Project future DAG runs for a selected window (7/14/30 days)
4. Aggregate runs into:
   • weekday × hour heatmap
   • hourly load totals
5. Identify:
   • peak hours
   • free hours
   • recommended low‑load windows

The backend exposes a JSON API consumed by the dashboard.

Architecture

Below is the high-level architecture of Airflow Schedule Insights.

flowchart LR

    subgraph Airflow["Apache Airflow 3.x"]
        NAV["Sidebar Navigation<br/>Schedule Insights"]
        PLUGIN["Airflow Plugin"]
        DAGBAG["DagBag<br/>Load DAG Definitions"]
        TIMETABLE["Airflow Timetables<br/>Schedule Logic"]
    end

    subgraph Backend["Plugin Backend (FastAPI)"]
        API["API Endpoint<br/>/smart-scheduling/heatmap"]
        SERVICE["Schedule Service"]
        PROJECTION["Run Projection Engine"]
        AGG["Aggregation Layer"]
    end

    subgraph Frontend["React Dashboard"]
        UI["Schedule Insights UI"]
        HEATMAP["Schedule Heatmap"]
        CURVE["Scheduler Load Curve"]
        HIST["Hourly Distribution"]
        WINDOWS["Daily Scheduler Load<br/>Free Windows"]
    end

    NAV --> UI
    UI -->|REST API| API

    API --> SERVICE
    SERVICE --> DAGBAG
    SERVICE --> TIMETABLE

    DAGBAG --> PROJECTION
    TIMETABLE --> PROJECTION

    PROJECTION --> AGG

    AGG --> HEATMAP
    AGG --> CURVE
    AGG --> HIST
    AGG --> WINDOWS

Loading

Project Structure

airflow_schedule_heatmap_plugin/
│
├── airflow_plugin.py
├── api.py
├── schedule_service.py
├── README.md
│
└── dist/
    ├── index.html
    └── assets/

dist/ contains the compiled React frontend.

---

Installation

Currently the plugin is installed by copying it into the Airflow plugins directory.

Example:

cp -R airflow_schedule_heatmap_plugin /opt/airflow/plugins/

Restart Airflow after copying.

docker compose restart

Then open:

http://localhost:8080/smart-scheduling/ui/

---

Frontend Development

The dashboard is a standalone React app.

Run locally

npm install
npm run dev

Important configuration

Set the correct base path in vite.config.js:

base: "/smart-scheduling/ui/"

Build frontend

npm run build

Copy build output into the plugin:

cp -R dist/* /opt/airflow/plugins/airflow_schedule_heatmap_plugin/dist/

Restart Airflow afterwards.

---

Backend Development

Main logic lives in:

schedule_service.py

Typical changes:

• modify schedule projection
• add new metrics
• improve recommendation logic
• add team/tag filters

API routes are defined in:

api.py

---

Local Development with Docker Airflow

Typical workflow using your Makefile.

Build Airflow image

make airflow-build

Initialize Airflow

make airflow-init

Start Airflow

make airflow-up

View logs

make airflow-logs

Restart after plugin changes

make airflow-restart

---

Testing

Test backend API directly:

http://localhost:8080/smart-scheduling/heatmap?days=14

Expected response:

{
  "window_days": 14,
  "total_dags": 301,
  "projected_runs": 5078,
  "heatmap": []
}

Open the dashboard:

http://localhost:8080/smart-scheduling/ui/

---

Troubleshooting

Blank page

Ensure frontend build exists in:

dist/index.html
dist/assets/

MIME errors

Fix Vite base path:

base: "/smart-scheduling/ui/"

Rebuild frontend.

Changes not visible

Restart Airflow after:

• Python changes
• copying frontend files

---

Future Enhancements

Possible improvements:

• tag / team filters
• schedule collision detection
• cron schedule suggestions
• scheduler pressure score
• AI generated schedule insights

---

Summary

Airflow Schedule Insights helps data platform teams understand scheduler load and place new DAGs intelligently.

It provides visibility into:

• schedule clustering
• peak scheduler hours
• low‑load scheduling windows

and acts as the foundation for a future Airflow Smart Scheduling Assistant.