🧠 Smart LLM Retry for Apache Airflow

A lightweight, explainable retry controller for Airflow tasks using a local Ollama LLM (no cloud calls, no external APIs).

---

🚀 Overview

Traditional Airflow retries don’t “know” why a task failed — they’ll blindly retry even for permanent errors like “Table Not Found”.
This plugin introduces intelligent, context-aware retries powered by a local LLM (via Ollama).

When a task fails, Airflow:

1. Triggers a custom retry callback
2. Sends the task’s error message + context to your local LLM
3. The LLM decides:
   • ✅ retry – transient error (network, timeout, etc.)
   • ❌ no_retry – deterministic error (syntax, missing table)
4. If the LLM decides no_retry, the callback immediately sets the task state to FAILED, preventing further retries.

---

🧩 Architecture

┌───────────────────────────┐
│       Airflow Task        │
│         (fails)            │
└─────────────┬──────────────┘
              │ on_retry_callback
              ▼
      ┌────────────────────┐
      │ Smart Retry Logic  │
      │  (Python callback) │
      └─────────┬──────────┘
                │
                ▼
     ┌─────────────────────┐
     │  Local Ollama LLM   │
     │ (llama3 / mistral)  │
     └─────────┬───────────┘
               │ JSON decision
               ▼
     ┌────────────────────────────┐
     │ Update task state → FAILED │
     │ or allow normal retry      │
     └────────────────────────────┘

🧠 Prerequisites

• Apache Airflow ≥ 2.6
• Python ≥ 3.8
• Ollama (local or remote)

  ollama pull llama3.1:8b-instruct
  ollama serve

---

🛠️ Quickstart

1. Build Docker image:

   docker build . --tag airflow_local:latest

2. Initialize Airflow with Docker Compose:

   docker compose up airflow-init

3. Start Airflow services:

   docker compose up -d

4. Stop services:

   docker compose down