Document how retry policies and durable execution work together (#73160)

Author: amoghrajesh

airflow-core/docs/core-concepts/resumable-tasks.rst

@@ -226,3 +226,42 @@ Comparison
      - Airflow 2.2
      - Airflow 3.3
      - Airflow 3.2
+
+.. _concepts-resumable-tasks-retry-policies:
+
+Retry policies and durable execution
+------------------------------------
+
+Long-running tasks typically need two separate things together: a retry
+policy, and durable execution as defined above. Each is configurable per
+task, and together they let a workflow span days without losing progress.
+
+A **retry policy** decides whether a failed attempt gets another try, and
+how long to wait before it. This is :class:`~airflow.sdk.RetryPolicy` and
+:class:`~airflow.sdk.RetryDecision` (see :ref:`concepts:retry-policies`), or
+the LLM-driven
+:class:`~airflow.providers.common.ai.policies.retry.LLMRetryPolicy`, which
+uses a model to read the error and make that call. That policy adds its own
+``fallback_rules``, applied when the classification call itself fails, so
+the decision does not depend on the model being reachable (see
+:doc:`apache-airflow-providers-common-ai:retry_policies`).
+
+**Durable execution** is what that next attempt resumes from. Backed by the
+task state store described above, it is what lets a task recover a
+checkpoint written by the attempt before it, rather than starting over.
+
+Long-running tasks, or LLM-driven tasks like agentic workflows, benefit
+most from both: the retry policy decides whether the error is worth
+retrying at all, and the task state store is what the retry resumes from.
+
+Consider using both when a task:
+
+* Runs long enough that a worker crash mid-run is a real risk, and
+* Needs a retry decision more nuanced than "always retry" or "never
+  retry", for example retrying rate limits but failing outright on bad
+  credentials.
+
+The two operate independently. A task with ``retries=5`` and a checkpoint
+still stops for good after six failed attempts (the initial attempt plus five
+retries), but each of those retries picks up from the last checkpoint instead of
+reprocessing files it already finished, or resubmitting a job that is still running.

airflow-core/docs/core-concepts/tasks.rst

@@ -314,6 +314,14 @@ DAG was triggered:
                 return RetryDecision.fail(reason="Backfill run -- not retrying")
             return RetryDecision.default()
 
+A retry policy only decides whether and when a task gets another attempt. It
+receives the full run ``context``, so it can inspect checkpointed progress
+through ``context["task_state_store"]`` when making that decision, but it
+never persists or restores that progress itself -- resuming from a checkpoint
+is the task's own work. For a task that also needs to resume from where it
+left off, pair a retry policy with the task state store described in
+:ref:`concepts-resumable-tasks-retry-policies`.
+
 .. _concepts:task-instance-heartbeat-timeout:
 
 Task Instance Heartbeat Timeout

providers/common/ai/docs/retry_policies.rst

@@ -26,6 +26,8 @@ retry decisions. It works with any LLM provider supported by pydantic-ai
 (OpenAI, Anthropic, Bedrock, Vertex, Ollama, etc.).
 
 For the core retry policy concepts, see :doc:`apache-airflow:core-concepts/tasks`.
+If the task also needs to survive a worker crash without losing its progress,
+see :ref:`apache-airflow:concepts-resumable-tasks-retry-policies`.
 
 Setup
 -----