fix: retry transient HTTP errors (529/502/503/504) without --wait

CLAUDE.md

@@ -282,8 +282,8 @@ Configurable patterns detect rate limit and quota errors in claude/codex output:
 - On match, ralphex exits gracefully with pattern info and help command suggestion
 
 Transient retry patterns for wrapper-level stalls:
-- `claude_retry_patterns`: comma-separated transient Claude/fya markers retried like executor timeouts. Default: `FYA_TRANSIENT_TIMEOUT`
-- Retry patterns are checked before limit and error patterns. They do not use `wait_on_limit`; the phase receives timeout-style metadata and applies its existing bounded retry behavior
+- `claude_retry_patterns`: comma-separated transient Claude/fya markers retried like executor timeouts. Default: `FYA_TRANSIENT_TIMEOUT,API Error: 529,API Error: 502,API Error: 503,API Error: 504`. The transient HTTP errors (529 Overloaded, 502/503/504 gateway) live here, not in `claude_limit_patterns`: they are short-lived server hiccups, so they auto-retry without requiring `--wait` (500 is intentionally excluded — it can be a deterministic server failure, caught by the broad `API Error:` error pattern). Precedence is retry → limit → error, so a no-signal 529 matches the retry tier first; a 529 with a signal present falls through to the broad `API Error:` error pattern
+- Retry patterns are checked before limit and error patterns. They do not use `wait_on_limit`; the phase receives timeout-style metadata and applies its existing bounded retry behavior. The task loop (`pkg/processor/phase/task.go`) and review-iteration loop (`pkg/processor/phase/review.go`) wait a short fixed `retryBackoff` (5s, defined in `pkg/processor/phase/phase.go`) before re-running the timed-out/transiently-failed iteration; the first-review soft-skip path is not a retry and has no backoff
 - Retry detection is suppressed when `result.Signal` is non-empty: a completed run that emitted a structured signal (e.g. `ALL_TASKS_DONE`) must not be discarded and re-run just because the output text mentions a retry marker. `patternError(recentText, signal)` (`pkg/executor/executor.go`) gates only the retry tier on the signal; limit and error patterns still fire regardless (they surface loudly rather than silently re-running)
 
 Limit patterns for wait+retry behavior:

README.md

@@ -969,9 +969,9 @@ Provider-related CLI flags (`--claude-command`, `--claude-args`, `--external-rev
 | `color_info` | Informational messages color (hex) | `#b4b4b4` |
 | `claude_error_patterns` | Patterns to detect in claude output (comma-separated) | `You've hit your limit,You've hit your session limit,API Error:,cannot be launched inside another Claude Code session,Not logged in,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit` |
 | `codex_error_patterns` | Patterns to detect in codex output (comma-separated) | `Rate limit exceeded,rate limit reached,429 Too Many Requests,quota exceeded,insufficient_quota,You've hit your usage limit` |
-| `claude_limit_patterns` | Limit patterns for claude triggering wait+retry (comma-separated) | `You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit,API Error: 529,API Error: 502,API Error: 503,API Error: 504` |
+| `claude_limit_patterns` | Limit patterns for claude triggering wait+retry (comma-separated) | `You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit` |
 | `codex_limit_patterns` | Limit patterns for codex triggering wait+retry (comma-separated) | `Rate limit exceeded,rate limit reached,429 Too Many Requests,quota exceeded,insufficient_quota,You've hit your usage limit` |
-| `claude_retry_patterns` | Transient claude/fya markers retried like executor timeouts (comma-separated) | `FYA_TRANSIENT_TIMEOUT` |
+| `claude_retry_patterns` | Transient claude/fya markers retried like executor timeouts (comma-separated) | `FYA_TRANSIENT_TIMEOUT,API Error: 529,API Error: 502,API Error: 503,API Error: 504` |
 | `wait_on_limit` | Wait duration before retrying on rate limit (e.g., `1h`, `30m`) | disabled |
 | `session_timeout` | Per-session timeout for task/review executor (e.g., `30m`, `1h`). Applies to Claude calls in default executor mode and every executor call under `executor = codex`; external codex/custom review in Claude mode is not affected | disabled |
 | `idle_timeout` | Kill executor session when no output for specified duration (e.g., `5m`). Resets on each output line. Applies to the claude executor in default mode and to every executor call under `--codex`; external codex review in default-claude mode is NOT affected (preserves master behavior). Custom review is also not affected | disabled |
@@ -980,7 +980,7 @@ Colors use 24-bit RGB (true color), supported natively by all modern terminals (
 
 Error patterns use case-insensitive substring matching. When a pattern is detected in claude or codex output, ralphex exits gracefully with an informative message suggesting how to check usage/status. Multiple patterns are separated by commas, with whitespace trimmed from each pattern.
 
-**Transient retry:** Claude retry patterns (`claude_retry_patterns`) are checked before limit and error patterns. They are for wrapper-level stalls such as fya's `FYA_TRANSIENT_TIMEOUT`; matches are retried through the existing timeout-style phase path and do not use `wait_on_limit`.
+**Transient retry:** Claude retry patterns (`claude_retry_patterns`) are checked before limit and error patterns. They cover wrapper-level stalls such as fya's `FYA_TRANSIENT_TIMEOUT` and transient server-side HTTP errors (`API Error: 529` Overloaded and the `502`/`503`/`504` gateway errors); matches are retried through the existing timeout-style phase path and do not use `wait_on_limit`, so they recover automatically without `--wait`. The task and review retry loops wait a short fixed backoff (5s) before re-running the failed iteration. `API Error: 500` is intentionally excluded — it can be a deterministic server failure and is caught by the broad `API Error:` error pattern instead.
 
 **Rate limit retry:** Limit patterns (`claude_limit_patterns`, `codex_limit_patterns`) work similarly but support optional wait+retry behavior. When `--wait` is set (or `wait_on_limit` in config), a limit pattern match triggers a wait followed by automatic retry instead of exiting. Without `--wait`, limit patterns fall through to error pattern behavior. Limit patterns are checked before error patterns — if the same string matches both, the limit pattern takes priority when wait is enabled.
 

llms.txt

@@ -188,9 +188,9 @@ Requirements: codex CLI ≥ 0.130.0. Older versions silently ignore unknown `-c`
 
 **Idle timeout:** `--idle-timeout` flag (or `idle_timeout` config option) kills executor sessions when no output is received for a specified duration. Unlike session timeout (fixed wall-clock limit), idle timeout resets on each output line and only fires when the session goes silent. Useful for detecting hung sessions that completed work but didn't exit. Applies to the claude executor in default mode and to every executor call under `--codex` (task/review/finalize); external codex review in default-claude mode is NOT affected — that path keeps master semantics so users with `idle_timeout` set for claude don't see new early-terminations on the external review phase. Custom external review is also not affected. Disabled by default.
 
-**Transient retry:** `claude_retry_patterns` config option detects transient claude/fya markers and retries them through the existing timeout-style phase path. Default: `FYA_TRANSIENT_TIMEOUT`. Retry patterns are checked before limit and error patterns and do not use `wait_on_limit`.
+**Transient retry:** `claude_retry_patterns` config option detects transient claude/fya markers and retries them through the existing timeout-style phase path. Default: `FYA_TRANSIENT_TIMEOUT,API Error: 529,API Error: 502,API Error: 503,API Error: 504`. The transient HTTP errors (529 Overloaded and the 502/503/504 gateway errors) are auto-retried here rather than gated behind `--wait`, since they are short-lived server hiccups, not account-quota limits. Retry patterns are checked before limit and error patterns and do not use `wait_on_limit`. The task and review retry loops wait a short fixed backoff (5s) before re-running a timed-out or transiently-failed iteration.
 
-**Rate limit retry:** `--wait` flag (or `wait_on_limit` config option) enables automatic retry when rate limits are detected. Limit patterns (`claude_limit_patterns`, `codex_limit_patterns`) are checked before error patterns — when a limit pattern matches and wait is configured, ralphex waits the specified duration and retries. Without `--wait`, limit matches fall through to error pattern behavior (exit). Default limit patterns: `You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit,API Error: 529,API Error: 502,API Error: 503,API Error: 504` (claude), `Rate limit exceeded,rate limit reached,429 Too Many Requests,quota exceeded,insufficient_quota,You've hit your usage limit` (codex). The codex defaults are tightened so that review findings that *talk about* rate limiting in a codebase do not trip a false positive. Users who customized `codex_limit_patterns` or `codex_error_patterns` to an earlier default (e.g. `Rate limit,quota exceeded` or `Rate limit,quota exceeded,You've hit your usage limit`) keep their customization on update — comment the line out to inherit the new embedded default. Pattern matching scans both stdout and stderr (live, untruncated) so detection survives the 5-line / 256-rune error-context tail.
+**Rate limit retry:** `--wait` flag (or `wait_on_limit` config option) enables automatic retry when rate limits are detected. Limit patterns (`claude_limit_patterns`, `codex_limit_patterns`) are checked before error patterns — when a limit pattern matches and wait is configured, ralphex waits the specified duration and retries. Without `--wait`, limit matches fall through to error pattern behavior (exit). Default limit patterns: `You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit` (claude), `Rate limit exceeded,rate limit reached,429 Too Many Requests,quota exceeded,insufficient_quota,You've hit your usage limit` (codex). The transient HTTP errors `API Error: 529/502/503/504` are no longer in the claude limit set — they moved to `claude_retry_patterns` so they auto-retry without `--wait`. The codex defaults are tightened so that review findings that *talk about* rate limiting in a codebase do not trip a false positive. Users who customized `codex_limit_patterns` or `codex_error_patterns` to an earlier default (e.g. `Rate limit,quota exceeded` or `Rate limit,quota exceeded,You've hit your usage limit`) keep their customization on update — comment the line out to inherit the new embedded default. Pattern matching scans both stdout and stderr (live, untruncated) so detection survives the 5-line / 256-rune error-context tail.
 
 **Plan move behavior:** `move_plan_on_completion` config option controls whether completed plans move to `docs/plans/completed/` on success. Default `true` (existing behavior). Set to `false` for workflows that manage plan file lifecycle externally, such as spec-driven tooling with separate archive steps.
 

pkg/config/defaults/config

@@ -264,8 +264,8 @@ codex_error_patterns = Rate limit exceeded,rate limit reached,429 Too Many Reque
 
 # claude_retry_patterns: transient claude/fya output markers to retry like timeouts
 # comma-separated list of substrings (case-insensitive matching)
-# default: FYA_TRANSIENT_TIMEOUT
-claude_retry_patterns = FYA_TRANSIENT_TIMEOUT
+# default: FYA_TRANSIENT_TIMEOUT,API Error: 529,API Error: 502,API Error: 503,API Error: 504
+claude_retry_patterns = FYA_TRANSIENT_TIMEOUT,API Error: 529,API Error: 502,API Error: 503,API Error: 504
 
 # ------------------------------------------------------------------------------
 # rate limit wait and retry
@@ -275,8 +275,9 @@ claude_retry_patterns = FYA_TRANSIENT_TIMEOUT
 # comma-separated list of substrings (case-insensitive matching)
 # when detected AND wait_on_limit is set, ralphex waits and retries
 # when detected WITHOUT wait_on_limit, falls through to error pattern behavior
-# default: You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit,API Error: 529,API Error: 502,API Error: 503,API Error: 504
-claude_limit_patterns = You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit,API Error: 529,API Error: 502,API Error: 503,API Error: 504
+# transient HTTP errors (529/502/503/504) live in claude_retry_patterns, not here
+# default: You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit
+claude_limit_patterns = You've hit your limit,You've hit your session limit,Your usage allocation has been disabled by your admin,You've hit your org's monthly usage limit
 
 # codex_limit_patterns: patterns to detect rate limits in codex/external output
 # comma-separated list of substrings (case-insensitive matching)

pkg/config/values_test.go

@@ -72,9 +72,9 @@ func TestValuesLoader_Load_EmbeddedOnly(t *testing.T) {
 	assert.Empty(t, values.CommitTrailer)
 	assert.Equal(t, []string{"You've hit your limit", "You've hit your session limit", "API Error:", "cannot be launched inside another Claude Code session", "Not logged in", "Your usage allocation has been disabled by your admin", "You've hit your org's monthly usage limit"}, values.ClaudeErrorPatterns)
 	assert.Equal(t, []string{"Rate limit exceeded", "rate limit reached", "429 Too Many Requests", "quota exceeded", "insufficient_quota", "You've hit your usage limit"}, values.CodexErrorPatterns)
-	assert.Equal(t, []string{"You've hit your limit", "You've hit your session limit", "Your usage allocation has been disabled by your admin", "You've hit your org's monthly usage limit", "API Error: 529", "API Error: 502", "API Error: 503", "API Error: 504"}, values.ClaudeLimitPatterns)
+	assert.Equal(t, []string{"You've hit your limit", "You've hit your session limit", "Your usage allocation has been disabled by your admin", "You've hit your org's monthly usage limit"}, values.ClaudeLimitPatterns)
 	assert.Equal(t, []string{"Rate limit exceeded", "rate limit reached", "429 Too Many Requests", "quota exceeded", "insufficient_quota", "You've hit your usage limit"}, values.CodexLimitPatterns)
-	assert.Equal(t, []string{"FYA_TRANSIENT_TIMEOUT"}, values.ClaudeRetryPatterns)
+	assert.Equal(t, []string{"FYA_TRANSIENT_TIMEOUT", "API Error: 529", "API Error: 502", "API Error: 503", "API Error: 504"}, values.ClaudeRetryPatterns)
 	assert.Zero(t, values.WaitOnLimit)
 	assert.False(t, values.WaitOnLimitSet)
 }

pkg/processor/phase/phase.go

@@ -21,6 +21,10 @@ const (
 	planIterationDivisor   = 5
 )
 
+// retryBackoff is the pause before re-running an iteration that timed out or hit
+// a transient retry pattern, applied in the task and review retry loops.
+const retryBackoff = 5 * time.Second
+
 // Config contains the runner settings consumed by phase engines.
 type Config struct {
 	PlanDescription       string

pkg/processor/phase/review.go

@@ -85,7 +85,10 @@ func (p *ReviewPhase) Loop(ctx context.Context, prefix string) error {
 		}
 
 		if execResult.TimedOut {
-			p.log.Print("session timed out, retrying review iteration...")
+			p.log.Print("session timed out, retrying review iteration after %s...", retryBackoff)
+			if err := p.policy.Sleep(ctx, retryBackoff); err != nil {
+				return fmt.Errorf("interrupted: %w", err)
+			}
 			continue
 		}
 

pkg/processor/phase/review_test.go

@@ -4,6 +4,7 @@ import (
 	"errors"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -170,16 +171,18 @@ func TestReviewPhase_Loop_TimeoutContinues(t *testing.T) {
 	log := newMockLogger("progress.txt")
 	exec := newTaskPhaseMockExecutor(nil)
 	phase, _ := reviewPhaseFromRunner(t, reviewPhaseTestOpts{cfg: Config{MaxIterations: 50}, exec: exec, log: log})
-	phase.policy = newScriptedTestPolicy(log,
+	policy := newScriptedTestPolicy(log,
 		ExecutionResult{TimedOut: true},
 		ExecutionResult{Result: executor.Result{Output: "review done", Signal: status.ReviewDone}},
 	)
+	phase.policy = policy
 
 	err := phase.Loop(t.Context(), "")
 
 	require.NoError(t, err)
 	assert.Len(t, exec.RunCalls(), 2)
 	assertLogContains(t, log, "retrying review iteration")
+	assert.Equal(t, []time.Duration{retryBackoff}, policy.sleepCalls, "timeout retry waits the backoff once")
 }
 
 func TestReviewPhase_First_CodexTimeoutSurfacesAsError(t *testing.T) {

pkg/processor/phase/task.go

@@ -96,7 +96,10 @@ func (p *TaskPhase) Run(ctx context.Context) error {
 		}
 
 		if execResult.TimedOut {
-			p.log.Print("%s session timed out, retrying task iteration...", execName)
+			p.log.Print("%s session timed out, retrying task iteration after %s...", execName, retryBackoff)
+			if err := p.policy.Sleep(ctx, retryBackoff); err != nil {
+				return fmt.Errorf("interrupted: %w", err)
+			}
 			continue
 		}
 

pkg/processor/phase/task_test.go

@@ -293,7 +293,7 @@ func TestTaskPhase_Run_BreakAbort(t *testing.T) {
 	assert.Len(t, exec.RunCalls(), 1)
 }
 
-func TestTaskPhase_Run_TimeoutContinuesWithoutSleeping(t *testing.T) {
+func TestTaskPhase_Run_TimeoutBacksOffThenContinues(t *testing.T) {
 	planFile := writeTaskPhasePlan(t, "# Plan\n### Task 1: first\n- [x] done")
 	exec := newTaskPhaseMockExecutor(nil)
 	phase := taskPhaseFromRunner(t, taskPhaseTestOpts{
@@ -312,7 +312,7 @@ func TestTaskPhase_Run_TimeoutContinuesWithoutSleeping(t *testing.T) {
 
 	require.NoError(t, err)
 	assert.Len(t, exec.RunCalls(), 2)
-	assert.Empty(t, policy.sleepCalls)
+	assert.Equal(t, []time.Duration{retryBackoff}, policy.sleepCalls, "timeout retry waits the backoff once")
 }
 
 func writeTaskPhasePlan(t *testing.T, content string) string {