When the embedding provider rate-limits requests, 1.1-beta2 drains the queue 15 to 25 times slower than 1.0 did for the same documents. A job that took about 8 minutes on 1.0 now takes 36 to 50 minutes. Three problems in the retry path cause this, and a fourth change is what made them start showing up.
Steps to reproduce:
- Point the vectorizer at a rate-limited provider (Voyage free tier: 3 RPM, 10K TPM).
- Set batch_size=25 and max_retries=10.
- Queue about 47 chunks (roughly 400 tokens each) in one table.
- Enable vectorization and poll the queue every 20 seconds: SELECT status, count(*) FROM pgedge_vectorizer.queue GROUP BY status.
Expected: The queue drains in a few minutes. After a 429, the pending items retry together as a batch, spaced out by the provider's Retry-After hint (seconds).
Actual: The first batch of 25 embeds right away. The remaining 22 then drain 3 at a time per minute, and the wait between each group grows: 1 minute, then 2, then 3, then 4, then 11. Total is 36 to 50 minutes. Nothing ends up failing. It just crawls.
Root cause:
Retried items are no longer batched (src/worker.c:1553-1556):
/* If any items have been retried, process individually to isolate failures */
if (has_retries && n_items > 1)
effective_batch_size = 1;
Once a rate limit pushes any item to attempts > 0, every following request carries a single 400-token chunk against a 10,000-token budget. Throughput drops from 25 chunks per request to 1, so it hits the 3 RPM wall immediately.
The retry wait grows without limit and ignores Retry-After (src/worker.c:1885):
SET status = 'pending', attempts = attempts + 1,
next_retry_at = NOW() + (attempts + 1) * INTERVAL '1 minute'
A rate limit that clears in seconds forces waits of 1, 2, 3, up to 11 minutes. The provider's status code and Retry-After header are thrown away: provider_common.c:504-509 turns every non-200 response into a plain text message and loses the 429.
A 429 spends a retry attempt. A temporary rate limit is counted against the item's max_retries the same as a real error, so ordinary throttling burns through the retry budget and drives the growing backoff in #2.
What made this surface: max_retries went from ignored to honored (sql/pgedge_vectorizer--1.1.sql:441-451, issue #26). In 1.0 the setting did nothing, so every item used the column default of 3. In 1.1 it takes effect. A config with max_retries=10 (which did nothing on 1.0) now means each throttled item retries up to 10 times through the slow path above. That is why the slowdown appeared right after upgrading, with no config change.
What actually happens in 1.1-beta2
START: 47 chunks pending, all with attempts = 0
┌─ Minute 0 ────────────────────────────────────────────────┐
│ Worker pulls a batch of 25 (all fresh) │
│ -> 1 request, 25 chunks x 400 = 10,000 tokens │
│ -> Voyage: 200 OK ✓ (exactly fills the 10K/min budget) │
│ RESULT: 25 embedded, 22 still pending │
│ │
│ Worker pulls the next batch: 22 remaining │
│ -> 1 request, ~8,800 tokens │
│ -> but the 10K/min budget for this minute is gone │
│ -> Voyage: 429 Too Many Requests ✗ │
└────────────────────────────────────────────────────────────┘
│
▼
The 429 is where the bug kicks in
│
┌─────────────────┴─────────────────┐
│ Bug 1: those 22 items get marked │
│ "retry", and now have attempts > 0 │
└─────────────────┬─────────────────┘
▼
┌─────────────────────────────────────┐
│ Bug 2: each one gets │
│ next_retry_at = now + (attempts+1) │
│ minutes (1 min, then 2, then 3...) │
└─────────────────┬───────────────────┘
▼
┌─────────────────────────────────────┐
│ Bug 3: the 429 counts as a failed │
│ attempt, eating into max_retries │
└─────────────────┬───────────────────┘
▼
┌─ From now on, because attempts > 0 ────────────────────────┐
│ Bug 1 again: the worker stops batching and sends │
│ ONE chunk per request │
│ │
│ Minute ~1: 3 single-chunk requests succeed (the 3/min │
│ limit), 4th gets 429 -> those items pushed out │
│ RESULT: +3 embedded │
│ Minute ~2: 3 more succeed, gap now larger │
│ Minute ~3: +3 │
│ Minute ~4: +3 │
│ Minute ~11: +3 (the per-item wait keeps growing) │
│ ... │
└────────────────────────────────────────────────────────────┘
│
▼
22 leftover chunks drip out 3 at a time over 30+ minutes
TOTAL: 36 to 50 minutes
When the embedding provider rate-limits requests, 1.1-beta2 drains the queue 15 to 25 times slower than 1.0 did for the same documents. A job that took about 8 minutes on 1.0 now takes 36 to 50 minutes. Three problems in the retry path cause this, and a fourth change is what made them start showing up.
Steps to reproduce:
Expected: The queue drains in a few minutes. After a 429, the pending items retry together as a batch, spaced out by the provider's Retry-After hint (seconds).
Actual: The first batch of 25 embeds right away. The remaining 22 then drain 3 at a time per minute, and the wait between each group grows: 1 minute, then 2, then 3, then 4, then 11. Total is 36 to 50 minutes. Nothing ends up failing. It just crawls.
Root cause:
Retried items are no longer batched (src/worker.c:1553-1556):
/* If any items have been retried, process individually to isolate failures */
if (has_retries && n_items > 1)
effective_batch_size = 1;
Once a rate limit pushes any item to attempts > 0, every following request carries a single 400-token chunk against a 10,000-token budget. Throughput drops from 25 chunks per request to 1, so it hits the 3 RPM wall immediately.
The retry wait grows without limit and ignores Retry-After (src/worker.c:1885):
SET status = 'pending', attempts = attempts + 1,
next_retry_at = NOW() + (attempts + 1) * INTERVAL '1 minute'
A rate limit that clears in seconds forces waits of 1, 2, 3, up to 11 minutes. The provider's status code and Retry-After header are thrown away: provider_common.c:504-509 turns every non-200 response into a plain text message and loses the 429.
A 429 spends a retry attempt. A temporary rate limit is counted against the item's max_retries the same as a real error, so ordinary throttling burns through the retry budget and drives the growing backoff in #2.
What made this surface: max_retries went from ignored to honored (sql/pgedge_vectorizer--1.1.sql:441-451, issue #26). In 1.0 the setting did nothing, so every item used the column default of 3. In 1.1 it takes effect. A config with max_retries=10 (which did nothing on 1.0) now means each throttled item retries up to 10 times through the slow path above. That is why the slowdown appeared right after upgrading, with no config change.
What actually happens in 1.1-beta2
START: 47 chunks pending, all with attempts = 0
┌─ Minute 0 ────────────────────────────────────────────────┐
│ Worker pulls a batch of 25 (all fresh) │
│ -> 1 request, 25 chunks x 400 = 10,000 tokens │
│ -> Voyage: 200 OK ✓ (exactly fills the 10K/min budget) │
│ RESULT: 25 embedded, 22 still pending │
│ │
│ Worker pulls the next batch: 22 remaining │
│ -> 1 request, ~8,800 tokens │
│ -> but the 10K/min budget for this minute is gone │
│ -> Voyage: 429 Too Many Requests ✗ │
└────────────────────────────────────────────────────────────┘
│
▼
The 429 is where the bug kicks in
│
┌─────────────────┴─────────────────┐
│ Bug 1: those 22 items get marked │
│ "retry", and now have attempts > 0 │
└─────────────────┬─────────────────┘
▼
┌─────────────────────────────────────┐
│ Bug 2: each one gets │
│ next_retry_at = now + (attempts+1) │
│ minutes (1 min, then 2, then 3...) │
└─────────────────┬───────────────────┘
▼
┌─────────────────────────────────────┐
│ Bug 3: the 429 counts as a failed │
│ attempt, eating into max_retries │
└─────────────────┬───────────────────┘
▼
┌─ From now on, because attempts > 0 ────────────────────────┐
│ Bug 1 again: the worker stops batching and sends │
│ ONE chunk per request │
│ │
│ Minute ~1: 3 single-chunk requests succeed (the 3/min │
│ limit), 4th gets 429 -> those items pushed out │
│ RESULT: +3 embedded │
│ Minute ~2: 3 more succeed, gap now larger │
│ Minute ~3: +3 │
│ Minute ~4: +3 │
│ Minute ~11: +3 (the per-item wait keeps growing) │
│ ... │
└────────────────────────────────────────────────────────────┘
│
▼
22 leftover chunks drip out 3 at a time over 30+ minutes
TOTAL: 36 to 50 minutes