Skip to content

fix(embed): gate embed backlog on true unembedded count, not IVF index lag#73

Open
tenequm wants to merge 1 commit into
mainfrom
fix/embed-backlog-gate-after-copy
Open

fix(embed): gate embed backlog on true unembedded count, not IVF index lag#73
tenequm wants to merge 1 commit into
mainfrom
fix/embed-backlog-gate-after-copy

Conversation

@tenequm

@tenequm tenequm commented Jun 29, 2026

Copy link
Copy Markdown
Owner

Problem

After a fresh pond copy over S3, the embed stage shows a progress bar that looks permanently hung on what is actually a no-op.

Root cause: pond copy carries already-embedded rows into the destination with the vector intact (projection: None), but the IVF_SQ vector index has not folded those rows yet. The embed stage gated its backlog on unindexed_vector_backlog - the index-lag count - so it read ~104,994 "unindexed" rows while 0 were actually unembedded. The stage then spins up a progress bar sized to ~105k that can never make progress, because there is nothing to embed.

The index-lag count is a poor proxy for the embed backlog: it counts "rows the index hasn't absorbed," which after a copy is a large set of fully-embedded rows.

Approach

Gate on embed_backlog_count instead - the true eligible-unembedded count (search_text present, embedding_model null), read off the narrow embedding_model column. On the post-copy store this correctly reports 0, so the stage stays silent (the summary's indexes line confirms "semantic ready" downstream).

The narrow-column scan was the expensive thing that originally motivated the index-lag proxy, but the v0.10.1/v0.10.2 backlog work already made embed_backlog_count cheap (manifest-only / narrow-column read), so there is no longer any perf reason to gate on the coarser proxy.

swapped (model-swap detection) is unchanged - it still drives the force/bail path, the IVF drop, and the bar_total == 0 && !swapped silent-skip.

Changes

  • src/main.rs: run_embed_stage_with_limit reads store.embed_backlog_count() unconditionally; replaces the swapped ? embed_backlog_count : unindexed_vector_backlog branch.
  • src/sessions.rs: removes the now-unused Store::unindexed_vector_backlog (no other callers).

Net: +9 / -23.

Validation

  • cargo fmt --check clean
  • cargo clippy --locked -- -D warnings clean
  • cargo test --locked - 235 passed, 0 failed (169 lib + 24 bin + 42 integration), incl. embed_backlog_count_tracks_eligible_unembedded_rows and embed_worker_drains_the_backlog

…x lag

After a fresh `pond copy` over S3, the destination holds already-embedded
rows the IVF_SQ index has not folded yet (copy carries the vector via
projection: None). The embed stage gated its progress bar on
`unindexed_vector_backlog` - the index-lag count - which then read ~104,994
"unindexed" while 0 rows were actually unembedded, rendering a progress bar
that looks permanently hung on a pure no-op.

Gate on `embed_backlog_count` instead (search_text present, embedding_model
null) read off the narrow model-id column - the true eligible-unembedded
count. The 0.10.1/0.10.2 narrow-column work already made this count cheap, so
there is no perf reason to gate on the coarser index-lag proxy. Drops the now
unused `unindexed_vector_backlog`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant