diff --git a/‎CHANGELOG.md‎
Lines changed: 127 additions & 0 deletions b/‎CHANGELOG.md‎
Lines changed: 127 additions & 0 deletions
diff --git a/‎src/everos/config/__init__.py‎
Lines changed: 2 additions & 1 deletion b/‎src/everos/config/__init__.py‎
Lines changed: 2 additions & 1 deletion
diff --git a/‎src/everos/config/default.toml‎
Lines changed: 12 additions & 0 deletions b/‎src/everos/config/default.toml‎
Lines changed: 12 additions & 0 deletions
diff --git a/‎src/everos/config/settings.py‎
Lines changed: 38 additions & 0 deletions b/‎src/everos/config/settings.py‎
Lines changed: 38 additions & 0 deletions
diff --git a/‎src/everos/core/persistence/lancedb/base.py‎
Lines changed: 21 additions & 3 deletions b/‎src/everos/core/persistence/lancedb/base.py‎
Lines changed: 21 additions & 3 deletions
@@ -7,6 +7,133 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **`[cascade]` settings section** — the four maintenance cadences
+  (`optimize_heartbeat_seconds`, `optimize_prune_interval_seconds`,
+  `optimize_prune_retention_seconds`, `optimize_rebuild_interval_seconds`) are
+  now configurable. They were already constructor arguments on `CascadeWorker`,
+  but `CascadeConfig` did not carry them and no production path passed one, so
+  the defaults were unreachable — which is why the 12h rebuild sweep could not
+  be exercised by any soak run shorter than half a day. The deadlines that
+  bound a hung call are deliberately **not** exposed: they are hang-catchers
+  sized from measured durations, where too low manufactures failures on a
+  healthy table and too high leaves a wedged one invisible for longer. Note
+  `optimize_prune_retention_seconds` has a second effect worth reading before
+  tuning — it also decides how long index files keep a manifest naming them,
+  and below LanceDB's 7-day unverified window they then wait out the full 7
+  days.
+
+### Fixed
+
+- **Reads now carry a deadline** (`count` / `get_by_id` / `find_where` /
+  `find_where_paginated` / `search`). The write-side deadline work skipped them
+  on the reasoning that a read takes no lock and so blocks no writer — true, but
+  the cascade drain loop reads on every batch and advances strictly one batch at
+  a time, so a read that never returns stops the **whole md → LanceDB
+  projection**: claimed rows stay `processing` forever, nothing new is indexed,
+  and `/health` still reports healthy because a hang raises nothing. Budget 60s
+  (~1000x the measured 62ms flat scan over 117k rows); expiry raises the
+  retryable `VectorStoreBusyError`.
+- **Background loops are supervised.** The drain / heartbeat / rebuild loops were
+  plain `create_task` coroutines: one uncaught exception ended that loop
+  permanently, and because the worker holds a strong reference to the task the
+  interpreter never printed "Task exception was never retrieved" either — the
+  loop's job simply stopped happening with zero output. Each now runs under a
+  supervisor that logs and restarts with escalating backoff (5s / 15s / 45s),
+  then asks the process to exit via `SIGTERM` so a restarting supervisor
+  (systemd, Docker, k8s) can recover it. The restart budget counts consecutive
+  *quick* crashes, not crashes over the process lifetime — a body that ran 60s+
+  before raising starts a fresh incident, so independent transients days apart
+  cannot pool into a process exit (same windowed counting as systemd's
+  `StartLimitIntervalSec`). A done-callback covers the case the supervisor
+  itself ends unexpectedly.
+- **The optimize-failure alert is reachable again.** The fallback rebuild reset
+  the same counter the health verdict reads, so a table failing 100% of the time
+  cycled `1..5 → 0 → 1..` and the threshold value existed only during the
+  sub-second rebuild — roughly 1% observable against a 30s scrape, so
+  `cascade.healthy` stayed green while the table never reclaimed a version. The
+  rate limiter now lives in its own counter (`failures_since_fallback`); only a
+  successful optimize clears the alert streak. Same shape as the cross-kind
+  `max()` masking bug: a remediation path refreshing the signal meant to report
+  it.
+- **A rebuild no longer leaves the column without an FTS index.** `rebuild_indexes`
+  dropped every index and recreated it, on the assumption that LanceDB falls
+  back to a brute-force scan meanwhile. That holds for vector search and **not**
+  for FTS: with no inverted index a BM25 query raises `Cannot perform full text
+  search unless an INVERTED index has been created`, and since the recall legs
+  are gathered without `return_exceptions`, one failing leg 500s the whole
+  search request. Now uses `create_index(replace=True)`, which swaps atomically
+  — measured 0 failures across 49 queries spanning 3 replaces, versus 55
+  failures for the same test against drop-then-create — and collapses the live
+  index fragment set exactly as before (7 index files back to 4).
+- **The empty-index-dir sweep is bounded by lance's own threshold.** lance's
+  `cleanup.rs` unlinks a superseded index's files but never its directory — it
+  contains no `rmdir` at all, which is structural rather than an oversight: it
+  targets object stores, where paths are flat keys and an empty directory does
+  not exist. Only a local filesystem materialises them, and a soak run reached
+  13061 dirs, 98% empty. everos sweeps them, now with three independent
+  guarantees instead of a self-chosen age: `rmdir` cannot delete a non-empty
+  directory (the kernel refuses it, so no file can be lost and there is no
+  check-then-act window), live index UUIDs are excluded via `list_indices()`,
+  and anything else must outlive `UNVERIFIED_THRESHOLD_DAYS = 7` — lance's own
+  bound for deciding an unreferenced index UUID is dead rather than mid-build.
+  The previous 300s was our invention, which is what made it indefensible.
+  Two consequences worth knowing: the age gate reads the dir's mtime, which
+  POSIX bumps when lance's cleanup empties it, so the effective reclaim horizon
+  is up to ~14 days (file wait + age gate) and the ceiling-load steady state is
+  ~1.8M dirs / ~7GB; and a sweep that blows its 60s deadline is swallowed
+  inside `prune()` — the cleanup commit already succeeded, so escaping would
+  bill a prune "failure" (feeding the fallback-rebuild threshold) and stall the
+  prune-staleness clock for a cleanup stall that did not happen.
+- **The optimize runner's wait on an in-flight rebuild is bounded too.** The two
+  maintenance jobs park on each other — whichever arrives second waits — so an
+  unbounded wait on this side is the same hazard as the one already fixed on
+  the rebuild side, just seen from the other end: the kind's task slot stays
+  occupied, `_schedule_optimize` keeps short-circuiting on it, and that table
+  quietly stops being pruned. It was left open on the argument that
+  `rebuild_indexes` carries its own 300s deadline, which covers its critical
+  section but not the task's dispatch and teardown, so the transitive bound was
+  never real. Now bounded at 180s, logging
+  `cascade_lancedb_optimize_skipped_rebuild_unfinished` and skipping the beat
+  rather than compacting under a live rebuild — the two commit on the same
+  manifest, which is what the wait exists to prevent.
+- **A rebuild that loses a commit race is retried** instead of waiting out the
+  full 12h cadence. Lance labels the conflict `Retryable` and it is: a
+  concurrent writer in another process won the manifest, nothing is wrong with
+  the table. Retries are scheduled on the kind (10min / 30min / 3h) rather than
+  slept through, so the other kinds in the sweep are not parked behind the
+  backoff. A soak run at a 600s cadence hit 3 conflicts in 119 attempts, all
+  while a concurrent CLI storm was running.
+
+- **The index-rebuild sweep can no longer park forever** waiting on the
+  optimize runner. That wait had no deadline, and the runner's loop condition is
+  "keep going while there is unindexed data" — which under sustained writes is
+  never, since the drain loop re-raises the flag every second against a 10s
+  cooldown. Now bounded at 180s, logging
+  `cascade_lancedb_rebuild_skipped_optimize_unfinished` and skipping the sweep
+  rather than dropping indices under a live optimize. **This makes the stall
+  visible, not absent**: under sustained ingest every sweep still times out, so
+  active index-UUID / FTS `part_N` growth stays unbounded there. The functional
+  fix requires the optimize runner to yield when a rebuild is pending, which
+  changes the optimize/rebuild mutual-exclusion contract and needs its own
+  validation — the rebuild cadence is 12h, longer than any soak run so far, so
+  the periodic sweep has never been exercised under load.
+- **The memory-root lock wait is bounded and visible.** Acquisition polls with
+  `LOCK_NB` instead of blocking inside a worker thread: a blocking `flock` could
+  not be bounded or cancelled — cancelling the awaiting coroutine left the
+  thread to acquire the lock later with nobody to release it. The wait itself is
+  by design (the second process is supposed to wait, then find the migration
+  already done), but it now logs `memory_root_lock_waiting` and gives up after
+  `timeout_seconds` (default 30min) instead of leaving a server startup looking
+  like a hang whose last message is `lifespan_provider_startup name=lancedb`.
+  The default sits an order of magnitude above the worst legitimate hold (a
+  large migration is minutes) on purpose: the wait is already visible from the
+  first poll, and against the one case the bound exists for — a holder that is
+  alive but wedged — giving up at 5 minutes buys nothing over 30, while a bound
+  near the legitimate hold turns a slow migration into startup crashes for
+  every waiting process.
+
 ### Changed
 
 - **`cascade_lancedb_optimize_conflict` now records `pruned`** — which
 
@@ -4,7 +4,7 @@
     from everos.config import (
         Settings, MemorySettings, SqliteSettings, LanceDBSettings,
         LLMSettings, EmbeddingSettings, RerankSettings,
-        BoundaryDetectionSettings,
+        BoundaryDetectionSettings, CascadeSettings,
         load_settings, resolve_root,
     )
 
@@ -13,6 +13,7 @@
 """
 
 from .settings import BoundaryDetectionSettings as BoundaryDetectionSettings
+from .settings import CascadeSettings as CascadeSettings
 from .settings import EmbeddingSettings as EmbeddingSettings
 from .settings import LanceDBSettings as LanceDBSettings
 from .settings import LLMSettings as LLMSettings
 
@@ -146,6 +146,18 @@ threshold = 0.65
 time_window_days = 7.0
 
 
+[cascade]
+# Maintenance cadences (seconds). Deadlines that bound a hung call are not
+# here on purpose — they live next to the code they guard and are sized from
+# measurement, not preference.
+optimize_heartbeat_seconds = 60.0
+optimize_prune_interval_seconds = 300.0
+# Passed to LanceDB as cleanup_older_than. Only has to outlive an in-flight
+# read, but it also decides how long index files keep a manifest naming them:
+# under LanceDB's 7-day unverified window they then wait out the full 7 days.
+optimize_prune_retention_seconds = 60.0
+optimize_rebuild_interval_seconds = 43200.0
+
 [observability]
 # OpenTelemetry tracing export. Off by default; pure OTLP/HTTP, vendor-neutral
 # (Langfuse, an OTel Collector, or any OTLP backend). EverOS ships no vendor SDK.
 
@@ -338,6 +338,43 @@ class LanceDBSettings(BaseModel):
     index_cache_size_bytes: int = 16 * 1024 * 1024
 
 
+class CascadeSettings(BaseModel):
+    """Cascade maintenance cadences.
+
+    These are *how often* each background job runs, not how long it is allowed
+    to take — the deadlines that bound a hung call stay as constants next to the
+    code they guard, sized from measurement, because a wrong value there either
+    masks a hang or manufactures failures.
+
+    ``optimize_heartbeat_seconds``:
+      Idle sweep that offers every kind to the optimizer, so an unindexed tail
+      left by a crash is merged even without new writes.
+
+    ``optimize_prune_interval_seconds``:
+      How often the heavy beat runs: reclaim the files of superseded dataset
+      versions. Raise it if the write-lock hold is disruptive, lower it if disk
+      transients are.
+
+    ``optimize_prune_retention_seconds``:
+      Passed straight to LanceDB as ``cleanup_older_than`` — versions replaced
+      longer ago than this become eligible for deletion. It only has to outlive
+      an in-flight read (sub-second). Shorter shrinks the transient footprint of
+      superseded data fragments, but note it also decides how long index files
+      keep a manifest that names them: below LanceDB's 7-day unverified window,
+      index files lose that reference and wait out the full 7 days.
+
+    ``optimize_rebuild_interval_seconds``:
+      Full index rebuild per kind, which collapses the active index fragment
+      count that every ``optimize()`` grows. Bounded by rebuild cost, not
+      correctness — a missed sweep only defers cleanup.
+    """
+
+    optimize_heartbeat_seconds: float = 60.0
+    optimize_prune_interval_seconds: float = 300.0
+    optimize_prune_retention_seconds: float = 60.0
+    optimize_rebuild_interval_seconds: float = 12 * 60 * 60.0
+
+
 class KnowledgeSearchSettings(BaseModel):
     """``[knowledge.search]`` — retrieval tuning for the knowledge module."""
 
@@ -417,6 +454,7 @@ class Settings(BaseSettings):
     boundary_detection: BoundaryDetectionSettings = BoundaryDetectionSettings()
     memorize: MemorizeSettings = MemorizeSettings()
     clustering: ClusteringSettings = ClusteringSettings()
+    cascade: CascadeSettings = CascadeSettings()
     multimodal: MultimodalSettings = MultimodalSettings()
     knowledge: KnowledgeSettings = KnowledgeSettings()
     observability: ObservabilitySettings = ObservabilitySettings()
 
@@ -104,11 +104,28 @@ def to_arrow_schema(cls) -> pa.Schema:
         )
 
     @classmethod
-    async def ensure_fts_indexes(cls, table: AsyncTable) -> None:
+    async def ensure_fts_indexes(
+        cls, table: AsyncTable, *, replace: bool = False
+    ) -> None:
         """Create FTS indexes on every column in :attr:`BM25_FIELDS`.
 
         Idempotent: columns that already have an index are skipped, so
-        this is safe to call on every startup. The FTS config is fixed
+        this is safe to call on every startup.
+
+        ``replace=True`` rebuilds each column's index in place instead of
+        skipping it — used by :meth:`LanceRepoBase.rebuild_indexes`, which
+        needs a fresh index but must never leave the column *without* one.
+        Dropping first would do that, and a BM25 query in that window does not
+        degrade — it raises ``Cannot perform full text search unless an
+        INVERTED index has been created`` (measured; vector search does fall
+        back to a flat scan, FTS does not). Since the recall legs are gathered
+        without ``return_exceptions``, that window turns into a 500 on the
+        whole search request. ``create_index(replace=True)`` is atomic: 49
+        concurrent queries across 3 replaces saw 0 failures, and it collapses
+        the live fragment set exactly as drop+create does (7 index files back
+        to 4, measured).
+
+        The FTS config is fixed
         to the app-layer pre-tokenisation + LanceDB normalisation
         convention (designed for **multilingual mixed content**):
 
@@ -145,10 +162,11 @@ async def ensure_fts_indexes(cls, table: AsyncTable) -> None:
         indices = await table.list_indices()
         indexed_cols = {col for idx in indices for col in (idx.columns or [])}
         for field in cls.BM25_FIELDS:
-            if field in indexed_cols:
+            if field in indexed_cols and not replace:
                 continue
             await table.create_index(
                 column=field,
+                replace=replace,
                 config=FTS(
                     with_position=False,
                     base_tokenizer="whitespace",