Event pruning stalls on busy workspaces because it waits for the whole workspace to be idle

[brsbl]

Summary

bb's event cleanup (thread pruning) only runs when the entire workspace is idle: no thread is running or starting, no interaction is waiting on the user, and no project deletion or environment provisioning is in progress. On a workspace where some thread is almost always busy or waiting on a question, the cleanup rarely gets to run, so rows it would normally delete pile up in events. The expected behavior is that cleanup keeps making progress while you work and skips only the threads that are busy.

Versions and environment

• bb server 0.45.1-nightly.37211842106.1 (built from a2bbfaadaa), self-hosted on a 4-core / 8 GB Linux VM, reached through bb Connect.
• The database is 11.6 GB. events takes 8.7 GB, plus about 1.5 GB of indexes on it, across about 4,600 threads.
• Code checked at main df913a79dc.

Steps to reproduce

1. Keep at least one thread running, or one thread waiting on a permission or question, for an extended period, which is typical with several agents.
2. Let thread-event-pruning run on its 10-second interval.
3. At debug log level, each run logs Thread pruning skipped while app work is active and advances nothing.

Expected vs actual

• Expected: cleanup keeps working through resolved deltas, old usage and rate-limit snapshots, and turn diffs while other threads are busy, skipping only those threads.
• Actual: any active thread or pending interaction anywhere stops the whole sweep (reason: "busy").

Evidence

• The gate: isDatabaseMaintenanceIdle requires zero active threads, zero pending interactions, zero provisioning, and zero project deletions.
• Where it stops the sweep: thread-pruning-sweep.ts.
• Per-run budget: 50 ms and 64 advances (THREAD_PRUNING_SWEEP_LIMITS). Space comes back at most 20,000 pages (about 78 MiB) per hourly incremental vacuum (DATABASE_INCREMENTAL_VACUUM_MAX_PAGES).
• Backlog on a live workspace: I ran a one-off script against that workspace's database that calls the same advanceThreadPruning/getNextThreadPruningPolicy code, throttled (one advance every 250 ms, backing off on SQLITE_BUSY). In the first 5.5 minutes it removed 7,538 rows that cleanup had never reached. That's only about 3 MB. These policies don't touch item/started and item/completed payloads, which make up most of the size.
• Why the gate was added: it came with 1cfa17779e ("add retention maintenance sweeps"). The commit doesn't say it's needed for correctness. advanceThreadPruning already runs with busy_timeout = 0 and batches of 500, so it won't block app writes.

Proposed change

1. Replace the global idle gate in the pruning sweep with a per-thread exclusion. Skip events for threads that are active, starting, or have a pending interaction, and keep advancing everything else. Turn completion and archiving already do per-thread live pruning (LIVE_BATCH_SIZE), so this is consistent with what's there.
2. Keep the global gate for full VACUUM, which really does block, but consider raising DATABASE_INCREMENTAL_VACUUM_MAX_PAGES so freed pages come back faster than 78 MiB per hour.
3. Add a test showing the sweep advances while an unrelated thread is active, and doesn't touch the active thread's events.

What I ruled out

• Not a duplicate: Prototype direct timeline reads from settled items #4399 (compacting settled history) and Add opt-in Storage & retention with durable cleanup #4476 (opt-in storage and retention plugin) are about pruning more kinds of data. This issue is about the existing policies being starved. Fixing this alone won't shrink item/* payloads.
• Vacuum works: auto_vacuum is 2 (incremental), and freelist_count was 4, so incremental vacuum is reclaiming space. The backlog is rows that were never deleted.

Suggested priority and effort

Medium priority, small effort. It affects any workspace with frequently busy threads. Nothing is lost; the database just grows and search competes for page cache on small hosts. The workaround is to leave the workspace fully idle for long stretches.

BB-Thread: Quick switcher in palette

AGENT GENERATED

[bb-slop-cop]

🚨 SLOP COP 🚨 · new-issue-autopilot

• Classification: Bug · Priority Medium · Effort Low. Added threads; preserved perf.
• Report: https://get-bb.github.io/reports/issues/5005.html
• Verdict: REPRODUCED, root-cause confidence high; applied confirmed-repro.
• Root cause: The workspace-wide idle gate exits before any pruning advance when an unrelated thread or interaction is busy, starving eligible settled history.
• Evidence: The authored regression fails in all four busy states on trusted main in two clean checkouts. After the fix, 196 relevant tests, typechecks, focused lint, and diff checks pass.
• Fix: Ready PR Fix settled-thread pruning starvation #5007 — transactional per-thread eligibility and resumed-batch rechecks; 216 changed lines across four files. Live pruning and vacuum behavior remain unchanged. Not merged.

AGENT GENERATED