feat(backend): implement event indexer pagination and backpressure co…

[sweetesty]

Closes #475

⸻

This PR implements dynamic pagination and adaptive backpressure controls for the Soroban event indexer to ensure safe catch-up behavior, stable RPC usage, and correct recovery after restarts.

Indexer architecture

• Enhanced backend/src/indexer/ledger_follower.ts to support controlled ledger-range processing
• Indexer now operates in bounded batches instead of large unbounded RPC queries
• Designed to safely switch between catch-up mode and normal steady-state polling

Dynamic range pagination

• Introduced configurable maxBlockLimit (default: 100 blocks per batch)
• Updated pollOnce() to fetch events within:
  • lastProcessedLedger → min(latestLedger, lastProcessedLedger + currentBatchSize)
• Prevents oversized ledger range queries during backlog recovery
• Ensures gradual catch-up without overwhelming RPC nodes

Catch-up loop optimization

• Refactored loop() to adapt polling behavior:
  • Fast polling mode (50ms sleep) during catch-up
  • Normal interval polling when fully synced
• Improves responsiveness while maintaining stability under load

Adaptive backpressure (AIMD model)

• Implements Additive Increase / Multiplicative Decrease strategy:
  • Multiplicative decrease: when event density > 5.0 events/block
    • Batch size is halved
    • 500ms throttle delay applied
  • Additive increase: when event density is low
    • Batch size increases gradually by +10 blocks
    • Capped at configured maxBlockLimit
• Ensures system self-adjusts under varying event loads

State persistence & deduplication safety

• Indexer progress is persisted in indexer_state after each processed batch
• Guarantees safe restart without duplicate event processing
• Ensures consistent continuation from last committed ledger

Testing & verification

• Added unit tests in backend/tests/ledger_follower.test.ts
• Mock-based test suite validates:
  • Dynamic pagination behavior
  • Multiplicative backpressure response under high density
  • Additive recovery under low load conditions
• All tests pass successfully

Acceptance criteria met:

• ✅ Indexer safely processes historical ledger backlogs without RPC overload
• ✅ Adaptive batching prevents timeout and throttling issues
• ✅ No duplicate events on restart due to persisted state
• ✅ System dynamically responds to event density changes
• ✅ Verified via automated unit tests

[vercel]

@sweetesty is attempting to deploy a commit to the mAzI's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@sweetesty Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits