Skip to content

Regularly refresh materialized view view from indexer#1013

Merged
emmazzz merged 9 commits into
mainfrom
emma/add-net-deposit-refresh-to-indexer
May 14, 2026
Merged

Regularly refresh materialized view view from indexer#1013
emmazzz merged 9 commits into
mainfrom
emma/add-net-deposit-refresh-to-indexer

Conversation

@emmazzz

@emmazzz emmazzz commented May 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Move materialized view refreshes from the API reader to the indexer write process.
  • Add a generic background refresh service for materialized views, with the refreshed views centralized in the code-owned MATERIALIZED_VIEWS_TO_REFRESH registry. The current registry refreshes net_deposits_hourly.
  • Keep the refresh interval configurable with a 60 second default and support for disabling with 0.
  • Track refresh outcomes and timing with materialized_view_refresh_successes_total{view="..."}, materialized_view_refresh_failures_total{view="..."}, and materialized_view_refresh_duration_seconds{view="..."}.

Key decisions

  • Keep /get_net_deposits read-only so Cloud SQL read-only connections are not asked to run REFRESH MATERIALIZED VIEW.
  • Use the indexer write-capable database handle for refreshes because it already owns migrations and derived data maintenance.
  • Have /get_net_deposits split MV and raw balance reads using each asset’s latest refreshed MV bucket: use MV buckets before that cutoff, then scan raw balances from the cutoff to the requested timestamp.
  • Use a static code registry instead of auto-discovering every Postgres materialized view, so each future synthesized view can opt in after confirming it supports concurrent refresh.
  • Attach the refresh service to the indexer service lifecycle so it is shut down with the rest of the indexer.
  • Keep the materialized view approach for now rather than introducing an incremental rollup table in this PR.

Test plan

  • cargo fmt -p deepbook-indexer
  • cargo build -p deepbook-indexer
  • cargo build -p deepbook-server
  • git diff --check -- crates/indexer/src/main.rs

@emmazzz emmazzz changed the title [codex] Refresh net deposits view from indexer Regularly refresh materialized view view from indexer May 14, 2026
@emmazzz emmazzz marked this pull request as ready for review May 14, 2026 07:07

/// Query net deposits using the materialized view for efficiency.
/// Triggers a concurrent refresh of the view and queries for net deposits up to timestamp.
/// The indexer refreshes this view with a write-capable database connection.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving REFRESH MATERIALIZED VIEW off the API reader path is the right direction for readonly DBs, but this query still assumes the view is fresh for every bucket before hour_bucket_ms. With a 30s async indexer refresh, a request just after an hour boundary, or during refresh failures, can miss late writes from the previous completed hour: they are not in the MV yet, and the raw balances scan starts at the new hour. To keep the reader readonly and exact, the writer/indexer refresh path should publish a refresh watermark, and this query should use the MV only before that watermark while scanning raw balances from the watermark to timestamp_ms.

@emmazzz emmazzz merged commit 9eea8d7 into main May 14, 2026
4 checks passed
@emmazzz emmazzz deleted the emma/add-net-deposit-refresh-to-indexer branch May 14, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants