Skip to content

Add data-retention policy for new time-series tables, aligned with BACKUP_RETENTION_DAYS #48

Description

@christabel888

Context: These new time-series tables (#45-#47) will grow unbounded without a retention policy. .env.example already has BACKUP_RETENTION_DAYS for backups -- this is the equivalent for time-series data.

Where:

  • .env.example -- add a retention config (e.g. TIMESERIES_RETENTION_DAYS)
  • A cleanup job in src/jobs/, following the pattern of the existing JOB_CACHE_CLEANUP_ENABLED job

What to do:

Suggested approach:

  1. Define the policy first (e.g. keep daily granularity 1 year, then drop or downsample) before writing the cleanup job.
  2. Follow the existing JOB_CACHE_CLEANUP_ENABLED/JOB_CACHE_CLEANUP_INTERVAL_SECONDS pattern for the new cleanup job's config.
  3. Apply this to Design and migrate time-series schema for daily network metrics #45, Design and migrate time-series schema for soroban contract metrics #46, Design and migrate time-series schema for wallet balance history #47's tables plus Add historical top-movers snapshot storage for trend/share cards #37's top-movers snapshots -- one retention job covering all of them, not four separate ones, unless their retention windows genuinely differ.

Watch out for:

  • Test the deletion query on a copy of realistic-volume data before running it against anything real -- an unbounded DELETE without a LIMIT/batching strategy can lock a table for a long time on a large dataset.

Definition of done:

  • Old rows are pruned/downsampled per the documented policy, verified in a test

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-uiphase-0-hardeningCross-cutting engineering health: tests, docs, infra

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions