Skip to content

Add monitoring/alerting for new aggregation jobs via src/monitor.rs and src/alerts.rs #44

Description

@christabel888

Context: New scheduled jobs (#2, #34, plus the extended contract listener from #20) are new failure surfaces. src/monitor.rs and src/alerts.rs already exist for this kind of thing.

Where:

  • src/monitor.rs, src/alerts.rs

What to do:

  • Add health checks / alert rules for the new jobs (job hasn't run in N intervals, job erroring repeatedly), following whatever pattern already covers the existing jobs.

Suggested approach:

  1. Look at how src/monitor.rs/src/alerts.rs already cover the existing jobs (corridor refresh, anchor refresh, etc.) and add the same shape of health check for Add daily-active-accounts computation job in src/jobs #2's DAA job, Add Soroban contract-invocation ingestion pipeline in src/ingestion #20's extended listener, and Add hourly background job to compute top movers in src/jobs #34's top-movers job.
  2. Alert on "hasn't run successfully in N expected intervals" and "erroring repeatedly," not just "is the process running."

Watch out for:

  • Alert fatigue is a real risk if thresholds are too tight for jobs with long intervals (e.g. an hourly job) -- calibrate the "hasn't run" threshold to a few missed intervals, not one.

Definition of done:

  • New jobs are covered by the same monitoring/alerting as existing jobs, not silently unmonitored

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions