Skip to content

Production drills not executed: restore, paging, security disclosure #149

Description

@dantaik

Context

All 13 incident-response runbooks under docs/runbooks/ are written and operationally complete. None have been drilled against the real production environment. The backup-drill-prod.yml GitHub workflow exists but its secrets (PROD_LITESTREAM_*) must be populated and a manual run logged.

Acceptance criteria

Restore drill

  • PROD_LITESTREAM_* secrets populated in the repo settings.
  • Manual workflow_dispatch run of backup-drill-prod.yml for both hub and watchtower.
  • Artifacts (restore logs, restored SQLite checksums) attached to a launch-evidence note.
  • Recovery time vs. RPO/RTO targets recorded.

Paging drill

  • Real ALERTMANAGER_PAGER_WEBHOOK_URL Secret populated via secrets-bootstrap.sh --bootstrap-monitoring.
  • Synthetic critical alert injected (e.g., temporarily set HubLiquidityLow threshold high enough to fire).
  • PagerDuty / Slack receipt confirmed; primary on-call acknowledges within target.
  • Timing and screenshots archived.

Security disclosure drill

  • Submit a non-vulnerability stub via GitHub private advisory; verify it routes to the maintainer notification surface.
  • Once PGP is published (#tbd), repeat via encrypted email and verify decryption.
  • Acknowledgement timestamps recorded against the 24h SLA from SECURITY.md.

Severity

High — GA blocker. Cannot promise on-call response or recovery time without having exercised the paths.

Evidence pointers

  • final_readiness_report.html §10.2, §12 DOC-02, §13.1 item 5.
  • docs/runbooks/{backup-restore,hub-incident,watchtower-down,security-disclosure}.md.
  • .github/workflows/backup-drill-prod.yml.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions