Skip to content

Suite of functional tests using Borja's new xmtpd E2E test framework#1809

Merged
api-Hypernova merged 8 commits intomainfrom
hypernova/payere2e
Mar 15, 2026
Merged

Suite of functional tests using Borja's new xmtpd E2E test framework#1809
api-Hypernova merged 8 commits intomainfrom
hypernova/payere2e

Conversation

@api-Hypernova
Copy link
Copy Markdown
Contributor

@api-Hypernova api-Hypernova commented Mar 11, 2026

Add functional E2E test suite using the xmtpd E2E test framework

  • Adds 11 new E2E tests covering chaos scenarios (latency, bandwidth throttle, connection reset, network partition, node down, compound fault, attestation fault), multi-payer attribution, settlement verification, sync verification, sustained load, rate registry changes, and stuck-state detection.
  • Introduces a chain.Contracts reader in pkg/e2e/chain/contracts.go to query on-chain PayerReportManager state for settlement assertions.
  • Adds DisableProxy/EnableProxy to the chaos controller, node handles, and gateway handles to support network partition tests.
  • Refactors the observe.Observer in pkg/e2e/observe/observe.go to use pooled DB connections instead of opening/closing a connection per query.
  • Adds WorkerRepeatIntervalMinutes and WorkerSpreadMinutes config options to PayerReportOptions, plumbed through to the worker scheduler so tests can use short intervals.
  • Extends types.Environment to support multiple named clients per node and an AddRates method for on-chain rate registry updates during test runs.
  • Environment setup now cleans up stale xmtpd-e2e-* Docker networks and idle e2e_* Postgres databases before each run.

Changes since #1809 opened

  • Removed worker timing configuration from payer report infrastructure [95fddc8]
  • Changed client interface and implementation to use common.Address instead of string for address representation [95fddc8]
  • Increased timeout and duration values across E2E tests [95fddc8]
  • Removed default environment variable settings for payer report worker configuration in E2E test nodes [4c9737c]

Macroscope summarized a7c8170.

Ubuntu and others added 4 commits March 11, 2026 14:09
…ture

- Configurable payer report worker timing via env vars
  (XMTPD_PAYER_REPORT_WORKER_REPEAT_INTERVAL_MINUTES,
  XMTPD_PAYER_REPORT_WORKER_SPREAD_MINUTES) reducing e2e test
  times from 60+ min to ~2-4 min per payer-report-dependent test
- 11 new test cases: chaos-attestation-fault, chaos-bandwidth-throttle,
  chaos-compound-fault, chaos-connection-reset, chaos-network-partition,
  multi-payer, rate-registry-change, settlement-verification,
  stuck-state-detection, sustained-load, sync-verification
- Background traffic generator (GenerateTraffic) for continuous
  envelope publishing across minute boundaries
- Contract reader for on-chain rate registry operations
- Enhanced observer with payer report status polling
- Client payer address derivation for per-payer attribution tests
- Proxy toggle (DisableProxy/EnableProxy) for true network partitions
- DB cleanup with safe concurrent connection handling

16/16 tests pass in ~30 min total with 2-min worker cycles.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Set env.Network after createDockerNetwork so cleanup removes networks
- Add cleanupStaleNetworks at startup to remove orphans from crashed runs
- Allow N-wei rounding dust in fee distribution equality check (N = node count)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Observer: add connection pooling (lazy-init per connStr) to prevent
  ~600 connections during long polling waits
- chain_ops: replace sync.Once with mutex+nil-check so transient init
  failures can be retried
- chaos: fix RemoveAllToxics to return real errors instead of silent nil;
  change ProxyAddress to return (string, error) instead of panicking;
  extract addToxic helper eliminating 4x duplication
- client: fix TOCTOU race in GenerateTraffic by holding lock through
  goroutine setup
- environment: add observer.Close() in cleanup
- gitignore: add xmtpd-e2e binary, remove accidentally committed binary

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The function was added during conflict resolution but is not referenced
anywhere in the test suite, and its ecdsa import was missing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@api-Hypernova api-Hypernova self-assigned this Mar 11, 2026
Comment thread pkg/e2e/runner/environment.go Outdated
Comment thread pkg/payerreport/workers/runner.go Outdated
Comment thread pkg/e2e/tests/multi_payer.go Outdated
Comment thread pkg/e2e/types/node_handle.go
Comment thread pkg/e2e/chaos/chaos.go
- Use pq.QuoteIdentifier for SQL identifier safety in dropE2EDatabases
- Fix multi_payer envelope count: wait for envelopesPerPayer*2 (both payers replicate to all nodes)
- Guard DisableProxy/EnableProxy in NodeHandle against nil Chaos controller
- Document toggle non-idempotency contract in DisableProxy

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Local linter rewrote fmt.Errorf to errors.New but swapped fmt for errors
without updating the import block.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@api-Hypernova api-Hypernova changed the title Suite of functional tests using Borja's new xmtpd E2E test framework! Suite of functional tests using Borja's new xmtpd E2E test framework Mar 12, 2026
@api-Hypernova api-Hypernova marked this pull request as ready for review March 12, 2026 17:14
@api-Hypernova api-Hypernova requested a review from a team as a code owner March 12, 2026 17:14
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Mar 12, 2026
@macroscopeapp
Copy link
Copy Markdown
Contributor

macroscopeapp Bot commented Mar 12, 2026

Approvability

Verdict: Approved

This PR adds E2E functional tests to the test framework with no changes to production runtime code. All modified files are owned by the author, who maintains the e2e test package.

You can customize Macroscope's approvability policy. Learn more.

@api-Hypernova api-Hypernova requested a review from fbac March 13, 2026 03:56
Comment thread pkg/config/options.go Outdated
Comment thread pkg/e2e/observe/observe.go
Comment thread pkg/e2e/client/client.go Outdated
- Revert configurable worker timing from production code (options.go,
  runner.go, server.go) per fbac's feedback: e2e tests should observe
  the real system, not alter it
- Update all payer-report-dependent test timeouts from 10min to 75min
  to accommodate default 60-min worker cycles
- Change Client interface from PayerAddress() string to Address()
  common.Address, matching the GatewayHandle pattern for type safety
- Simplify ClientHandle.Address() to delegate to client.Address()
  instead of re-parsing the key
@macroscopeapp macroscopeapp Bot dismissed their stale review March 13, 2026 17:32

Dismissing prior approval to re-evaluate 95fddc8

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Mar 13, 2026
The production code no longer supports XMTPD_PAYER_REPORT_WORKER_REPEAT_INTERVAL_MINUTES
and XMTPD_PAYER_REPORT_WORKER_SPREAD_MINUTES (reverted in previous commit). Passing
these unknown env vars caused go-flags to reject the config and crash the node on startup.
@macroscopeapp macroscopeapp Bot dismissed their stale review March 13, 2026 20:55

Dismissing prior approval to re-evaluate 4c9737c

@api-Hypernova api-Hypernova requested a review from fbac March 14, 2026 17:00
@api-Hypernova api-Hypernova merged commit 34646fe into main Mar 15, 2026
16 of 17 checks passed
@api-Hypernova api-Hypernova deleted the hypernova/payere2e branch March 15, 2026 03:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants