feat: rate limit per peer to mitigate decrypt burst

[otimizeinformatica]

Problem

On active WhatsApp groups, sender key rotation causes messages to arrive with encryption iterations far ahead of the current sender chain. The libsignal-go library groups/GroupCipher.go:getSenderKey ratchets forward up to 2000 steps per message to catch up.

When many messages from the same peer arrive concurrently, this can result in ~200k crypto operations in seconds. CPU/memory saturate and long message payloads get truncated during decrypt — we've observed real user reports of messages arriving with truncated bodies (only the URL kept, description dropped).

Root symptom in production: p50 latency wa→db of 14s, p90 of 850s, max of 3000s during burst events. Recovery only via container restart.

Solution

Per-peer token bucket rate limiter as a defensive layer at the wuzapi wrapper level. No changes to libsignal or whatsmeow.

Changes

• ratelimit.go new: token bucket, sync.Once lazy init, configured via env
• wmiau.go: guard at the top of case *events.Message

Configuration

Env	Default	Meaning
RATE_LIMIT_PEER_MSG_PER_SEC	20	sustained rate per peer
RATE_LIMIT_PEER_ENABLED	true	set false to disable feature

Capacity = rate × 2 burst tolerance 2s. Peer key = Sender JID or Chat|Sender for groups.

Behavior

• Peers within limit: unaffected default 20/s handles all normal traffic
• Peer that exceeds: excess msgs dropped silently, warn log emitted with msg_id and peer for observability
• Legitimate conversation less than 5 msgs/sec always passes
• Bombardment during sender key rotation more than 50 msgs/sec is flattened

Not a root cause fix

The underlying issue is in libsignal-go — getSenderKey lacks a cost bound. A proper fix would add a per-sender rate limit in the crypto layer itself. That path requires forking libsignal, whatsmeow, and wuzapi.

This PR chooses the minimally invasive path: single fork, non-crypto code, easily reversible via env flag. Happy to abandon this if a crypto-layer fix lands upstream in libsignal-go.

Test plan

• Fork built and deployed alongside main wuzapi port 8091 on our staging setup
• Container running SQLite backend, isolated from prod
• Idle stability observed OK — ready for canary QR pairing to measure real-world impact vs baseline

Open as draft while we validate in canary mode.

[asternic]

Hi @otimizeinformatica , thanks for your contribution. Forking or changing libsignal or whatsmeow is out of the question so I think your approach is fine. Let me ask you, what happens if the rate limit kicks in in those busy groups? Does meta retries silently? Does it disrupt or breaks anything?

[otimizeinformatica]

Great question, and you nailed the weakest point of this v1 design. Let me be upfront about what actually happens today:

Current behavior on drop

The return in my handler happens inside case *events.Message, but whatsmeow has already called sendAck(node, 0) by the time dispatchEvent runs — Meta gets the ACK regardless of what our handler decides. So:

• Meta considers the message delivered ✓✓
• No retry, no re-send
• Message never reaches the webhook, never persisted
• Silent data loss

This is fine for the pathological "sender-key rotation storm" case where dropping is objectively better than crashing, but it's obviously wrong for legitimate high-traffic groups. Both hit the same bucket right now.

What I'll change (v2)

Instead of dropping, enqueue in memory per peer and drain at the configured rate from a goroutine. Concretely:

• Per-peer channel (buffered, ex. 500), dispatcher goroutine that pulls at RATE_LIMIT_PEER_MSG_PER_SEC
• When bucket is exhausted, event is queued (not dropped) until slot frees
• Only if the per-peer queue overflows (client stuck, backpressure impossible to sustain) we then drop with warn log — that's the crash-avoidance fallback
• Sensible defaults so busy legit groups (chat ativity) never hit the overflow path

Trade-off is latency: bursty messages get processed serially with a small delay rather than all-at-once. That's the whole point — we're smoothing the cost curve of getSenderKey ratchet-forward rather than incurring 100× concurrent ratchets.

No disruption to normal traffic

Default rate (20 msg/s per peer) is well above any real conversation. Legit chat < 5 msg/s is untouched. Only sender-key-rotation bombardments (typically 50-200 msg/s from a single sender chain) hit the flow control.

Will push v2 shortly. Happy to keep the drop-fallback behind a flag if you'd prefer a stricter "never lose a message" default.

[otimizeinformatica]

Thanks for the feedback! I've implemented v2 that enqueues rather than drops.

Branch: otimizeinformatica:feat/peer-rate-limit-v2 · commit d0183bf

Approach

Instead of a separate queue+dispatcher goroutine per peer, I leveraged the fact that whatsmeow spawns a goroutine per event. The rate limiter now blocks the peer's goroutine on the slow path when tokens are unavailable, effectively creating natural backpressure without buffering messages in userspace or losing them below the timeout threshold.

• Fast path: token available → passes immediately (~zero overhead, same as v1)
• Slow path: no token → sleeps with poll interval 1/rate seconds (clamped 50ms–500ms) until refill
• Queue-full defense: if waiters > MAX_WAITERS_PER_PEER (default 200) → drops (protects memory from runaway peer)
• Timeout fallback: if slow path exceeds MAX_WAIT_MS (default 30s) → drops (protects against a permanently saturated peer)
• Stats goroutine: logs passed_fast / passed_slow / dropped_full / dropped_timeout every 5min

Why not a dispatcher goroutine per peer

I considered chan queuedMsg + go dispatcher() per peer, but:

1. The event handler in wmiau.go returns synchronously (case *events.Message). Handing off the message to another goroutine would break return semantics and require refactoring the handler signature and everything downstream that relies on the current call being the one processing the event.
2. Peer bucket lifetime management gets complex (need to close channels, kill idle dispatchers, sync stops during shutdown).
3. Whatsmeow's own goroutine-per-event model already provides the "worker pool" behavior we need — blocking that goroutine is idiomatic Go.

Blocking is a smaller change to the handler surface, requires no additional goroutines except one for stats, and has zero risk of message loss below the timeout threshold.

Config (new env vars)

RATE_LIMIT_PEER_MSG_PER_SEC=20      # unchanged from v1
RATE_LIMIT_PEER_ENABLED=true        # unchanged from v1
RATE_LIMIT_PEER_MAX_WAIT_MS=30000   # NEW: max time a msg can wait (default 30s)
RATE_LIMIT_PEER_MAX_WAITERS=200     # NEW: max concurrent waiters per peer (default 200)

Legacy Allow() is kept for backward compat (fast path only, no wait) but the wmiau.go integration now calls WaitOrTimeout().

Tests

Added ratelimit_test.go covering:

• TestFastPath — token available, passes immediately
• TestDisabled — enabled=false bypasses everything, no stats mutation
• TestBurstConsumesCapacity — 20 fast + 1 slow after refill
• TestTimeout — permanently saturated peer times out
• TestQueueFull — 20 concurrent goroutines on a slow peer with maxWaiters=5 → some full drops, some timeouts, no lost accounting
• TestPerPeerIndependence — saturating peer1 doesn't affect peer2/peer3
• TestConcurrentSamePeer — 100 concurrent goroutines, total == 100 (no race, -race clean)

All 7 tests pass with -race:

=== RUN   TestFastPath
--- PASS: TestFastPath (0.00s)
=== RUN   TestDisabled
--- PASS: TestDisabled (0.00s)
=== RUN   TestBurstConsumesCapacity
--- PASS: TestBurstConsumesCapacity (0.10s)
=== RUN   TestTimeout
--- PASS: TestTimeout (0.05s)
=== RUN   TestQueueFull
    ratelimit_test.go:137: TestQueueFull: droppedFull=15 droppedTimeout=5
--- PASS: TestQueueFull (0.51s)
=== RUN   TestPerPeerIndependence
--- PASS: TestPerPeerIndependence (0.00s)
=== RUN   TestConcurrentSamePeer
    ratelimit_test.go:188: N=100: fast=100 slow=0 full=0 timeout=0
--- PASS: TestConcurrentSamePeer (0.00s)
PASS
ok      command-line-arguments  1.716s

Canary status

v2 is currently running on an isolated test container against a low-traffic chip since 2026-08-18. passed_slow / dropped_* are still 0 because canary traffic hasn't been bursty enough to exhaust the bucket — will report metrics after production-shaped load.

Happy to iterate if you'd like different defaults, log formats, or extra tests.