fix(pool): bound the on-release ping so a dead connection can't leak its permit

[rslowinski]

Fixes #4349.

Problem

When a PoolConnection is dropped, the pool spawns a task (return_to_pool) that pings the connection before returning it to the idle queue. If the peer has gone away silently — no RST/FIN, e.g. a NAT/firewall expired the flow, or a query was abandoned via tokio::time::timeout — the ping request is written successfully and then the read waits forever for a response that never comes.

Because that wait happens inside the drop-spawned task, it strands the task's DecrementSizeGuard, so the connection's pool permit is never released. Once this has happened max_connections times, every subsequent acquire() fails with PoolTimedOut and the pool never recovers without a process restart. There are no error logs, because nothing errors — the future simply never completes.

#4349 has the full write-up and a standalone reproduction.

Fix

Bound the on-release ping() with a timeout, following the pattern already used for the close_on_drop path (CLOSE_ON_DROP_TIMEOUT in take_and_close). On timeout the connection is treated as unresponsive and discarded with close_hard() — a local socket shutdown that cannot itself block on the dead peer — which releases the permit so the pool can open a replacement. No public API change; the bound is a sibling const of the existing one (5s).

The close_on_drop path was already bounded this way; this closes the same gap on the default return-to-pool path.

Test

tests/postgres/pool.rs drives a real PgPool against an in-process fake server that completes the startup handshake and then goes silent (keeps reading, never replies) — the exact state of a silently-dropped flow. It checks out a connection, drops it (arming the on-release ping against the silent peer), and asserts the next acquire() succeeds by opening a fresh connection rather than failing with PoolTimedOut.

• Needs no DATABASE_URL (connects only to its own local listener), so it runs in CI.
• Gated to the tokio runtime cells via required-features = ["postgres", "runtime-tokio"] (it uses a tokio fake server), and skipped under the other runtime matrix cells.
• Passes with the fix (~5s, the ping bound); without the fix it fails with PoolTimedOut at acquire_timeout — verified locally by reverting the fix.

Notes for reviewers

• I reused a hardcoded 5s bound to match CLOSE_ON_DROP_TIMEOUT rather than introduce a config knob; happy to make it configurable or derive it from acquire_timeout instead if you'd prefer.
• The timeout arm calls close_hard() (socket shutdown only). If you'd rather also guard close_hard/close against a hypothetically-blocking shutdown, I can wrap those too, but they don't await the peer today.

This change was developed with AI assistance; I've reviewed it end-to-end and can speak to the design and trade-offs.

[adriangb]

Sorry about the noise, agent went a bit wild, I reigned it in and rewrote / went over this comment by hand.

We hit the same failure in production: a server restart during a query execution causes the query to hang forever. We noticed this because we have a ~ hot loop that is constantly hitting postgres, so we have a high chance of having an in-flight query where the timing of an upstream restart causes it to hang.

A gap found working on #4354 (now closed in favor of this PR): return_to_pool has three unbounded awaits, not one. The ping is the common case, but the same permit is held across:

• the after_release hook, which is arbitrary user code and, more to the point, usually runs a query on the same dead socket;
• self.close().await on the pool-closed and beyond-max_lifetime paths. close() is a graceful close: it writes Terminate and waits for the server to drop the connection, which on a silent peer never happens.

The max_lifetime one is not hypothetical for us: we set it, so every aged-out connection takes that path, and an aged-out connection on a dead socket leaks its permit exactly like the ping case.

The fix is mechanical: give each the same bound and fall back to close_hard(), which does no I/O. Cancelling close() also drops the connection and releases the permit, so a bounded close is safe:

async fn close_bounded(self) {
    let _ = crate::rt::timeout(RETURN_TO_POOL_PING_TIMEOUT, self.close()).await;
}