fix(pool): bound each step of returning a connection to the pool

[adriangb]

Fixes #4349.

The problem

Floating::return_to_pool holds the pool's DecrementSizeGuard while it does I/O on the connection being returned: the after_release hook, the ping() that checks the connection is still usable, and the graceful close() on the max-lifetime and pool-closed paths. None of those are bounded.

That is fine when a connection is either healthy or visibly broken. It is not fine when the socket is dead in a way it cannot report. If the server disappeared without closing the connection (a failover, a killed container, a dropped NAT mapping) and the client has nothing left to retransmit, no RST is ever provoked and the read never completes. The spawned task that is returning the connection then holds its permit forever, and the pool shrinks by one usable connection.

Repeat that max_connections times and every acquire() fails with PoolTimedOut from then on, even after the server comes back, because the permits are held by tasks parked on dead sockets.

We hit this in production behind a maintenance loop that had a query in flight when its Postgres instance went away.

The fix

Each step gets RETURN_TO_POOL_TIMEOUT (5s, matching the existing CLOSE_ON_DROP_TIMEOUT a few lines up), and on expiry the connection is discarded with close_hard(), which does no I/O. Cancelling close() likewise drops the connection and releases the permit, so the bounded-close path is safe too.

Test

pool_recovers_from_a_hanging_after_release uses an after_release hook that never completes, as a deterministic stand-in for a socket that never answers. On main it exhausts acquire_timeout and fails after 30s; with this change the pool is usable again after the 5s bound:

# on main
test result: FAILED. 0 passed; 1 failed; finished in 30.05s

# with this change
test pool_recovers_from_a_hanging_after_release ... ok
test result: ok. 1 passed; 0 failed; finished in 5.08s

Note on #3582

I saw in #4146 that the new pool architecture in #3582 already covers this with a timeout of its own. This is meant as the fix for main until that lands, and should be straightforward to drop when it does. Happy to close it if you would rather not carry an interim change.

[adriangb]

Closing in favour of #4350, which does the same thing, was opened first, and is by the author of #4349. I found it only after opening this. Their fake-server test is also a better reproduction than the hanging after_release hook I used here.

I have left the one piece of this that #4350 does not cover as a comment there: return_to_pool holds the permit across two other unbounded awaits, the after_release hook and the graceful close() on the pool-closed and beyond-max_lifetime paths, and the max_lifetime one is a path we actually hit. Happy to send that as a follow-up once #4350 lands.