feat: TCP keepalive for Postgres and MySQL connections#4355
Open
adriangb wants to merge 1 commit into
Open
Conversation
adriangb
force-pushed
the
feat/tcp-keepalive
branch
5 times, most recently
from
July 24, 2026 20:25
ee406e3 to
1e8c1d9
Compare
Adds `TcpKeepalive` and `net::connect_tcp_with_keepalive`, plus `tcp_keepalive()` on `PgConnectOptions` and `MySqlConnectOptions` and libpq-compatible URL parameters for Postgres (`keepalives`, `keepalives_idle`, `keepalives_interval`, `keepalives_count`). Keepalive is off by default, so nothing changes for existing users. Motivation: without it, a connection whose server disappeared *without* closing the socket never finds out. A failover, a killed container, or a dropped NAT mapping leaves the client blocked reading a response that will never arrive; there is nothing left to retransmit, so no RST is ever provoked and the read waits forever. `TCP_NODELAY`, which is all SQLx sets today, does not help, and a server-side `statement_timeout` cannot fire on a server that is gone. This is the case @abonander allowed for in transact-rs#3559 (comment): "I suppose that's still preferable to it hanging forever on a read that will never complete." Worth adding on the objection raised there, that a keepalive timeout is only noticed the next time the socket is used: that is not true of a blocked reader. When the probes are exhausted the kernel sets `sk_err` to `ETIMEDOUT` and wakes anyone parked on the socket, so the pending read fails rather than waiting for someone to poke it. We hit this in production: a maintenance loop that had a query in flight when its Postgres instance went away stopped doing work permanently, while other loops in the same process recovered in seconds. Reproduced by holding an `ACCESS EXCLUSIVE` lock so the query blocks server-side, then removing the server from the network and restarting it. Implementation follows transact-rs#3559 by @xuehaonan27, rebased onto the current `connect_tcp` and reduced to an additive API: `connect_tcp` keeps its signature and delegates, so external drivers are unaffected.
adriangb
force-pushed
the
feat/tcp-keepalive
branch
from
July 24, 2026 21:10
1e8c1d9 to
be545ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds TCP keepalive support, closing #3540. This is @xuehaonan27's #3559 rebased onto the current
connect_tcpand reduced to an additive API; the design credit is theirs.Why
Without keepalive, a connection whose server disappeared without closing the socket never finds out. A failover, a killed container, or a dropped NAT mapping leaves the client blocked reading a response that will never arrive. There is nothing left to retransmit, so no RST is ever provoked, and the read waits forever.
TCP_NODELAY, which is all SQLx sets today, does not help, and a server-sidestatement_timeoutcannot fire on a server that is gone.We hit this in production: we have a ~ hot loop that is constantly querying so we are almost guaranteed to hit this every server restart (pgbouncer scale down in our case). It reproduces reliably by holding an
ACCESS EXCLUSIVElock so the query blocks server-side, then removing the server from the network and restarting it. Notably a restart w/o an in-flight query won't reproduce: the query has to be already acknowledged and executing, which is exactly the common case during a real failover.@abonander, in #3559 (comment) you allowed for this case: "I suppose that's still preferable to it hanging forever on a read that will never complete." One correction on the objection raised there, that a keepalive timeout is only noticed the next time we use the socket: that is not true for a blocked reader. When the probes are exhausted the kernel sets
sk_errtoETIMEDOUTand wakes anyone parked on the socket, so a pending read fails rather than waiting for someone to poke it. That is precisely the case that hangs today.Since #3559 was opened, three other users have reported the same silent-stale-connection failure in it (Aug 2025, Dec 2025, Feb 2026), and the author of the follow-up #4159 moved to
tokio-postgresover it.What this adds
sqlx_core::net::TcpKeepalive, withidle/interval/retries, cfg-gated for the platforms where socket2 does not support the latter two.net::connect_tcp_with_keepalive.connect_tcpkeeps its signature and delegates withNone, so external drivers (Compile-time support for external drivers #3889) are unaffected.PgConnectOptions::tcp_keepalive()andMySqlConnectOptions::tcp_keepalive().keepalives,keepalives_idle,keepalives_interval,keepalives_count. These previously hit theignoring unrecognized connect parameterbranch, so anyone copying a libpq DSN silently got no keepalive.Keepalive stays off by default, so behaviour is unchanged for existing users. (libpq defaults it on with a 2h idle; I did not want to change a default in this PR, but happy to if you would prefer parity.)
Tests
Unit tests for the builder and for the URL parameters, including that
keepalives=0disables it again after another parameter turned it on. Thesetsockoptitself is not unit-testable without a socket; it is exercised by every connection made with keepalive configured.I am carrying this as a patched fork in the meantime, so I am happy to iterate on the API shape or split MySQL out if that makes review easier.