Repository navigation
ApacheHttpClient never reuses pooled connections when a TLS client certificate is configured (tlsKeyManagersProvider) #7405
Description
Describe the bug
When an ApacheHttpClient presents a client certificate (via tlsKeyManagersProvider) to a server that requests one (mutual TLS), it never reuses a pooled connection. Every request opens a new TCP connection with a full TLS handshake. Once the pool for the route is full, Apache closes an idle connection to make room. The same client without a client certificate reuses a single connection.
Cause: Apache HttpClient 4.5 tracks a per-connection user identity.
-
After each response, MainClientExec asks DefaultUserTokenHandler for a token. With a client certificate, that token is the TLS session's local principal. The connection goes back to the pool with that token as its state.
-
The pool leases a connection only to a request whose context carries an equal token. RouteSpecificPool#getFree(null) returns only entries whose state is null.
-
ApacheHttpClient#execute creates a new HttpClientContext for every request (ApacheUtils.newClientContext(...)) and never passes one in from the caller. So every request leases with a null token and can never match a pooled mTLS connection.
Regression Issue
- Select this option if this issue appears to be a regression.
Expected Behavior
Sequential requests from one ApacheHttpClient to the same endpoint reuse a pooled connection, whether or not a client certificate is presented. docs/BestPractices.md says: "Each SDK client maintains its own HTTP connection pool so that connections can be reused by a new request to cut down the time to establish a new connection."
Current Behavior
With a client certificate, each request opens a new connection and does a full TLS handshake. Under load this means one handshake per request and constant connection churn. No error is logged; the only symptom is new connections.
client certificate: true, mode: default -> 5 requests used 5 TCP connection(s)
client certificate: false, mode: --no-client-cert -> 5 requests used 1 TCP connection(s)
Reproduction Steps
Create two throwaway keystores. The server accepts any client certificate; it only has to be presented.
keytool -genkeypair -alias server -keyalg RSA -dname CN=server -keystore server.p12 -storetype PKCS12 -storepass changeit -keypass changeit
keytool -genkeypair -alias client -keyalg RSA -dname CN=client -keystore client.p12 -storetype PKCS12 -storepass changeit -keypass changeit
Put software.amazon.awssdk:apache-client:2.44.7 and its runtime dependencies on the classpath, then run:
java -cp "$CLASSPATH" MtlsReuseRepro.java # mutual TLS: 5 connections
java -cp "$CLASSPATH" MtlsReuseRepro.java --no-client-cert # control: 1 connection
java -cp "$CLASSPATH" MtlsReuseRepro.java --constant-user-token # workaround: 1 connection
import com.sun.net.httpserver.HttpsConfigurator;
import com.sun.net.httpserver.HttpsParameters;
import com.sun.net.httpserver.HttpsServer;
import java.io.InputStream;
import java.net.InetSocketAddress;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.security.cert.X509Certificate;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.apache.http.HttpException;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequest;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.routing.HttpRoute;
import org.apache.http.impl.conn.DefaultRoutePlanner;
import org.apache.http.impl.conn.DefaultSchemePortResolver;
import org.apache.http.protocol.HttpContext;
import software.amazon.awssdk.http.FileStoreTlsKeyManagersProvider;
import software.amazon.awssdk.http.HttpExecuteRequest;
import software.amazon.awssdk.http.HttpExecuteResponse;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.SdkHttpConfigurationOption;
import software.amazon.awssdk.http.SdkHttpMethod;
import software.amazon.awssdk.http.SdkHttpRequest;
import software.amazon.awssdk.http.TlsKeyManagersProvider;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.utils.AttributeMap;
import software.amazon.awssdk.utils.IoUtils;
/**
* Usage: java MtlsReuseRepro [--no-client-cert | --constant-user-token]
*
* <p>Needs server.p12 and client.p12 (PKCS12, password "changeit") in the working directory.
*/
public class MtlsReuseRepro {
public static void main(String[] args) throws Exception {
String mode = args.length > 0 ? args[0] : "";
boolean clientCert = !mode.equals("--no-client-cert");
Set<Integer> clientPorts = ConcurrentHashMap.newKeySet();
HttpsServer server = HttpsServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.setHttpsConfigurator(
new HttpsConfigurator(serverSslContext()) {
@Override
public void configure(HttpsParameters params) {
SSLParameters ssl = getSSLContext().getDefaultSSLParameters();
ssl.setNeedClientAuth(clientCert);
params.setSSLParameters(ssl);
}
});
server.createContext(
"/",
exchange -> {
// Each TCP connection arrives from its own client port.
clientPorts.add(exchange.getRemoteAddress().getPort());
byte[] body = "ok".getBytes();
exchange.sendResponseHeaders(200, body.length);
exchange.getResponseBody().write(body);
exchange.close();
});
server.start();
ApacheHttpClient.Builder builder =
ApacheHttpClient.builder()
.tlsKeyManagersProvider(
clientCert
? FileStoreTlsKeyManagersProvider.create(
Path.of("client.p12"), "PKCS12", "changeit")
: TlsKeyManagersProvider.noneProvider());
if (mode.equals("--constant-user-token")) {
builder.httpRoutePlanner(new ConstantUserTokenRoutePlanner());
}
try (SdkHttpClient client =
builder.buildWithDefaults(
AttributeMap.builder()
.put(SdkHttpConfigurationOption.TRUST_ALL_CERTIFICATES, Boolean.TRUE)
.build())) {
URI uri = URI.create("https://127.0.0.1:" + server.getAddress().getPort() + "/");
for (int i = 0; i < 5; i++) {
HttpExecuteResponse response =
client
.prepareRequest(
HttpExecuteRequest.builder()
.request(SdkHttpRequest.builder().method(SdkHttpMethod.GET).uri(uri).build())
.build())
.call();
try (InputStream in = response.responseBody().orElseThrow()) {
IoUtils.drainInputStream(in); // releases the connection back to the pool
}
}
} finally {
server.stop(0);
}
System.out.println(
"client certificate: " + clientCert + ", mode: " + (mode.isEmpty() ? "default" : mode)
+ " -> 5 requests used " + clientPorts.size() + " TCP connection(s)");
}
/** Server key from server.p12; accepts any client certificate, which only has to be presented. */
private static SSLContext serverSslContext() throws Exception {
KeyStore keyStore = KeyStore.getInstance("PKCS12");
try (InputStream in = Files.newInputStream(Path.of("server.p12"))) {
keyStore.load(in, "changeit".toCharArray());
}
KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(keyStore, "changeit".toCharArray());
TrustManager acceptAnyClient =
new X509TrustManager() {
@Override
public void checkClientTrusted(X509Certificate[] chain, String authType) {}
@Override
public void checkServerTrusted(X509Certificate[] chain, String authType) {}
@Override
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
};
SSLContext context = SSLContext.getInstance("TLS");
context.init(kmf.getKeyManagers(), new TrustManager[] {acceptAnyClient}, null);
return context;
}
/** Workaround: put a constant user token in the context before the connection is leased. */
static final class ConstantUserTokenRoutePlanner extends DefaultRoutePlanner {
ConstantUserTokenRoutePlanner() {
super(DefaultSchemePortResolver.INSTANCE);
}
@Override
public HttpRoute determineRoute(HttpHost target, HttpRequest request, HttpContext context)
throws HttpException {
context.setAttribute(HttpClientContext.USER_TOKEN, "constant");
return super.determineRoute(target, request, context);
}
}
}
Output on 2.44.7:
client certificate: true, mode: default -> 5 requests used 5 TCP connection(s)
client certificate: false, mode: --no-client-cert -> 5 requests used 1 TCP connection(s)
client certificate: true, mode: --constant-user-token -> 5 requests used 1 TCP connection(s)
Possible Solution
Call disableConnectionState() in ApacheHttpClient#createClient, next to the existing disableContentCompression(), disableRedirectHandling() and disableAutomaticRetries(). An ApacheHttpClient presents the identity from its single TlsKeyManagersProvider (plus at most one proxy credential). So connection-level identity tracking can't tell users apart within one client; it only prevents reuse. If you'd rather keep the check by default, a builder option to turn it off would also work.
Workaround today: set a constant HttpClientContext.USER_TOKEN in a custom HttpRoutePlanner (the --constant-user-token mode above). That doesn't work for proxied clients, because the builder rejects a route planner combined with a proxy ("The httpRoutePlanner and proxyConfiguration can't both be configured.").
Additional Information/Context
-
We hit this with an S3 client talking to an S3-compatible endpoint that authenticates clients by TLS client certificate. Under load, the per-request handshake dominated request latency.
-
Current master still creates a new context per request in ApacheHttpClient#execute and never touches connection state. Apache5HttpClient follows the same pattern (Apache5Utils.newClientContext, no connection-state handling). I read that code but didn't run the repro against the Apache 5 client.
-
ApacheClientTlsAuthTest checks that mTLS requests succeed, not that connections are reused.
-
OpenFeign hit the same root cause in Reuse HttpContext in ApacheHttpClient for pooled mTLS connections OpenFeign/feign#2114 and fixed it with disableConnectionState() or a constant user token. Stack Overflow question 63829066 walks through the mechanism.
AWS Java SDK version used
2.44.7 (apache-client, with Apache HttpClient 4.5.14 and HttpCore 4.4.16)
JDK version used
openjdk version "21.0.8" 2025-07-15 LTS (Corretto-21.0.8.9.1)
Operating System and version
Debian GNU/Linux 13 (trixie), x86_64
Activity
debora-ito commentedon Oct 12, 2026
Hi @lawrenz1 thank you for reaching out.
Can you give more context on your use case?
How you found the issue, how measurable is the latency difference (if you have benchmarks to share), what's the impacted request load, and any additional info you have.
lawrenz1 commentedon Oct 12, 2026
Thanks for looking at this. Some context:
Use case: An S3 client that presents a TLS client certificate. Each request paid a full mutual-TLS handshake.
How we found it: Under load, the client opened about 1.35 new connections per request. A new mTLS connection took about 20.8 ms to establish, versus 8.9 ms on a reused connection, so handshake cost was a significant share of request latency.
Result: Enabling connection reuse increased request rate by 2–3x under the same workload. We can't share the workload details publicly, but the repro above should be enough to confirm the behavior.