The Linkerd proxy is refusing to connect to your Java application over HTTP/1.1 because the proxy is sending a Connection: close header that your application’s HTTP/1.1 server is misinterpreting.
Here are the common causes and their fixes:
Cause 1: Netty’s Default Connection: close Behavior
Diagnosis: Your Java application uses Netty as its HTTP server. By default, Netty’s HttpServerCodec can emit a Connection: close header on responses, which the Linkerd proxy then interprets as the connection being terminated.
Fix: Configure your Netty HttpServerCodec to explicitly disable Connection: close headers for HTTP/1.1.
// In your Netty server initialization
ChannelPipeline pipeline = ch.pipeline();
// ... other handlers
pipeline.addLast("codec", new HttpServerCodec(4096, 8192, 2048, false)); // The last parameter `false` disables Connection: close
// ... other handlers
Why it works: Setting the h2cEnabled parameter to false (which is the fourth parameter in HttpServerCodec’s constructor) instructs Netty not to add the Connection: close header for HTTP/1.1 connections, aligning with Linkerd’s expectations for persistent connections.
Cause 2: Spring Boot’s Embedded Tomcat/Jetty/Undertow Configuration
Diagnosis: If you’re using Spring Boot, its embedded web server (Tomcat, Jetty, or Undertow) might be configured to send Connection: close headers under certain conditions, especially when dealing with keep-alive.
Fix: For Tomcat, add the following to your application.properties or application.yml:
server.tomcat.keep-alive-timeout=0
server.tomcat.max-connections=200
For Jetty:
server.jetty.idle-timeout=0
For Undertow:
server.undertow.keep-alive-timeout=0
Why it works: Setting the keep-alive-timeout to 0 (or a very large value) effectively disables the server’s aggressive closing of idle keep-alive connections, allowing Linkerd to maintain them as intended. Adjusting max-connections can also help manage load.
Cause 3: Custom HTTP Server Libraries (e.g., Grizzly, Undertow direct usage)
Diagnosis: You’re using a Java HTTP server library directly, not through a framework like Spring Boot, and its default behavior for connection management leads to Connection: close headers being sent.
Fix: Consult the specific documentation for your HTTP server library. For example, with Grizzly, you might need to configure the HttpServer to disable sending Connection: close headers on HTTP/1.1 responses.
// Example for Grizzly
HttpServer server = HttpServer.createSimpleServer("http://localhost:8080", 1000);
// ... configure server
// Look for options related to connection management or keep-alive timeouts.
// This might involve setting specific filters or configurations on the protocol handler.
Why it works: Each library has its own mechanisms for managing HTTP connections. The goal is to find the specific configuration that prevents the server from explicitly signaling connection closure for HTTP/1.1 requests that Linkerd expects to keep alive.
Cause 4: HTTP/2 Downgrade Issues
Diagnosis: Linkerd typically prefers HTTP/2 for inter-service communication. If there’s an issue causing an HTTP/2 downgrade to HTTP/1.1 and your application’s HTTP/1.1 server isn’t configured for proper keep-alive, you might see this.
Fix: Ensure your application’s HTTP/1.1 server is robustly configured for keep-alive. If you suspect HTTP/2 is being unexpectedly downgraded, review Linkerd’s inbound and outbound proxy configurations, particularly settings related to proxy.proxy.http_version.
Why it works: By ensuring the HTTP/1.1 server is correctly handling keep-alive, you mitigate issues that arise when the protocol does fall back to HTTP/1.1, regardless of whether the downgrade was expected or not.
Cause 5: Outdated Linkerd Proxy Version
Diagnosis: Older versions of the Linkerd proxy might have had more aggressive connection closing behaviors or less robust handling of HTTP/1.1 keep-alive nuances from downstream applications.
Fix: Upgrade your Linkerd proxy to the latest stable version.
# Example for Helm
helm upgrade linkerd2 linkerd2/linkerd2 --version <latest-version> -n linkerd
Why it works: Newer versions of Linkerd include bug fixes and improvements in protocol handling, which may resolve subtle incompatibilities with various HTTP server implementations.
Cause 6: Application Logic Sending Connection: close
Diagnosis: It’s possible that your application code itself is explicitly setting the Connection: close header in its HTTP responses. This is less common for server-side frameworks but can happen with custom logic.
Fix: Review your application’s HTTP response handling code. Search for any instances where response.setHeader("Connection", "close") or equivalent is being called and remove or conditionally disable it.
Why it works: Directly removing the application’s explicit instruction to close the connection ensures that Linkerd’s expectation of a persistent connection is met.
After addressing these, you’ll likely encounter a 503 Service Unavailable error as the proxy retries connections to your application, which is a sign that the initial connection refused error has been resolved.