Articles

TCP Connection Lifecycle: SYN, SYN-ACK, ACK Explained

March 24, 2026 4 min read

Kafka brokers, especially when dealing with high throughput, heavily rely on efficient TCP connections. Tuning the network stack on your Kafka servers and clients can unlock significant performance gains.

Let’s see this in action. Imagine a Kafka producer sending data to a broker.

// Example Kafka Producer configuration snippet
producerConfig := sarama.NewConfig()
producerConfig.Net.MaxOpenRequests = 5 // Default is 5
producerConfig.Net.DialTimeout = 30 * time.Second // Default is 30s
producerConfig.Net.WriteTimeout = 30 * time.Second // Default is 30s

// ... producer initialization and sending logic ...

The producer establishes TCP connections to the broker. Each connection is a stream of bytes, and the efficiency of this stream directly impacts how quickly messages can be sent and received. High throughput means many such streams are active simultaneously, each demanding resources.

The core problem Kafka aims to solve is reliable, high-volume message streaming. TCP is the underlying transport, and its performance characteristics are paramount. When you see throughput bottlenecks, it’s often because the TCP connections aren’t keeping up.

Internally, Kafka uses net.Conn interfaces in Go (or similar socket abstractions in other languages) to communicate. The operating system’s TCP/IP stack handles the heavy lifting: segmenting data, managing acknowledgments (ACKs), retransmissions, and flow control. Tuning focuses on optimizing these OS-level parameters and how your application interacts with them.

You control several key levers:

  • Buffer Sizes: TCP uses send and receive buffers. If these are too small, data can get stuck, leading to lower throughput.
  • Connection Limits: The number of concurrent connections and requests per connection matters. Too few, and you underutilize resources; too many, and you risk overwhelming the OS or Kafka itself.
  • Timeouts: Network issues or slow peers can cause connections to hang. Appropriate timeouts prevent resources from being tied up indefinitely.
  • TCP Congestion Control: Algorithms like Cubic or BBR adjust sending rates based on network conditions.

Let’s dive into specific OS-level tuning. On Linux, you’ll often look at sysctl parameters.

TCP Receive Buffer: This dictates how much data the kernel can buffer for incoming TCP segments before your application reads it.

  • Diagnosis: Check current values: sysctl net.ipv4.tcp_rmem
  • Fix: Increase the maximum receive buffer size. For high throughput, a common recommendation is to set the maximum to 16MB or more.
    sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
    This sets min, default, and max buffer sizes. The kernel will dynamically adjust between min and max based on memory availability and network conditions, aiming for the default size. The 16777216 (16MB) is the critical part for high throughput.
  • Why it works: A larger receive buffer allows the kernel to accept more data from the network even if your application isn’t immediately reading it. This prevents the sender from having to slow down due to the receiver’s application buffer being full, thus maintaining higher throughput.

TCP Send Buffer: Similar to the receive buffer, but for outgoing data.

  • Diagnosis: Check current values: sysctl net.ipv4.tcp_wmem
  • Fix: Increase the maximum send buffer size. Again, 16MB is a good starting point for high throughput.
    sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
  • Why it works: A larger send buffer allows the application to write more data to the kernel before blocking. This means the application can continue preparing and sending more data without waiting for ACKs from the remote end, improving overall sending speed.

TCP Connection Backlog: This controls the queue size for incoming TCP connection requests that are still in the SYN-sent state (waiting for the handshake to complete).

  • Diagnosis: Check current values: sysctl net.core.somaxconn and sysctl net.ipv4.tcp_max_syn_backlog
  • Fix: Increase both values significantly.
    sudo sysctl -w net.core.somaxconn=4096
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096
    These values should be aligned or tcp_max_syn_backlog can be higher. The default for somaxconn is often 128, which is far too low for busy servers.
  • Why it works: With many clients connecting rapidly, the SYN backlog queue can fill up. Increasing it allows more incoming connections to be queued for processing, preventing new connection attempts from being dropped during brief bursts of activity.

TCP Keepalive: While not directly for throughput, misconfigured keepalives can lead to stale connections that aren’t cleaned up, or conversely, to connections being dropped too aggressively.

  • Diagnosis: Check current values: sysctl net.ipv4.tcp_keepalive_time, net.ipv4.tcp_keepalive_intvl, net.ipv4.tcp_keepalive_probes
  • Fix: Adjust based on your network environment. For stable networks, longer times are fine. For less stable ones, you might want faster detection of dead peers.
    # Example: Check every 60s, 5 probes, total 300s (5 mins) before considering dead
sudo sysctl -w net.ipv4.tcp_keepalive_time=300
sudo sysctl -w net.ipv4.tcp_keepalive_intvl=60
sudo sysctl -w net.ipv4.tcp_keepalive_probes=5
  • Why it works: Proper keepalive settings ensure that idle connections are eventually detected as dead and closed, freeing up resources. This is crucial in environments with many ephemeral connections or potential network disruptions.

File Descriptor Limits: Each TCP connection consumes file descriptors. High throughput implies many connections.

  • Diagnosis: Check current limits: ulimit -n (for current shell) and check /etc/security/limits.conf or systemd unit files for persistent settings.
  • Fix: Increase the open file descriptor limit for the Kafka user.
    # In /etc/security/limits.conf or a file in /etc/security/limits.d/
kafka hard nofile 100000
kafka soft nofile 100000
    Then restart Kafka.
  • Why it works: The operating system limits the number of open files (which includes network sockets) a process can have. Exceeding this limit will cause new connections or operations to fail, manifesting as various "too many open files" errors.

TCP Congestion Control Algorithm: The default algorithm might not be optimal for all network conditions.

  • Diagnosis: Check current algorithm: sysctl net.ipv4.tcp_congestion_control
  • Fix: Consider switching to BBR if your network has latency or packet loss. BBR aims to maximize throughput while minimizing latency.
    # First, ensure BBR is enabled in your kernel (usually is on modern Linux)
# sudo modprobe tcp_bbr
# sudo sysctl -w net.core.default_qdisc=fq
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
  • Why it works: BBR uses bandwidth and Round-Trip Time (RTT) measurements to control congestion, rather than just packet loss. This can lead to more stable and higher throughput, especially on lossy or high-latency links, by avoiding unnecessary retransmissions and optimizing buffer usage.

Beyond OS tuning, Kafka client and broker configurations also play a role. For example, in sarama (a popular Go Kafka client), increasing Net.MaxOpenRequests can allow a single TCP connection to handle more concurrent requests, reducing connection overhead.

producerConfig.Net.MaxOpenRequests = 10 // Example: Double the default

This allows one connection to have up to 10 outstanding requests (e.g., PINGs, PRODUCE requests) before opening a new connection.

If you tune all these parameters and still see issues, the next problem you’ll likely encounter is CPU saturation on the Kafka brokers, as they’ll be working hard to process the increased network I/O and Kafka-specific logic.

Want structured learning?

Take the full Internet Protocol Deep Dives course →