The Memcached server is rejecting new connections because its backlog queue for accepting new TCP connections is full. This typically happens when the application is creating connections faster than Memcached can accept and process them, or when the Memcached process itself is too busy to even acknowledge new connection attempts.

Cause 1: Application Connection Sprawl

Diagnosis: On the application server, run ss -tunap | grep <memcached_ip>:<memcached_port> | wc -l. If this number is consistently high (hundreds or thousands), your application is likely opening too many connections.

Fix: Implement connection pooling on the application side. Instead of opening a new connection for every request, reuse a set of established connections. For example, in Python with python-memcached, you might initialize the client once and reuse it:

from memcache import Client
mc = Client(['<memcached_ip>:11211'], socket_timeout=1.0, dead_timeout=300)
# ... later in your code, use mc to set/get

This works because connection pooling reduces the rate at which new connection requests hit Memcached, preventing the backlog from overflowing.

Cause 2: Memcached max_connections Too Low

Diagnosis: Check your Memcached startup configuration or systemd service file. Look for the -c or --max_connections flag. If it’s set to a low value (e.g., 1024), it might be too small for your workload.

Fix: Increase the max_connections value. For example, in your systemd service file /etc/systemd/system/memcached.service, modify the ExecStart line:

ExecStart=/usr/bin/memcached -m 64 -p 11211 -c 4096

Then reload the systemd daemon and restart Memcached:

sudo systemctl daemon-reload
sudo systemctl restart memcached

This increases the number of simultaneous connections Memcached can handle, giving it more breathing room.

Cause 3: High CPU Utilization on Memcached Server

Diagnosis: On the Memcached server, run top or htop and observe the %CPU usage for the memcached process. If it’s consistently near 100%, Memcached is struggling to keep up.

Fix:

  1. Optimize application queries: If you see frequent get operations for the same keys, consider using mget (multiple gets) to fetch them in a single round trip.
  2. Increase Memcached server resources: Add more CPU cores or upgrade to a faster CPU.
  3. Distribute Memcached: If possible, spread your cache across multiple Memcached instances to reduce the load on any single server.

High CPU means Memcached is spending too much time processing existing requests, delaying its ability to accept new ones.

Cause 4: Network Saturation or Latency

Diagnosis: On the Memcached server, run iftop -i <interface_name> (e.g., iftop -i eth0) to monitor real-time network traffic. If the bandwidth to/from the Memcached server is maxed out, network congestion could be the culprit. Also, ping <app_server_ip> from the Memcached server and ping <memcached_ip> from the app server to check for high latency or packet loss.

Fix:

  1. Increase network bandwidth: Upgrade your network interface cards or switch ports.
  2. Use a dedicated network: If Memcached traffic shares a network with other high-bandwidth applications, move it to a separate, dedicated network.
  3. Reduce data transfer: Optimize your application to store smaller objects in Memcached or to retrieve only necessary data.

Network bottlenecks can delay the TCP handshake and data packets, making it appear to Memcached that connections are arriving faster than they actually are, or preventing new connection packets from being processed promptly.

Cause 5: Memcached listen Queue Size (Kernel Level)

Diagnosis: On the Memcached server, check the current listen queue size: sysctl net.core.somaxconn. If this value is low (e.g., 128 or 1024) and your application is opening connections very rapidly, it might be the bottleneck.

Fix: Increase the net.core.somaxconn kernel parameter. Edit /etc/sysctl.conf and add or modify the line:

net.core.somaxconn = 4096

Apply the change immediately with sudo sysctl -p. This increases the maximum backlog queue size the kernel will allow for TCP connections, giving Memcached more buffer space to queue incoming connection requests before it processes them.

Cause 6: Slow Application Processing of Memcached Responses

Diagnosis: Monitor the response times of your application’s Memcached client library. Many libraries have built-in metrics or can be instrumented. If application threads are spending a lot of time waiting for Memcached responses, even if Memcached itself is responsive, this can indirectly lead to connection issues by tying up application resources that might otherwise manage connections.

Fix:

  1. Optimize application logic: Review the code that consumes Memcached data. Are there complex computations or I/O operations happening after retrieving data from Memcached?
  2. Increase application server resources: If the application servers are CPU or memory bound, they may not be able to process Memcached responses quickly enough.

When the application is slow to process responses, it can lead to a buildup of pending requests on the application side, which might manifest as a perceived need for more connections or a delay in releasing them, indirectly contributing to the connection pressure on Memcached.

You’ll likely hit "Memcached: Out of memory" next if your cache is growing too large.

Want structured learning?

Take the full Memcached course →