The iptables connection tracking subsystem is failing to keep up, causing legitimate network connections to be dropped and new ones to be rejected with "Too many links" errors. This happens when the system can’t track all the active network connections due to resource exhaustion.
Common Causes and Fixes
-
nf_conntrack_maxis too low:- Diagnosis: Check the current maximum connection tracking entries:
Compare this value to the current number of tracked connections:sysctl net.netfilter.nf_conntrack_max
Ifcat /proc/sys/netfilter/nf_conntrack_countnf_conntrack_countis consistently close tonf_conntrack_max, this is your problem. - Fix: Increase
nf_conntrack_max. A common starting point is1000000for busy servers.
To make this permanent, editsysctl -w net.netfilter.nf_conntrack_max=1000000/etc/sysctl.confor a file in/etc/sysctl.d/and add/modify the line:
Then runnet.netfilter.nf_conntrack_max = 1000000sysctl -pto apply. - Why it works: This directly increases the kernel’s capacity to store connection tracking information, allowing it to handle more concurrent connections.
- Diagnosis: Check the current maximum connection tracking entries:
-
nf_conntrack_tcp_looseis disabled (for TCP):- Diagnosis: Check the current value:
If it’ssysctl net.netfilter.nf_conntrack_tcp_loose0, the system is stricter about TCP state transitions. - Fix: Enable
nf_conntrack_tcp_looseby setting it to1:
Make this permanent by adding tosysctl -w net.netfilter.nf_conntrack_tcp_loose=1sysctl.confor a file insysctl.d/:
Apply withnet.netfilter.nf_conntrack_tcp_loose = 1sysctl -p. - Why it works: When
nf_conntrack_tcp_looseis enabled, the connection tracking module is more lenient with TCP state changes. This can help prevent connections from being dropped due to minor TCP state inconsistencies, especially under high load or with network intermediaries that might slightly alter TCP packet timing. It allows the conntrack module to accept packets that might be slightly out of strict TCP sequence but are still part of a valid, albeit imperfectly tracked, connection.
- Diagnosis: Check the current value:
-
nf_conntrack_tcp_max_retransis too low (for TCP):- Diagnosis: Check the current value:
A low value (default is often 3) might cause connections to be dropped prematurely if TCP retransmissions occur.sysctl net.netfilter.nf_conntrack_tcp_max_retrans - Fix: Increase
nf_conntrack_tcp_max_retrans. A value of5or10is often sufficient.
Make this permanent by adding tosysctl -w net.netfilter.nf_conntrack_tcp_max_retrans=10sysctl.confor a file insysctl.d/:
Apply withnet.netfilter.nf_conntrack_tcp_max_retrans = 10sysctl -p. - Why it works: This setting controls how many TCP retransmissions are allowed before the connection is considered dead and removed from the tracking table. Increasing it provides more tolerance for packet loss or network congestion, preventing valid connections from being prematurely terminated and freeing up conntrack entries.
- Diagnosis: Check the current value:
-
Hash Table Size (
nf_conntrack_hashsize) is too small:- Diagnosis: The hash table is used to quickly look up connection tracking entries. If it’s too small, lookups become slower, and more entries might collide, potentially leading to dropped packets or inefficient conntrack management. Check the current size:
If this value is significantly smaller thansysctl net.netfilter.nf_conntrack_hashsizenf_conntrack_max, it can be a bottleneck. The ideal size is often a power of 2 and roughly proportional tonf_conntrack_max. - Fix: Increase
nf_conntrack_hashsize. A common recommendation is to set it to a power of 2 that is at least half ofnf_conntrack_max. For example, ifnf_conntrack_maxis1000000, you might setnf_conntrack_hashsizeto65536.
Make this permanent by adding tosysctl -w net.netfilter.nf_conntrack_hashsize=65536sysctl.confor a file insysctl.d/:
Apply withnet.netfilter.nf_conntrack_hashsize = 65536sysctl -p. - Why it works: A larger hash table reduces the probability of hash collisions. This means the kernel can find existing connection tracking entries more quickly and efficiently, reducing the load on the conntrack subsystem and preventing it from becoming overwhelmed by lookups.
- Diagnosis: The hash table is used to quickly look up connection tracking entries. If it’s too small, lookups become slower, and more entries might collide, potentially leading to dropped packets or inefficient conntrack management. Check the current size:
-
UDP Connection Tracking Timeout (
nf_conntrack_udp_timeout):- Diagnosis: UDP is connectionless, but
iptablescan still track UDP "connections" for a period. If the timeout is too long, idle UDP "connections" can linger in the table, consuming resources. Check the current value:
The default is often 30 seconds.sysctl net.netfilter.nf_conntrack_udp_timeout - Fix: Decrease
nf_conntrack_udp_timeoutif you have many short-lived UDP flows. A value of15or30seconds is typical. If you have very long-lived UDP streams (e.g., some streaming protocols), you might need to increase it, but for general "too many links" issues, reducing it can help clear out stale entries.
Make this permanent by adding tosysctl -w net.netfilter.nf_conntrack_udp_timeout=30sysctl.confor a file insysctl.d/:
Apply withnet.netfilter.nf_conntrack_udp_timeout = 30sysctl -p. - Why it works: This setting dictates how long a UDP flow will be tracked after the last packet. A shorter timeout means stale UDP flows are removed from the conntrack table more quickly, freeing up memory and reducing the overall count of tracked connections, thus alleviating pressure on the system.
- Diagnosis: UDP is connectionless, but
-
Memory Pressure / Kernel OOM Killer:
- Diagnosis: If the system is generally low on memory, the kernel’s Out-Of-Memory (OOM) killer might be terminating processes, including network-related daemons or even parts of the networking stack, which can indirectly lead to connection tracking issues. Check
dmesgfor OOM killer messages:
Also, monitor overall memory usage with tools likedmesg | grep -i oomfree -hortop. - Fix: The fix here is to address the root cause of memory exhaustion, which might involve:
- Adding more RAM.
- Optimizing applications to use less memory.
- Reducing
vm.swappinessto discourage excessive swapping. - Configuring cgroups to limit memory usage for specific applications.
- If the issue is specifically with
iptablesitself consuming too much memory (less common for just tracking limits, more for complex rulesets), consider optimizing youriptablesrules.
- Why it works: By ensuring the system has sufficient available memory, you prevent the OOM killer from interfering with the
iptablesconnection tracking subsystem or other critical processes, allowing it to operate within its configured limits.
- Diagnosis: If the system is generally low on memory, the kernel’s Out-Of-Memory (OOM) killer might be terminating processes, including network-related daemons or even parts of the networking stack, which can indirectly lead to connection tracking issues. Check
After applying these fixes, you might encounter a new error if your iptables rules are excessively complex or if there’s a fundamental network configuration issue, such as a routing loop or a very high rate of malformed packets that the conntrack module is struggling to classify.