The nf_conntrack table is full, and the kernel is dropping incoming packets because it has no more space to track existing connections.
Here are the common causes and how to fix them:
Cause 1: Too Many Connections
The most frequent culprit is simply having an overwhelming number of active connections, often due to a traffic surge, a denial-of-service attack, or an application that’s opening connections too rapidly and not closing them properly.
-
Diagnosis: Check the current connection count and the table size.
conntrack -SLook for the
countvalue under thenf_conntracktable. If this number is consistently close to themaxvalue, you’ve found your problem. -
Fix: Temporarily increase the maximum number of connections.
sysctl -w net.netfilter.nf_conntrack_max=262144This command increases the
nf_conntrack_maxkernel parameter to 262,144. This gives the conntrack table more room to store connection states, allowing more connections to be tracked. You’ll want to make this permanent by addingnet.netfilter.nf_conntrack_max = 262144to/etc/sysctl.confand runningsysctl -p.
Cause 2: Long Connection Tracking Timeout
If connections are staying in the nf_conntrack table for longer than necessary, they can fill up the table even with moderate traffic. This is often seen with protocols that have long idle periods or with misconfigured timeouts.
-
Diagnosis: Examine the default timeout values for different connection types.
sysctl -a | grep nf_conntrack_tcp_timeout sysctl -a | grep nf_conntrack_udp_timeoutSpecifically, look at
net.netfilter.nf_conntrack_tcp_loose,net.netfilter.nf_conntrack_tcp_be_liberal,net.netfilter.nf_conntrack_tcp_max_retrans, and timeouts for UDP. High values here mean connections linger. -
Fix: Reduce the connection tracking timeouts. For example, to reduce the timeout for established TCP connections from 5 days (432000 seconds) to 1 hour (3600 seconds):
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=3600This tells the kernel to remove established TCP connections from the conntrack table after 3600 seconds (1 hour) of inactivity, freeing up space more quickly. Adjust UDP timeouts similarly if needed. Again, make these changes permanent in
sysctl.conf.
Cause 3: Unclean Connection States / Stuck Entries
Sometimes, connections can get stuck in the conntrack table due to network issues, application crashes, or buggy firewall rules, even if the actual connection is long gone. These stale entries consume valuable space.
-
Diagnosis: Manually inspect the conntrack table for unusual entries.
conntrack -LLook for connections that seem out of place, have been around for a very long time without activity, or are in unusual states (e.g.,
INVALID). -
Fix: Manually delete problematic entries. If you find a specific IP or port causing issues, you can delete all entries related to it:
conntrack -D --orig-src 192.168.1.100This command deletes all connection tracking entries originating from the IP address
192.168.1.100. For a more aggressive cleanup, you can periodically flush the entire table (use with extreme caution as it drops all active connections):conntrack -FThis flushes all entries from the conntrack table, freeing up all its space immediately. This is a temporary measure and doesn’t fix the underlying cause.
Cause 4: IP Address Spoofing or Invalid Packets
Packets with invalid source IP addresses or those that don’t match any existing connection state can be marked as INVALID and still consume space in the conntrack table, especially if there’s a flood of them.
-
Diagnosis: Filter conntrack entries for
INVALIDstate.conntrack -L | grep INVALID -
Fix: Implement firewall rules to drop invalid packets before they hit the conntrack table.
iptables -I INPUT -m conntrack --ctstate INVALID -j DROPThis
iptablesrule inserts a rule at the beginning of theINPUTchain that matches packets in theINVALIDconnection state and drops them. This prevents malformed or spoofed packets from occupying conntrack table slots.
Cause 5: Insufficient Table Size for the Hardware/Workload
While net.netfilter.nf_conntrack_max is the primary control, the actual physical memory available to the kernel for this purpose can also be a bottleneck on very high-traffic systems.
-
Diagnosis: Monitor system memory usage and correlate with conntrack table load.
free -h conntrack -SIf
nf_conntrack_maxis set reasonably high but the system is still running out of memory, or if theconntrack -Soutput shows a very highcount, you might need more RAM or a more efficient conntrack module. -
Fix: Increase the kernel’s
nf_conntrackmemory allocation. This is less common but can be tuned viasysctl:sysctl -w net.netfilter.nf_conntrack_max=524288And ensure your system has sufficient RAM. If you’re consistently hitting your
nf_conntrack_maxeven with reasonable timeouts and no obvious attacks, you may need to increase this value further, but be mindful of memory consumption.
Cause 6: Kernel Module Parameters for Specific Protocols
Certain protocols or network behaviors might require specific tuning of the nf_conntrack module itself, beyond just the maximum table size.
-
Diagnosis: Check module parameters.
sysctl -a | grep nf_conntrackLook for parameters like
nf_conntrack_bucketswhich determines the number of hash buckets for the conntrack table. -
Fix: Adjust
nf_conntrack_buckets. If yournf_conntrack_maxis set very high, you might also need to increasenf_conntrack_bucketsto ensure efficient lookups. The default is usuallynf_conntrack_max / 4.sysctl -w net.netfilter.nf_conntrack_buckets=131072This increases the number of hash buckets, potentially improving performance and capacity when the table is very large. This should generally be set in proportion to
nf_conntrack_max.
After addressing these, you’ll likely encounter iptables: No chain/target/match by that name. if you try to apply rules that have become obsolete or if the iptables-legacy vs iptables-nft transition hasn’t been managed correctly.