The nf_conntrack table is full, and the kernel is dropping new incoming packets because it has no room to track their state.
Cause 1: Connection Explosion
Diagnosis: Check the current connection count and the table size.
sudo conntrack -S
Look for a count value that’s very close to the max value. If count is, say, 90% of max, you’ve got a connection explosion.
Fix: Increase the nf_conntrack_max kernel parameter. This is usually done in /etc/sysctl.conf.
# In /etc/sysctl.conf
net.netfilter.nf_conntrack_max = 1048576 # Example value, adjust based on your needs
Then apply the change:
sudo sysctl -p
This tells the kernel to allocate more memory for the connection tracking table, giving it more space to record new connections.
Why it works: You’re literally giving the system more buckets to put connection information into.
Cause 2: Long-Lived Connections
Diagnosis: Identify connections that are lingering for an unusually long time.
sudo conntrack -L | awk '{print $1, $2, $3, $4, $5, $6}' | sort -k6 -r | head -n 20
This command lists connections and sorts them by their timeout (the 6th field). Look for connections with very high timeout values.
Fix: Reduce the nf_conntrack_tcp_timeout_established (and potentially other *_timeout_* values for different protocols) sysctl parameter.
# In /etc/sysctl.conf
net.netfilter.nf_conntrack_tcp_timeout_established = 3600 # Default is often 24 hours (86400)
Apply the change:
sudo sysctl -p
This makes the kernel forget about established TCP connections sooner, freeing up slots in the nf_conntrack table.
Why it works: By reducing the time the kernel waits before considering a connection "dead," you actively prune the tracking table.
Cause 3: Malformed Packets / DoS Attack
Diagnosis: Look for an unusually high number of packets with invalid flags or states in your firewall logs or Netfilter statistics.
# Check Netfilter packet statistics for dropped packets related to conntrack
sudo iptables -S | grep "conntrack"
sudo iptables -v -S
Also, examine your system logs for messages indicating malformed packets or nf_conntrack errors.
Fix: Implement stricter firewall rules to drop malformed packets before they hit the conntrack module.
# Example: Drop packets with invalid TCP flags
sudo iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
sudo iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
# ... other similar rules for malformed TCP/UDP packets
This requires careful tuning to avoid dropping legitimate traffic.
Why it works: You’re preventing bad data from even entering the connection tracking system, so it doesn’t waste valuable table space.
Cause 4: DNS Amplification Attacks (UDP Flood)
Diagnosis: Monitor UDP traffic, especially to your DNS servers. A sudden, massive spike in UDP traffic, particularly from a single source or to a single destination port (like 53), can exhaust nf_conntrack.
sudo netstat -s | grep -i udp
sudo ss -s | grep -i udp
Look for high numbers of dropped UDP packets or a rapid increase in UDP traffic.
Fix: Implement rate limiting on UDP traffic, especially for DNS queries.
# Example: Limit UDP traffic on port 53 to 100 packets per second
sudo iptables -A INPUT -p udp --dport 53 -m limit --limit 100/sec --limit-burst 200 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 53 -j DROP
This throttles incoming UDP traffic, preventing a single burst from overwhelming the system.
Why it works: You’re controlling the rate at which UDP packets arrive, ensuring the nf_conntrack table can keep up.
Cause 5: Buggy Application or Kernel Module
Diagnosis: If the table fills up rapidly and seemingly randomly, a buggy application or kernel module might be creating and immediately discarding connections, or holding onto them incorrectly.
# Use conntrack to list connections and sort by timestamp (if available or inferable)
# or use tools like `tcpdump` to inspect traffic patterns associated with the filling
sudo conntrack -L | tail -n 50
sudo tcpdump -i eth0 -n -c 1000 port 80 or port 443 | grep -E 'FIN|RST'
Look for a consistent pattern of very short-lived connections or connections that never reach an established state but still occupy nf_conntrack entries.
Fix:
- Update Software: Ensure your applications and kernel are up-to-date.
- Kernel Module Unload/Reload: If a specific module is suspected, try unloading and reloading it.
- Application Restart: Restart the suspected application.
- Disable Module: If a specific Netfilter module (e.g.,
nf_conntrack_ipv6) is causing issues, you might need to disable it or recompile the kernel with specific options.
Why it works: This addresses the root cause by fixing the underlying software defect that’s mismanaging connections.
Cause 6: Incorrect nf_conntrack_tcp_loose Setting
Diagnosis: The nf_conntrack_tcp_loose setting controls how strictly Netfilter adheres to TCP state. If set to 1 (enabled), it can track connections that don’t strictly follow the TCP handshake, potentially filling the table with "half-open" or invalid states.
sudo sysctl net.netfilter.nf_conntrack_tcp_loose
If this is 1, and you suspect non-standard TCP behavior is an issue, this might be contributing.
Fix: Set nf_conntrack_tcp_loose to 0.
# In /etc/sysctl.conf
net.netfilter.nf_conntrack_tcp_loose = 0
Apply the change:
sudo sysctl -p
This forces Netfilter to be more strict about tracking only properly established TCP connections.
Why it works: It prevents the nf_conntrack table from being polluted by connections that haven’t completed the expected TCP handshake.
The next error you’ll likely encounter is a sudden increase in network latency, followed by intermittent connectivity issues for established services.