iptables is dropping connections from a single IP address, and you’re getting "connection refused" errors. This isn’t a firewall rule blocking the IP outright; it’s the system’s defense mechanism against abusive or accidental over-connection, specifically hitting the iptables connection tracking limit.
Here’s what’s actually breaking: the Netfilter connection tracking module (conntrack) is designed to keep a stateful record of network connections. When an IP address establishes an excessive number of new connections within a short period, conntrack can hit its configured limit for that specific source IP. Once the limit is reached, conntrack stops tracking new connections from that IP, and iptables rules that rely on connection state (like stateful firewall rules) will drop them, resulting in connection failures.
Common Causes and Fixes
1. Exceeded conntrack Table Size for a Specific IP:
This is the most common culprit. The conntrack module has a global table size, but it also has internal mechanisms to limit the number of connections tracked per source IP to prevent resource exhaustion.
-
Diagnosis: Use
conntrackto inspect the current state. Look for entries related to the problematic IP.conntrack -S | grep <problematic_ip>You’ll likely see a high number of entries for that IP, or potentially no output if the limit is so strict it’s preventing any new entries. A more direct check is to look at the kernel’s accounting for connection tracking per IP.
sysctl net.netfilter.nf_conntrack_max_per_ipIf this value is set too low, or if the system is hitting its overall
nf_conntrack_max, it will start dropping. -
Fix: Increase the maximum number of connections tracked per IP address.
sysctl -w net.netfilter.nf_conntrack_max_per_ip=1000 echo "net.netfilter.nf_conntrack_max_per_ip = 1000" | tee -a /etc/sysctl.conf sysctl -pThis command tells the kernel to allow up to 1000 tracked connections originating from a single IP address. The
teeandsysctl -pensure this change persists across reboots.
2. Aggressive Client Behavior (e.g., Port Scanning, DDoS): A single client or a compromised machine on your network might be performing a port scan, a brute-force attack, or is part of a botnet, sending an overwhelming number of connection attempts.
-
Diagnosis: Use
iptablesto log packets from the suspected IP and then analyze the logs.# Add a logging rule *before* your established/related rules iptables -I INPUT 1 -s <problematic_ip> -j LOG --log-prefix "IPT_DROP: " --log-level 7 iptables -I INPUT 2 -s <problematic_ip> -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT # Then monitor logs tail -f /var/log/syslog | grep IPT_DROPIf you see a flood of log entries for that IP, it’s a strong indicator.
-
Fix: Implement a rate-limiting rule in
iptablesto cap the connection rate from the offending IP.iptables -I INPUT -s <problematic_ip> -m limit --limit 10/minute --limit-burst 5 -j ACCEPT iptables -I INPUT -s <problematic_ip> -j DROPThis setup allows 5 new connections immediately (
--limit-burst 5) and then limits new connections to 10 per minute (--limit 10/minute). Any subsequent connection attempts beyond this rate will be dropped. This prevents the IP from overwhelmingconntrack.
3. Buggy Application or Script:
A poorly written application or a script on the client side might be opening and closing connections in rapid succession, or failing to properly close them, leading to a build-up in the conntrack table.
-
Diagnosis: Network traffic analysis tools like
tcpdumpon the server can reveal the pattern of connections from the IP.tcpdump -ni <interface> host <problematic_ip> and port <service_port> -c 100Look for rapid SYN packets without corresponding FIN or RST, or a very high rate of connection establishment and teardown that seems unnatural.
-
Fix: The ideal fix is on the client side (fixing the application). If that’s not possible, you can use
iptablesto limit connections to the specific service port.iptables -I INPUT -p tcp --dport <service_port> -s <problematic_ip> -m limit --limit 5/minute --limit-burst 3 -j ACCEPT iptables -I INPUT -p tcp --dport <service_port> -s <problematic_ip> -j DROPThis applies rate limiting specifically to TCP connections destined for
<service_port>from the problematic IP.
4. Insufficient Global conntrack Table Size:
While the problem often manifests per IP, a generally undersized global conntrack table can lead to premature dropping of connections, which might disproportionately affect IPs that are already pushing the limits.
-
Diagnosis: Check the overall
conntracktable usage.cat /proc/sys/net/netfilter/nf_conntrack_count cat /proc/sys/net/netfilter/nf_conntrack_maxIf
nf_conntrack_countis consistently close tonf_conntrack_max, the table is full. -
Fix: Increase the global
conntracktable size.sysctl -w net.netfilter.nf_conntrack_max=262144 echo "net.netfilter.nf_conntrack_max = 262144" | tee -a /etc/sysctl.conf sysctl -pThis sets the maximum number of connection tracking entries to 262,144. The exact value depends on your server’s RAM and expected network load.
5. Stateful Firewall Rules Interfering:
Your existing iptables rules might be too aggressive in how they handle connection states, or there might be an ordering issue where a rule that should accept established connections is placed after a rule that might drop them based on a transient conntrack state.
-
Diagnosis: Review your
iptables -L -v -noutput carefully, paying attention to the order of rules and the use ofconntrackmodules (--ctstate). Ensure yourACCEPTrules forESTABLISHED,RELATEDconnections are high in the chain. -
Fix: Reorder your
iptablesrules to prioritize established connections.# Example: Ensure this rule is near the top of your INPUT chain iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT # Then add your other rules below it.Placing the
ESTABLISHED,RELATEDrule at the very top ensures that ongoing, valid connections are immediately accepted without being re-evaluated by subsequent, potentially more restrictive rules.
6. System Reboot or Service Restart:
Sometimes, after a system reboot or a restart of network-related services, conntrack might not initialize its state tables correctly, or a brief window of high connection attempts during startup can trigger the limits.
-
Diagnosis: This is less about a specific command and more about timing. If the issue appears immediately after a reboot and then subsides, this is a likely cause. Check
dmesgfor anyconntrackornf_conntrackrelated errors. -
Fix: Ensure
conntrackis properly configured to load its module and that its parameters are set correctly viasysctl.confand applied withsysctl -p. You might also consider adding a small delay before allowing external connections or implementing a rate limit at the very first packet that hits the server to smooth out startup bursts.
The next error you’ll likely encounter if you fix this problem is related to DNS resolution failures if the connection issues were impacting your DNS clients, or specific application errors if the application itself was not designed to handle transient connection drops gracefully.