Articles

Debug iptables Rule Matching with Trace and Logging (2026)

April 18, 2026 6 min read

iptables is dropping packets, and you’re seeing mysterious network connectivity issues. The core problem is that a packet, following the defined iptables rules, is being unexpectedly discarded or rejected by the firewall. This isn’t just a simple "port blocked" scenario; it’s about understanding the path a packet takes and where it gets derailed.

Common Causes and Fixes

  1. Incorrect Rule Order: iptables processes rules sequentially. The first matching rule dictates the packet’s fate.

    • Diagnosis: Use iptables -t nat -L -v -n --line-numbers and iptables -t filter -L -v -n --line-numbers (replace nat with mangle or raw if relevant) to list rules with packet/byte counts and line numbers.
    • Fix: Reorder rules using iptables -t <table_name> -R <chain_name> <rule_number> <new_rule_spec>. For example, to move rule 5 to be rule 2 in the INPUT chain of the filter table: iptables -t filter -R INPUT 5 iptables -t filter -I INPUT 2 -p tcp --dport 22 -j ACCEPT.
    • Why it works: By placing more specific or permissive rules earlier, you ensure legitimate traffic is handled before potentially broader, blocking rules are encountered.

  2. Stateful Firewall Misunderstanding: iptables tracks connection states. If the state tracking is confused or a packet arrives out of sequence, it might be dropped.

    • Diagnosis: Examine the conntrack table. Look for unexpected entries or a high number of invalid states. Commands like conntrack -L or conntrack -S can be helpful.
    • Fix: Flush the conntrack table if it’s clearly corrupted or full of stale entries: iptables -t raw -I PREROUTING -j CT --notrack (to stop tracking specific traffic) followed by iptables -t raw -D PREROUTING (to remove the rule). For a full flush, conntrack -F followed by restarting the conntrackd service if applicable.
    • Why it works: This resets the firewall’s memory of active connections, forcing it to re-evaluate incoming packets based on current rules rather than potentially outdated state information.

  3. Logging Module Overload/Misconfiguration: The LOG target, while useful for debugging, can consume significant resources if not used judiciously, potentially leading to dropped packets if the system can’t keep up.

    • Diagnosis: Check system logs (/var/log/syslog or /var/log/messages) for excessive iptables logging messages. Monitor CPU and I/O load.
    • Fix: Limit the rate of logging using the limit module. For example, to log no more than 5 packets per minute: iptables -A INPUT -p tcp --dport 80 -j LOG --log-prefix "HTTP_TRAFFIC: " --log-level 7 --limit 5/min --limit-burst 5. Remove or disable overly aggressive logging rules.
    • Why it works: This prevents the logging subsystem from becoming a bottleneck by throttling the rate at which it processes log messages, allowing the firewall to handle actual packet forwarding more efficiently.

  4. Invalid Packet State: Packets arriving with invalid iptables states (e.g., INVALID or UNTRACKED when they shouldn’t be) can be dropped by default policies or specific rules.

    • Diagnosis: Use iptables -t filter -L INPUT -v -n and look for packets hitting rules that drop INVALID states. The trace target is invaluable here.
    • Fix: Add a rule to explicitly allow ESTABLISHED,RELATED traffic early in your chains. iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT.
    • Why it works: This rule ensures that packets belonging to already established connections or related to them (like ICMP error messages) are immediately accepted, preventing them from being incorrectly flagged as invalid and dropped.

  5. Incorrect Interface Matching: Rules might be applied to the wrong network interface, causing traffic on a different interface to be processed by an unintended rule.

    • Diagnosis: Use iptables -L -v -n and check the Interface column for each rule. The trace target can also show which interface a packet is hitting.
    • Fix: Explicitly specify the interface using -i for incoming and -o for outgoing traffic. For example, to only apply a rule to traffic coming into eth0: iptables -A INPUT -i eth0 -p tcp --dport 443 -j ACCEPT.
    • Why it works: This ensures that the rule is only evaluated for traffic arriving on or departing from the intended network segment, preventing misapplication to traffic on other interfaces.

  6. IP Address/Port Mismatch: Simple typos or outdated IP addresses/port numbers in rules will cause them to never match, leading to the packet falling through to a default drop policy.

    • Diagnosis: Carefully review all IP addresses and port numbers in your iptables rules against your network configuration and expected traffic.
    • Fix: Correct the IP address or port number in the rule. For example, to fix a rule expecting traffic on port 8080 but should be 80: iptables -t filter -R INPUT 3 iptables -t filter -A INPUT -p tcp --dport 80 -j ACCEPT.
    • Why it works: Ensures the rule’s criteria accurately reflect the actual traffic it’s intended to manage.

Debugging with TRACE

The TRACE target is the most powerful tool for understanding why a packet is being dropped. It injects a packet into the TRACE chain, which is then processed by the kernel’s packet tracing mechanism.

  1. Enable Tracing:

    # Enable tracing for a specific packet (e.g., TCP port 22)
iptables -t raw -A PREROUTING -p tcp --dport 22 -j TRACE
iptables -t filter -A INPUT -p tcp --dport 22 -j TRACE
# You might need to add TRACE to other chains as well, depending on your setup.
# For example, for NAT:
# iptables -t nat -A PREROUTING -p tcp --dport 22 -j TRACE
# iptables -t nat -A POSTROUTING -p tcp --dport 22 -j TRACE
    • Why it works: This tells iptables to flag packets matching these criteria for detailed kernel-level tracing.

  2. Generate Traffic: From a client, attempt to connect to the service (e.g., ssh user@your_server_ip).

  3. View Trace Output: The trace information appears in kernel messages, usually accessible via dmesg or by monitoring /proc/kmsg. You’ll see output like:

    [ 1234.567890] TRACE: raw:PREROUTING:policy ACCEPT:rule 1
[ 1234.567891] TRACE: nat:PREROUTING:policy ACCEPT:rule 1
[ 1234.567892] TRACE: nat:PREROUTING:policy ACCEPT:rule 2
...
[ 1234.567900] TRACE: filter:INPUT:policy ACCEPT:rule 1
[ 1234.567901] TRACE: filter:INPUT:policy ACCEPT:rule 2
[ 1234.567902] TRACE: filter:INPUT:policy ACCEPT:rule 3 (DROP all non-established)
[ 1234.567903] TRACE: filter:INPUT:policy DROP:rule 4 (DROP all)
    • Why it works: This output shows the exact chain and rule number that iptables evaluated for the packet at each step. You can follow the packet’s journey and see where it deviates from your expectation.

  4. Clean Up Tracing Rules:

    iptables -t raw -D PREROUTING -p tcp --dport 22 -j TRACE
iptables -t filter -D INPUT -p tcp --dport 22 -j TRACE
# Remove TRACE rules from other chains as well.
    • Why it works: Removes the tracing overhead once debugging is complete.

After fixing the immediate packet drop issue, your next hurdle will likely be dealing with unexpected connection resets or timeouts from services that were previously working, often due to subtle timing differences or race conditions exposed by the corrected iptables rules.

Want structured learning?

Take the full Iptables course →