The Nginx upstream timed out (110: Connection timed out) error means Nginx, acting as a proxy, waited too long for a response from one of its backend servers.
Common Causes and Fixes
-
Backend Application is Slow or Unresponsive
- Diagnosis: Check your backend application logs (e.g., PHP-FPM logs, Node.js application logs, Python/Gunicorn logs) for errors or long processing times. If you can’t easily identify slow requests, enable slow query logging or request profiling in your application framework. You can also use
straceon the backend process to see where it’s spending its time:sudo strace -p $(pgrep -f your_backend_process_name) -c. - Fix: Optimize your backend code. This might involve adding indexes to database queries, caching frequently accessed data, or refactoring inefficient algorithms. For example, if a specific database query is taking 20 seconds, add a covering index to the relevant table.
- Why it works: By speeding up the backend’s execution, you reduce the time Nginx has to wait, preventing the timeout.
- Diagnosis: Check your backend application logs (e.g., PHP-FPM logs, Node.js application logs, Python/Gunicorn logs) for errors or long processing times. If you can’t easily identify slow requests, enable slow query logging or request profiling in your application framework. You can also use
-
Nginx
proxy_read_timeoutis Too Low- Diagnosis: Examine your Nginx configuration file (usually
/etc/nginx/nginx.confor files in/etc/nginx/conf.d/or/etc/nginx/sites-enabled/). Look for theproxy_read_timeoutdirective within yourlocationorproxy_passblock. The default is often 60 seconds. - Fix: Increase the
proxy_read_timeoutvalue. For example, changeproxy_read_timeout 60s;toproxy_read_timeout 180s;. Then reload Nginx:sudo systemctl reload nginx. - Why it works: This directive tells Nginx how long to wait for a response from the upstream server after the connection has been established and the request has been sent. Increasing it gives slow-running backend processes more time.
- Diagnosis: Examine your Nginx configuration file (usually
-
Nginx
proxy_connect_timeoutis Too Low- Diagnosis: Similar to
proxy_read_timeout, check your Nginx configuration forproxy_connect_timeout. This controls how long Nginx waits to establish a connection with the upstream server. The default is usually 60 seconds. - Fix: Increase
proxy_connect_timeout. For instance, changeproxy_connect_timeout 60s;toproxy_connect_timeout 10s;. Reload Nginx:sudo systemctl reload nginx. - Why it works: If the backend server is slow to accept new connections (e.g., due to high load or resource constraints), Nginx might time out before the connection is even fully established.
- Diagnosis: Similar to
-
Network Latency or Packet Loss Between Nginx and Backend
- Diagnosis: Use
pingandtraceroutefrom the Nginx server to the backend server:ping <backend_ip_address>andtraceroute <backend_ip_address>. Look for high latency (e.g., consistently over 50ms) or packet loss. Also, check firewall logs on both Nginx and backend servers for dropped packets. - Fix: Address network issues. This could involve optimizing routing, upgrading network hardware, or configuring firewalls to allow necessary traffic without excessive inspection. Ensure TCP MSS clamping is correctly configured if you suspect MTU issues:
sudo sysctl -w net.ipv4.tcp_mtu_probing=1. - Why it works: High latency or dropped packets increase the effective time it takes for Nginx to send a request and receive a response, potentially exceeding the configured timeouts.
- Diagnosis: Use
-
Backend Server Overload (CPU, Memory, Disk I/O)
- Diagnosis: Monitor the resource utilization on your backend server. Use tools like
top,htop,vmstat,iostat, anddmesg. Look for consistently high CPU usage (e.g., >90%), low free memory, or heavy disk activity. - Fix: Scale your backend resources. This could mean adding more CPU cores, increasing RAM, optimizing disk I/O (e.g., using faster drives), or horizontally scaling your application by adding more instances.
- Why it works: An overloaded backend server cannot process requests quickly enough, leading to delays that trigger Nginx timeouts.
- Diagnosis: Monitor the resource utilization on your backend server. Use tools like
-
Firewall or Security Group Blocking/Slowing Down Connections
- Diagnosis: Check firewall rules (
iptables -L -n -v,ufw status verbose) on both the Nginx and backend servers, as well as any intermediate network firewalls or cloud provider security groups. Look for rules that might be rate-limiting connections, performing deep packet inspection, or have connection tracking limits that are being hit. Temporarily disabling the firewall (if safe to do so) can help isolate the issue. - Fix: Adjust firewall rules to allow unrestricted or less restricted communication between Nginx and the backend on the relevant port. For example, if a firewall is rate-limiting, you might need to increase the limit or remove it for this specific traffic.
- Why it works: Aggressive firewall rules can interfere with the TCP handshake or delay packet transmission, contributing to timeouts.
- Diagnosis: Check firewall rules (
-
Backend Application Not Listening on the Correct Interface/Port
- Diagnosis: Verify which IP address and port your backend application is configured to listen on. Use
netstat -tulnp | grep <backend_port>on the backend server. Ensure Nginx is configured toproxy_passto that exact IP address and port. - Fix: Correct either the backend application’s listening configuration or the Nginx
proxy_passdirective. If the backend listens on127.0.0.1:8000, Nginx shouldproxy_pass http://127.0.0.1:8000;. If it listens on0.0.0.0:8000, ensure Nginx can reach that IP (e.g.,proxy_pass http://<backend_private_ip>:8000;). - Why it works: Nginx cannot connect to a backend that is not actively listening on the specified address and port.
- Diagnosis: Verify which IP address and port your backend application is configured to listen on. Use
After resolving these issues, the next error you might encounter is 502 Bad Gateway, indicating that Nginx successfully connected to the upstream but received an invalid response.