The Nginx no live upstreams error means Nginx tried to send a request to one of its configured backend servers, but all of them were marked as unhealthy or unavailable at that moment.
Here’s why this happens and how to fix it, starting with the most common culprits:
1. Backend Server Actually Down or Unresponsive
This is the simplest and most frequent cause. Nginx is correctly reporting that it can’t reach your application.
- Diagnosis: From the Nginx server, try to
curlthe health check endpoint of one of your upstream servers directly. For example, if your upstream is192.168.1.100:8080and your health check is/health, run:
If this command times out or returns a non-2xx/3xx status code, the backend is the problem.curl -I http://192.168.1.100:8080/health - Fix: Start the backend application server on
192.168.1.100:8080or fix whatever is causing it to fail. - Why it works: Nginx relies on the backend to respond to health checks. If the backend isn’t running or is erroring, Nginx correctly identifies it as unavailable.
2. Incorrect Health Check Configuration in Nginx
Nginx’s health check mechanism might be too strict, or it might be checking the wrong thing.
- Diagnosis: Examine your
nginx.confor site-specific configuration file. Look for theupstreamblock and its associatedhealth_checkdirective (this is part of the Nginx Plus feature set, or requires a third-party module for open-source Nginx).- Open-source Nginx: You likely don’t have a
health_checkdirective. Nginx marks servers as down if they fail to respond to the actual proxied request within theproxy_connect_timeout,proxy_send_timeout, orproxy_read_timeoutperiods. Check these timeouts. - Nginx Plus: Look for
health_checkwithin yourupstreamblock.
Ensure theupstream backend { server 192.168.1.100:8080; server 192.168.1.101:8080; # For Nginx Plus: health_check uri=/health interval=5s fails=3 passes=2; }urimatches a valid endpoint on your backend that returns a 2xx or 3xx status code. Ensureinterval,fails, andpassesare set appropriately for your application’s responsiveness.
- Open-source Nginx: You likely don’t have a
- Fix:
- Open-source Nginx: Increase
proxy_connect_timeout,proxy_send_timeout, andproxy_read_timeoutin yourserverorhttpblock if your backend is slow to respond but otherwise functional. For example:
Then reload Nginx:http { proxy_connect_timeout 10s; proxy_send_timeout 10s; proxy_read_timeout 10s; # ... other http settings }sudo systemctl reload nginx. - Nginx Plus: Adjust the
health_checkparameters. If your backend takes 5 seconds to respond to/healthbut is otherwise fine, you might needinterval=10s fails=5 passes=2or a more permissive URI. Reload Nginx after changes.
- Open-source Nginx: Increase
- Why it works: For open-source Nginx, the timeouts define how long Nginx waits for a response before marking a server as down. For Nginx Plus, the
health_checkdirective explicitly defines the polling mechanism, and tuning its parameters makes it more forgiving of temporary backend latency.
3. Network Connectivity Issues Between Nginx and Backend
Firewalls, routing problems, or DNS issues can prevent Nginx from reaching the backend servers.
- Diagnosis: From the Nginx server, try to
pingthe IP address of your backend server. Then, try to connect to the backend’s port usingtelnetornc:
Ifping 192.168.1.100 telnet 192.168.1.100 8080 # or nc -vz 192.168.1.100 8080pingworks buttelnet/ncfails, it’s likely a firewall blocking the specific port. Ifpingfails, it’s a more fundamental network or routing issue. Also, check DNS resolution if you’re using hostnames in yourupstreamblock:dig backend.example.com. - Fix:
- Firewall: Ensure that the firewall on the backend server (e.g.,
ufw,firewalld,iptables) and any network firewalls allow traffic from the Nginx server’s IP address to the backend server’s IP and port (e.g.,8080). - Routing: Verify network routes between the Nginx and backend servers.
- DNS: If using hostnames, ensure DNS is resolving correctly on the Nginx server. Update
/etc/resolv.confor your DNS configuration.
- Firewall: Ensure that the firewall on the backend server (e.g.,
- Why it works: Nginx needs a clear network path to communicate with its upstream servers. Firewalls, incorrect routes, or faulty DNS resolution break this path.
4. Incorrect server or listen Directive on the Backend
The backend application might not be listening on the IP address or port Nginx is trying to connect to.
- Diagnosis: SSH into your backend server (
192.168.1.100). Check which ports are being listened on:
Look for a process listening onsudo ss -tulnp | grep 8080 # or sudo netstat -tulnp | grep 80800.0.0.0:8080(all interfaces) or192.168.1.100:8080(specific IP). If it’s listening on127.0.0.1:8080, it will only accept connections fromlocalhoston that machine, not from the Nginx server. - Fix: Reconfigure your backend application to listen on
0.0.0.0(all available network interfaces) or the specific IP address of the interface that Nginx will connect to (e.g.,192.168.1.100). For example, in a Node.js app:app.listen(8080, '0.0.0.0');. Restart the backend application. - Why it works: The
listendirective determines which network interfaces and ports an application binds to. If it’s bound too restrictively (e.g., to127.0.0.1), external connections won’t be accepted.
5. Nginx max_fails and fail_timeout Settings Too Aggressive
Even if a backend server is temporarily slow, Nginx might mark it as failed too quickly and for too long.
- Diagnosis: Review the
serverdirectives within yourupstreamblock in Nginx configuration.
Theupstream backend { server 192.168.1.100:8080 max_fails=3 fail_timeout=30s; server 192.168.1.101:8080 max_fails=3 fail_timeout=30s; }max_failsparameter is the number of consecutive failed requests or health check failures before a server is marked as down.fail_timeoutis the duration for which the server is considered down. If these are too low, temporary blips can cause theno live upstreamserror. - Fix: Increase
max_failsandfail_timeout. For example:
Then reload Nginx:upstream backend { server 192.168.1.100:8080 max_fails=5 fail_timeout=60s; server 192.168.1.101:8080 max_fails=5 fail_timeout=60s; }sudo systemctl reload nginx. - Why it works: These parameters control Nginx’s resilience to backend failures. Increasing them allows Nginx to tolerate more transient issues before declaring a backend server unhealthy.
6. Backend Application Crashing Under Load or Due to Errors
The backend might be up and running initially but crashes frequently when Nginx sends it traffic, especially during health checks if the health check endpoint itself is resource-intensive or buggy.
- Diagnosis: Check the logs of your backend application for errors, segmentation faults, out-of-memory errors, or unhandled exceptions. Monitor resource usage (CPU, memory) on the backend server.
- Fix: Debug and fix the root cause of the application crashes. Optimize the application, fix bugs, or increase the resources available to the backend server.
- Why it works: A crashing application is, by definition, not live. Resolving the instability makes it consistently available.
After fixing these issues, the next error you might encounter is a 502 Bad Gateway if Nginx can reach the upstream, but the upstream returns an invalid response.