Nginx is failing to establish an SSL/TLS connection with a client because the client’s SSL handshake is timing out while waiting for the server’s "Server Hello" message.
Here’s a breakdown of the common culprits and how to fix them:
1. Client-Side TLS Version Mismatch or Unsupported Cipher Suite:
-
Diagnosis: Check your Nginx error logs (
/var/log/nginx/error.logby default). Look for messages likeSSL_do_handshake() failed (SSL: error:141A906E:SSL routines:tls_process_server_hello:first record does not look like a TLS recordorSSL_do_handshake() failed (SSL: error:141A9073:SSL routines:tls_process_server_hello:record header error). On the client side, you can useopenssl s_client -connect your_domain.com:443 -tls1_2or-tls1_3to test specific TLS versions. If these fail, try a different version. -
Fix: Ensure your Nginx configuration explicitly allows modern TLS versions and strong cipher suites. In your
nginx.confor relevantserverblock:ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384'; ssl_prefer_server_ciphers off; -
Why it works: Some older clients or misconfigured systems might try to connect using TLSv1.0 or TLSv1.1, which are deprecated and insecure. By explicitly defining
TLSv1.2andTLSv1.3, you ensure Nginx only negotiates with clients supporting these versions. Similarly, specifying strong cipher suites prevents negotiation with weak or outdated encryption algorithms.ssl_prefer_server_ciphers off;is often recommended with TLSv1.3 as the client’s preference is generally better.
2. Firewall Blocking or Interfering with SSL Handshake:
- Diagnosis: A firewall (either on the server, a network device, or even the client’s local firewall) might be inspecting or blocking the initial SSL handshake packets. Use
tcpdumpon the server to see if SYN packets for port 443 are arriving and if any RST (reset) packets are sent back.
If you see SYN packets but no subsequent handshake packets or RSTs, the firewall might be dropping them.sudo tcpdump -i eth0 'tcp port 443 and tcp[tcpflags] & tcp-syn != 0' -n - Fix: If a network firewall is the culprit, ensure port 443 (TCP) is open and that no Intrusion Prevention System (IPS) or deep packet inspection is interfering with the SSL handshake. If it’s a server-side firewall like
ufworfirewalld:# For ufw sudo ufw allow 443/tcp # For firewalld sudo firewall-cmd --permanent --add-service=https sudo firewall-cmd --reload - Why it works: Firewalls can be configured to block or alter traffic. Explicitly allowing TCP traffic on port 443 ensures that the handshake packets can reach Nginx and that Nginx’s responses can get back to the client without being dropped or modified.
3. Incorrect SSL Certificate or Key Configuration:
- Diagnosis: While less likely to cause a "Waiting for Server Hello" error directly (more common for "certificate verify failed" or "no shared cipher"), an improperly configured certificate path or an invalid key can sometimes lead to handshake issues. Verify your certificate and key files exist and are readable by the Nginx user.
Ensure the permissions are something likels -l /etc/nginx/ssl/your_domain.crt ls -l /etc/nginx/ssl/your_domain.key644for the certificate and600for the key. - Fix: Double-check your
ssl_certificateandssl_certificate_keydirectives in your Nginxserverblock to ensure they point to the correct, valid certificate and private key files.
If you are using a certificate chain, ensure it’s correctly concatenated in thessl_certificate /etc/nginx/ssl/your_domain.crt; ssl_certificate_key /etc/nginx/ssl/your_domain.key;ssl_certificatefile. - Why it works: Nginx needs valid certificate and key files to present to the client during the handshake. If these are missing, corrupted, or misconfigured, Nginx cannot complete its part of the handshake, leading to timeouts.
4. Server Resource Exhaustion (CPU/Memory/File Descriptors):
- Diagnosis: If the server is overloaded, Nginx workers might not be able to process incoming SSL handshake requests in a timely manner. Check system resource usage with
top,htop, orvmstat. Look for high CPU utilization, low free memory, or a high number of open file descriptors.
Compare this to your system’s# Check file descriptors sudo lsof -p $(pgrep nginx) | wc -lulimit -nfor the Nginx user. - Fix:
- Increase worker processes: In
nginx.conf:worker_processes auto; # or a specific number like 4 - Adjust
worker_connections: Innginx.confwithin theeventsblock:events { worker_connections 4096; # Default is often 1024 } - Increase file descriptor limit: Edit
/etc/security/limits.confand add:
Then, ensure Nginx is started with these limits (often requires systemd unit file modification or re-login for the Nginx user if not using systemd).* soft nofile 65536 * hard nofile 65536
- Increase worker processes: In
- Why it works: SSL handshakes, while quick, do consume CPU and memory. If Nginx workers are already busy or hitting limits on open connections, new handshake requests can get queued indefinitely, leading to the timeout. Increasing worker processes and connections allows Nginx to handle more concurrent requests. Raising file descriptor limits ensures Nginx can open enough sockets for all its connections.
5. SNI (Server Name Indication) Issues:
- Diagnosis: If you host multiple SSL-enabled sites on the same IP address, SNI is crucial. If the client is not sending the correct
Hostheader or if Nginx is not configured to handle SNI properly, it might present the wrong certificate or fail to find a matching server block. Check Nginx logs for any mention of SNI or missingserver_name. - Fix: Ensure all your
serverblocks that listen on port 443 have aserver_namedirective, and that thessl_certificateandssl_certificate_keyare correctly specified for each.
For clients that don’t support SNI (very old ones), you’d need a separate IP address or a default server block.server { listen 443 ssl; server_name example.com; ssl_certificate /etc/nginx/ssl/example.com.crt; ssl_certificate_key /etc/nginx/ssl/example.com.key; # ... other config } server { listen 443 ssl; server_name another.com; ssl_certificate /etc/nginx/ssl/another.com.crt; ssl_certificate_key /etc/nginx/ssl/another.com.key; # ... other config } - Why it works: SNI allows a server to present different SSL certificates based on the hostname the client requests. If this mechanism fails, Nginx might not know which certificate to use, causing the handshake to stall.
6. Outdated OpenSSL Library or Nginx Version:
- Diagnosis: Very rarely, bugs in older versions of OpenSSL or Nginx could cause handshake issues, especially with newer TLS features or cipher suites. Check your installed versions:
openssl version nginx -v - Fix: Update your server’s operating system packages, including OpenSSL and Nginx, to the latest stable versions.
# Example for Debian/Ubuntu sudo apt update && sudo apt upgrade # Example for CentOS/RHEL sudo yum update - Why it works: Updates often contain bug fixes and security patches that can resolve subtle interoperability issues with SSL/TLS protocols.
After fixing these, the next error you might encounter is a 502 Bad Gateway if your backend application servers are not configured to handle HTTPS, or if there’s an issue with the proxy_pass configuration.