The EMFILE error means a process tried to open more files than the system allows for that process, or more than the system itself can handle in total.
This usually manifests as applications failing to start, network services becoming unresponsive, or general system instability. The core issue is that file descriptors, which represent open files, network sockets, pipes, and other I/O resources, are a finite resource. When a process exhausts its limit, or the system-wide limit, it can no longer acquire new ones.
Here are the common causes and how to fix them:
1. Per-Process File Descriptor Limit Exceeded
-
Diagnosis: Check the current limits for the user running the affected process.
sudo su - <username> -c "ulimit -n"Then, check the
Max open filessetting for the system.cat /proc/sys/fs/file-maxIf the process’s limit is significantly lower than the number of files it’s trying to open (which you might infer from application logs or by monitoring
lsof <pid> | wc -l), you need to increase it. -
Fix: Edit
/etc/security/limits.confand add or modify lines like these for the specific user or group:<username> hard nofile 65536 <username> soft nofile 65536Replace
<username>with the actual username and65536with a sufficiently high number. For system services, you might edit their specificsystemdservice file. Find the service file (e.g.,/etc/systemd/system/my_service.service) and add:[Service] LimitNOFILE=65536Then reload
systemdand restart the service:sudo systemctl daemon-reload sudo systemctl restart my_service.service -
Why it works: The
ulimitcommand andlimits.conf(orsystemd’sLimitNOFILE) directly control the maximum number of file descriptors a process can acquire. By increasing this, you give the process more room to open files. Thesoftlimit is the one actively enforced, while thehardlimit is the ceiling that a non-privileged user can raise the soft limit to.
2. System-Wide File Descriptor Limit Exceeded
-
Diagnosis: Check the current system-wide limit.
cat /proc/sys/fs/file-maxThen, check the total number of open file descriptors currently in use across the system.
cat /proc/sys/fs/file-nrThis output shows three numbers:
free_descriptors allocated_descriptors max_descriptors. Ifallocated_descriptorsis close tomax_descriptors, the system is running out. -
Fix: Temporarily increase the system-wide limit by running:
sudo sysctl -w fs.file-max=200000To make it permanent, edit
/etc/sysctl.confand add or modify the line:fs.file-max = 200000Then apply the change:
sudo sysctl -pReplace
200000with a suitable value. -
Why it works:
fs.file-maxsets the absolute maximum number of file descriptors that the entire kernel can allocate. If this limit is too low for the aggregate demand of all processes, even if individual processes are within their limits, the system will reject new file descriptor requests.
3. Leaking File Descriptors in an Application
-
Diagnosis: Identify the process causing the issue. Use
lsofto see all open files for a process, andlsof -p <pid> | wc -lto count them. If the count is unexpectedly high and growing, or if it’s consistently high, a leak is likely. Look for patterns in thelsofoutput – many identical entries (e.g., sockets to the same host, or log files) can indicate a leak.sudo lsof -p <pid> | grep '(deleted)'This command can sometimes reveal files that have been unlinked but are still held open by a process, a common symptom of leaks.
-
Fix: This is application-specific. The fix involves modifying the application’s code to ensure that every file descriptor opened (sockets, files, pipes, etc.) is properly closed when it’s no longer needed. This often means ensuring
close()is called in all relevant code paths, including error handling. For example, ensurefclose()is called forFILE*pointers, orclose()for raw file descriptors. -
Why it works: File descriptors are a finite resource. If an application opens them and never closes them, they accumulate until the process hits its per-process limit, or the system hits its global limit. Correctly closing unused descriptors frees them up for reuse.
4. Excessive Number of Network Connections (Sockets)
-
Diagnosis: Use
netstat -anp | grep ESTABLISHED | wc -lorss -sto see the number of active network connections. If a specific application is responsible, usesudo ss -tp | grep <pid>to see its connections. Many connections to the same remote host, or a large number of listening sockets, can consume file descriptors. -
Fix: Tune the application’s connection handling. For web servers, this might involve increasing
worker_processes,max_clients, orkeepalive_timeoutin Nginx/Apache. For databases, it could be adjustingmax_connections. For custom applications, review connection pooling and session management. Also, tune TCP/IP parameters related to connection reuse and timeout settings in/etc/sysctl.conf(e.g.,net.ipv4.tcp_tw_reuse,net.ipv4.tcp_fin_timeout). -
Why it works: Each established network connection, and each listening socket, consumes a file descriptor. Optimizing how applications manage these connections, or tuning the kernel’s network stack behavior, can reduce the number of descriptors required.
5. Large Number of Open Log Files or Temporary Files
-
Diagnosis: Use
lsof | grep logorlsof | grep tmpto see if a particular process is holding an unusually high number of log files or temporary files open. This can happen if logging is misconfigured or if temporary files are not being cleaned up. -
Fix:
- Logging: Configure log rotation (e.g., using
logrotate) to close and reopen log files periodically. Ensure applications themselves properly close and reopen log files if they are managing their own rotation. - Temporary Files: Implement a cleanup strategy for temporary directories (e.g.,
/tmp,/var/tmp) usingtmpwatchorsystemd-tmpfiles. Ensure applications clean up their own temporary files.
- Logging: Configure log rotation (e.g., using
-
Why it works: Similar to general file descriptor leaks, repeatedly opening new log files or temporary files without closing old ones consumes descriptors. Proper rotation and cleanup strategies ensure that descriptors associated with these files are released.
6. Incorrectly Configured inotify Watches
-
Diagnosis: While not directly "open files,"
inotifywatches also consume kernel resources that are counted towards the file descriptor limit.lsof | grep inotifyor checking/proc/sys/fs/inotify/max_user_watchescan reveal if a process is creating an excessive number of watches. -
Fix: Increase the system-wide
max_user_watcheslimit:sudo sysctl -w fs.inotify.max_user_watches=524288Make it permanent by adding to
/etc/sysctl.conf:fs.inotify.max_user_watches = 524288Then run
sudo sysctl -p. For specific applications, investigate why they are creating so many watches. -
Why it works:
inotifyis a kernel subsystem for monitoring filesystem events. Each watch uses a small amount of kernel memory and is managed similarly to a file descriptor in terms of resource accounting. An excessive number of watches, often seen with file synchronization tools or IDEs, can exhaust this limit.
After addressing these, you might encounter ENFILE errors if you’ve exhausted the system’s total available file descriptors, not just per-process limits.