[Errno 24] Too many open files: fix the right limit
[Errno 24] Too many open files means one specific counter ran out — the number of file descriptors this process is allowed to hold at once — and the fix is a limit change in whichever layer imposed the lowest ceiling, not a code rewrite. The trouble is that four different failures wear the same sentence, and two of them print a number that has nothing to do with the real ceiling, so people raise ulimit -n, restart the service, and watch it fail at exactly the same place.
TL;DR: read /proc/<pid>/limits first — it shows the limit the running process actually obeys, which is frequently not the one your shell reports. A bare ulimit -n 16 lowers the hard limit too, so you cannot raise it back without root (bash answers cannot modify limit: Invalid argument). And Node.js raises its own soft limit to the hard limit at startup, so lowering ulimit -n before node server.js does nothing at all — measured below, 32 in, 524288 in the process.
Everything below was measured on this host: Linux 6.17, Python 3.13.5, Node v26.5.1, bash 5.2, uid 10000, soft RLIMIT_NOFILE 4096 / hard 524288, fs.file-max 65535. Every quoted block is pasted from a run of the exact code shown above it. The container and systemd configurations at the end are labelled doc-cited: this host has no Docker daemon and no systemd, so those files were read, not executed.
Four failures, one sentence
Before touching a limit, identify which of these you are looking at. They have different causes, different blast radii, and only one of them is fixed by raising nofile.
| Symptom | errno / code | Scope | What is exhausted |
|---|---|---|---|
Too many open files | EMFILE = 24 | one process | RLIMIT_NOFILE soft limit of that process |
Too many open files in system | ENFILE = 23 | whole host | fs.file-max, the kernel's global file table |
ValueError: filedescriptor out of range in select() | no errno — userland | one call site | the compiled-in FD_SETSIZE of 1024 |
| silent stall, client sees a healthy TCP handshake | EMFILE inside accept() | one listener | fds available to the accepting process |
The distinction between the first two is the one that wastes the most time, because the messages differ by a single word. EMFILE is per-process and you can fix it yourself up to the hard limit. ENFILE is system-wide and no ulimit in your shell will help — the kernel file table is full for everyone, usually because some other process on the box leaked descriptors. The file that settles it is /proc/sys/fs/file-nr: its first field is the number of allocated file handles and its third field is the ceiling. If those two are equal, or within a few hundred of each other, your problem is not your service.
The last two rows are the ones that produce articles like this one. select() fails with a Python-level ValueError that never reaches the kernel, so no log line about file descriptors appears anywhere. And a server that hits EMFILE while accepting leaves the connection in the backlog with the handshake already completed — the client's connect() returns success and then the request hangs until timeout.
Read the limit that actually applies
Your shell's ulimit is a snapshot of the shell. A process started by systemd, by a container runtime, by cron, or by a supervisor may have been handed a completely different pair of numbers. The authority is /proc/<pid>/limits, all lowercase, no S — the file is limits, and it works for any process owned by your uid:
echo "soft limit: $(ulimit -Sn)"
echo "hard limit: $(ulimit -Hn)"
echo "--- the authority for a running process ---"
grep -E 'Max open files' /proc/self/limits
echo "--- system-wide ceilings ---"
echo "fs.file-max = $(cat /proc/sys/fs/file-max)"
echo "fs.nr_open = $(cat /proc/sys/fs/nr_open)"soft limit: 4096
hard limit: 524288
--- the authority for a running process ---
Max open files 4096 524288 files
--- system-wide ceilings ---
fs.file-max = 65535
fs.nr_open = 1048576Three numbers matter and they are not the same number. fs.nr_open (1048576 here) is the ceiling for a single process: no setrlimit() call can push a hard limit above it, even as root. fs.file-max (65535 here) is the ceiling for the whole kernel. The soft limit is what your program is stopped by today; the hard limit is the highest value an unprivileged process may set the soft limit to.
To read a process you do not own, use prlimit, which is part of util-linux and needed no privileges on this host for /proc/self:
prlimit --pid $$ --nofile
cat /proc/$$/limits | grep -E 'Max open files'For a container, docker inspect --format '{{.HostConfig.Ulimits}}' <name> returns the pair that was applied at creation, and it is authoritative over anything you set with ulimit inside the shell you docker exec into — a new shell in a running container has the same limits, so ulimit -n there changes nothing for PID 1.
Reproduce it on demand
You cannot debug a limit you cannot trigger. This is the shortest deterministic reproduction: set the soft limit low inside the process itself, then allocate descriptors until the kernel refuses. It finishes in milliseconds and touches nothing outside its own fd table.
import os
import resource
import socket
resource.setrlimit(resource.RLIMIT_NOFILE, (16, 524288))
print("soft=%d hard=%d" % resource.getrlimit(resource.RLIMIT_NOFILE))
held = []
try:
while True:
held.append(os.open("/etc/hostname", os.O_RDONLY))
except OSError as exc:
print("open() -> %s: %s" % (type(exc).__name__, exc))
print("open() -> errno=%s strerror=%r" % (exc.errno, exc.strerror))
print("fds opened before the ceiling: %d" % len(held))
try:
socket.socket()
except OSError as exc:
print("socket() -> %s: %s (errno=%s)" % (type(exc).__name__, exc, exc.errno))
for fd in held:
os.close(fd)
print("open() after cleanup ->", os.open("/etc/hostname", os.O_RDONLY))soft=16 hard=524288
open() -> OSError: [Errno 24] Too many open files: '/etc/hostname'
open() -> errno=24 strerror='Too many open files'
fds opened before the ceiling: 13
socket() -> OSError: [Errno 24] Too many open files (errno=24)
open() after cleanup -> 3Two details in that output are worth memorising. The first is 13: with a soft limit of 16, the process held fds 0, 1, 2 (stdin, stdout, stderr) plus the 13 it opened — the ceiling counts descriptors, not files, so a socket, a pipe, an eventfd and an epoll handle all consume the same budget as an open file. The second is the shape of the message: open() names the path it failed on, socket() has no path to name and prints a bare errno. When you grep logs for [Errno 24], the version without a filename is the one that came from a socket or a pipe, which is where leaks usually live.
There is a nastier variant. During a lazy import at the fd ceiling, the error names a file that has nothing to do with your code:
connect() -> errno=24 [Errno 24] Too many open files: '/usr/lib/python3.13/encodings/idna.py'That is the same limit, reported from the import machinery inside the socket library. If you see Python blaming a stdlib .py file for Too many open files, the interpreter is out of descriptors, not missing a module.
Raising it: what works without privileges
The rules are small and mostly undocumented in one place. An unprivileged process may lower both limits freely, may raise the soft limit up to the current hard limit, and may not raise the hard limit at all — that needs CAP_SYS_RESOURCE. The trap is that bash's ulimit -n without a flag sets both limits at once.
echo "start: soft=$(ulimit -Sn) hard=$(ulimit -Hn)"
( ulimit -S -n 65536; echo " ulimit -S -n 65536 -> soft=$(ulimit -Sn) hard=$(ulimit -Hn)" )
( ulimit -n 16
echo " ulimit -n 16 -> soft=$(ulimit -Sn) hard=$(ulimit -Hn)"
m=$(ulimit -S -n 4096 2>&1); echo " raise back -> ${m#*ulimit: }" )
( m=$(ulimit -H -n 1048576 2>&1); echo " raise hard -> ${m#*ulimit: }" )start: soft=4096 hard=524288
ulimit -S -n 65536 -> soft=65536 hard=524288
ulimit -n 16 -> soft=16 hard=16
raise back -> open files: cannot modify limit: Invalid argument
raise hard -> open files: cannot modify limit: Operation not permittedThat wording is bash's, and it is the reason the block above is labelled language-bash. The identical commands under /bin/sh — dash on Debian and Ubuntu, and the shell you land in with docker exec on most slim images — report the same two failures with different words (this host links /bin/sh -> dash):
raise back -> ulimit: error setting limit (Invalid argument)
raise hard -> ulimit: error setting limit (Operation not permitted)Dash prefixes each line with the script name and line number, for example check_sh.sh: 5:, and never uses the phrase cannot modify limit. Same errno, same cause, different string — so a log search built on bash's wording finds nothing inside a sh-based container. If you parse that output, pin the interpreter.
The second line is the trap: ulimit -n 16 left the hard limit at 16 as well, so the shell can never climb back up — even though the kernel's fs.nr_open is 1048576 and the process is nowhere near it. The message is Invalid argument (EINVAL), not Operation not permitted (EPERM), because the requested soft limit would sit above the hard limit and the kernel rejects the shape of the request before checking privileges. Use ulimit -S -n for the soft limit and ulimit -H -n for the hard limit, always.
Below both bash and Python there is one syscall, setrlimit(RLIMIT_NOFILE, …). Called directly through libc with ctypes, the raw returns on this host were:
| Request | Result | Written as |
|---|---|---|
setrlimit(8192, 8192) | rc=0 errno=0 | lowering is always allowed |
setrlimit(8192, 600000) | rc=-1 errno=1 | Operation not permitted |
setrlimit(2000000, 2000000) | rc=-1 errno=1 | Operation not permitted — EPERM is returned before the nr_open check |
Python wraps the same call and pre-empts it: resource.setrlimit(RLIMIT_NOFILE, (8192, 600000)) raises ValueError: not allowed to raise maximum limit without ever entering the kernel, which is why you never see EPERM from a Python service that misconfigures its own limits. The failure mode is a ValueError during startup, which reads like a bug in the code rather than a permission boundary.
Where the value comes from in a real deployment, in order of who wins: the cgroup/container runtime applies its own limits at creation, then the supervisor (systemd, s6, supervisord) applies LimitNOFILE= or its equivalent, then the process inherits down to whatever shells it spawns. Setting ulimit -n in a profile script only affects interactive logins. For the container layer, the runtime flag is the one to use, for example docker run --ulimit nofile=65536:65536 … — labelled doc-cited here, because this host has no Docker daemon to run it against. The same applies to a systemd drop-in, which is read from disk at unit start and cannot be changed for a running process:
# /etc/systemd/system/myapp.service.d/nofile.conf [doc-cited, no systemd on this host]
[Service]
LimitNOFILE=65536Both of those raise the hard limit for the new process, which is what you need for the runtime in the next section.
Your runtime may have already raised the limit for you
This is the part that breaks the usual advice. The two most common runtimes on a Linux box do not treat RLIMIT_NOFILE the same way. Both children in the test below are exec'd with a soft limit of 32 and a hard limit of 524288; each one then reads its own /proc/self/limits:
import os
import resource
import subprocess
import tempfile
PY = ("print('python :', next(l.strip() for l in open('/proc/self/limits')"
" if 'Max open files' in l))")
JS = ("const fs=require('fs');"
"console.log('node :',"
"fs.readFileSync('/proc/self/limits','utf8')"
".split('\\n').find(l=>l.includes('Max open files')).trim());")
path = os.path.join(tempfile.mkdtemp(), "limits.cjs")
with open(path, "w") as fh:
fh.write(JS)
def lower():
resource.setrlimit(resource.RLIMIT_NOFILE, (32, 524288))
print("each child is exec'd with soft=32 hard=524288")
for cmd in (["python3", "-c", PY], ["node", path]):
out = subprocess.run(cmd, capture_output=True, text=True, preexec_fn=lower)
print(out.stdout.strip() or out.stderr.strip())each child is exec'd with soft=32 hard=524288
python : Max open files 32 524288 files
node : Max open files 524288 524288 filesPython obeyed the limit it was given. Node ignored it and raised its own soft limit to the hard limit during startup, because libuv's platform initialisation does exactly that. The practical consequence: on a Node service, the number you set with ulimit -S -n in a shell or an init script is irrelevant, and the number that matters is the hard limit — which means LimitNOFILE= in a systemd unit drop-in or --ulimit nofile= at container creation, not a ulimit line in a startup wrapper. It also explains the reverse complaint: a Node process that suddenly opens thousands of descriptors after a deploy is not ignoring your configuration, it is using a budget that was always there. If you need to cap a Node service, test the cap against a live process, not against your shell.
This asymmetry is worth knowing for timing too. Under a soft limit of 32, Node still managed hundreds of openSync calls in a bounded loop, because it had already raised its ceiling to 524288 by the time user code ran. Python, in the same harness, fails at the 13th open.
The 1024-descriptor wall in select()
A separate ceiling hides in userland. select() takes an fd_set, a bitset whose size is fixed at compile time by FD_SETSIZE, which on glibc is 1024. An fd number of 1024 or higher does not produce an errno at all — CPython checks the range before the syscall and raises ValueError. That is why the error never appears in dmesg, never produces an [Errno 24], and usually gets logged as an application bug.
import os
import resource
import select
resource.setrlimit(resource.RLIMIT_NOFILE, (8192, 524288))
held = [os.open("/etc/hostname", os.O_RDONLY) for _ in range(1050)]
high = max(held)
print("highest fd number in use: %d" % high)
try:
select.select([high], [], [], 0)
except ValueError as exc:
print("select() -> ValueError: %s" % exc)
poll = select.poll()
poll.register(high, select.POLLIN)
print("poll() -> ok, events=%s" % poll.poll(0))
for fd in held:
os.close(fd)highest fd number in use: 1052
select() -> ValueError: filedescriptor out of range in select()
poll() -> ok, events=[(1052, 1)]The fix is select.poll(), which passes an array of pollfd structures to the kernel instead of a fixed bitset, so fd numbers are unbounded. Two operational notes from the same run. First, this failure needs no high limit at all: any process that has opened 1024 things at some point will eventually hold a descriptor above 1023, and under a soft limit of 4096 that is entirely normal. Second, select.epoll() is the other correct choice, but this host's seccomp policy denies it — select.epoll() returned PermissionError: [Errno 1] Operation not permitted — so poll() is what the block above uses and what should be in your code. If you are porting from select to an event loop and you only change the import, check that the library you picked is not backed by select underneath: several old single-file HTTP servers are, and they inherit the same wall.
Server-side EMFILE does not tell the client
The worst variant is the one where nothing is logged as a failure. A listener that runs out of descriptors while calling accept() does not lose the client's connection — the TCP handshake already completed, and the connection waits in the accept queue. The client sees a successful connect(), sends its request, and waits. Here is the whole sequence reproduced deterministically, with no threads and no timing involved:
import os
import resource
import socket
resource.setrlimit(resource.RLIMIT_NOFILE, (32, 524288))
srv = socket.socket()
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("127.0.0.1", 0))
srv.listen(64)
client = socket.create_connection(("127.0.0.1", srv.getsockname()[1]), timeout=2)
print("client connect() succeeded; queued in the backlog, not yet accepted")
held = []
try:
while True:
held.append(os.open("/etc/hostname", os.O_RDONLY))
except OSError as exc:
print("fd table full (%d fds held): %s" % (len(held), exc))
try:
srv.accept()
except OSError as exc:
print("accept() -> %s: %s (errno=%s)" % (type(exc).__name__, exc, exc.errno))
os.close(held.pop())
srv.accept()
print("after freeing one fd, accept() returned a connection")
client.close()
for fd in held:
os.close(fd)
srv.close()client connect() succeeded; queued in the backlog, not yet accepted
fd table full (27 fds held): [Errno 24] Too many open files: '/etc/hostname'
accept() -> OSError: [Errno 24] Too many open files (errno=24)
after freeing one fd, accept() returned a connectionThree things follow from that transcript. The listener survives and resumes the moment a descriptor is freed, so a service in this state looks healthy in systemctl status and answers the next request after the storm. The queued connection is not lost — it is served late, after whichever request happened to occupy the descriptor. And the backlog is a buffer, not a queue guarantee: once listen(64)'s queue fills as well, further SYNs are dropped and clients see a retry-then-timeout rather than a refusal. If you own the service, this is the moment to check ss -lnt for a listener with a full Recv-Q: a listen socket whose Recv-Q equals its backlog is the signature of a process that cannot accept fast enough or has run out of descriptors.
Find the leak before raising anything
Raising the limit is a stopgap: a process that leaks descriptors will fill any budget you give it, and it will do so more slowly, which is worse, because the failure moves from "during the incident" to "three weeks later at 04:00". The inventory command that needs no lsof and no psutil works on /proc alone. For a process holding sockets and files, the one-liner is:
ls -l /proc/$$/fd 2>/dev/null | awk '/ -> /{print $NF}' | sed -E 's/:\[[0-9]+\]$//' | sort | uniq -c | sort -rn | head
ls /proc/$$/fd | wc -lReplace $$ with the pid you are investigating. The first command groups descriptors by what they point at — socket, pipe, anon_inode, or a path — and the second counts them. Three details in those two lines are deliberate: /proc/$$/fd is listed with -l so the symlink target is visible, the filter / -> / drops ls's total 0 header, and the 2>/dev/null hides a race — the listing's own descriptor can vanish between the directory read and the stat, which prints ls: cannot access '/proc/…/fd/4': No such file or directory and looks like a problem when it is not. For socket and pipe targets the kernel writes socket:[12345], so collapsing by type needs sed -E 's/:\[[0-9]+\]$//'; a pattern without the closing bracket matches nothing and leaves you with one histogram row per descriptor. A count that climbs while request volume is flat is a leak, and the grouped list tells you which kind: a growing block of socket entries is an unclosed client connection, a growing block of anon_inode is usually an event loop registration that was never unregistered, and a growing block of paths ending in .log is a rotating file handle that nobody closed. Compare the count against Max open files from /proc/$pid/limits and you know how long you have.
Two more readings that come from the same directory and cost nothing. ls -l /proc/$pid/fd | grep -c 'socket:' against the value ss -s reports for that process tells you whether the sockets are your own outbound connections or inbound traffic. And for a container, the descriptors of PID 1 are visible from the host as ls -l /proc/$(docker inspect -f '{{.State.Pid}}' name)/fd, which is the only reliable way to count them without adding an agent.
Checklist
Order matters, because two of these steps can hide the real one:
- Read
/proc/<pid>/limitsfor the running process, notulimitin your shell. - Distinguish
EMFILE(24, per-process) fromENFILE(23, host-wide) in the log line — the host-wide case is not yours to raise. - Reproduce with a low soft limit in a throwaway process before changing production.
- Raise the soft limit with
ulimit -S -n, and never with a bareulimit -n, which also lowers the hard limit. - Put the permanent value in the layer that owns the process:
LimitNOFILE=in a systemd drop-in, or--ulimit nofile=at container creation. - For a Node service, set the hard limit — Node raises its soft limit to it at startup, so soft-limit advice does not apply.
- Replace
select()withpoll()before blaming the kernel for aValueError. - Count descriptors over time and fix the leak, or the same incident returns on a longer timer.
Two related per-process ceilings behave the same way and are worth reading next. The inotify watch limit fails as System limit for number of file watchers reached inside Docker and has its own arithmetic for the per-instance versus per-user counters; the task limit produces zombie processes instead of an errno. Both are the same class of mistake as this one: the number that stops you is not the number you were watching.
Noah Lindqvist writes about independent infrastructure operations: kernel limits, supervisors, and the failure modes that only appear on hardware you own.