Too many open files (EMFILE): ulimit, systemd, Docker
Too many open files (EMFILE): ulimit, systemd, Docker
Too many open files — errno 24, EMFILE — almost never means the process opened too many files. It means the kernel refused one more file descriptor because a counter crossed a ceiling that was set somewhere you were not looking: a login shell you never used, a systemd unit you never edited, or a daemon default inherited by a container. Sockets, pipes, epoll instances, eventfd and inotify descriptors all count against the same number, which is why a service that reads no files at all can still die under load with this message in its log.
Everything below was executed on one arm64 Linux host (kernel 6.17.0-1019-oracle, containerised, uid 10000, Python 3.13.5, Node v26.5.1, no systemd, no reachable Docker daemon). Each transcript is the unedited stdout of the command printed directly above it. Where a limit can only be observed on a running systemd or a live Docker daemon, the section says so and quotes the manual instead of inventing a measurement — the host fingerprint is part of the evidence, not decoration.
TL;DR — four different limits, one message
- The enforced number is the soft limit (
RLIMIT_NOFILE), and it is per process. The hard limit is only the ceiling the soft limit may be raised to. - Measured here: soft 4096, hard 524288. A process that opens 4096 descriptors gets
[Errno 24] Too many open fileseven though the machine has 65535 descriptors left system-wide. EMFILEis per process;ENFILEis system-wide (/proc/sys/fs/file-max). The text is identical, the fix is not.- Raising the soft limit is free; raising the hard limit is not. Measured: an unprivileged process raising its hard limit gets
EPERM (errno 1); a soft limit above the hard limit getsEINVAL (errno 22). NoCAP_SYS_RESOURCE, no deal. ulimit -nin your shell proves nothing about your service. A systemd unit's limit comes fromLimitNOFILE=, or fromDefaultLimitNOFILE=, which "defaults to 1024:524288" (systemd-system.conf(5)).- In a container the limit comes from the runtime. Docker's
--ulimit nofile=soft:hardexists precisely because "setting ulimit settings in a container requires extra privileges not available in the default container" (Docker docs). - Then fix the leak. The counter proves how far you got, not why. A bounded batch turns 300 opens into a constant four descriptors — measured below.
What the kernel is actually counting
Every open file, socket, pipe, epoll handle and event loop backend is one entry in the process's file descriptor table, and open(2), socket(2), pipe(2), epoll_create(2) and eventfd(2) all allocate from the same pool. When the pool is exhausted the call returns -1 with errno 24, and the message you see is whatever your language runtime decided to print around it.
Python says it plainly. This probe lowers the soft limit to 64 for its own process only, then opens /proc/self/status in a loop and closes nothing:
import errno, os, resource
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print(f"start: soft={soft} hard={hard}")
# Lower only the SOFT limit; the hard limit stays where it is.
resource.setrlimit(resource.RLIMIT_NOFILE, (64, hard))
print("after setrlimit: soft=%d hard=%d" % resource.getrlimit(resource.RLIMIT_NOFILE))
fds = []
try:
for _ in range(200):
fds.append(os.open("/proc/self/status", os.O_RDONLY))
except OSError as exc:
print(f"opened {len(fds)} fds, then: errno={exc.errno} "
f"({errno.errorcode[exc.errno]}) strerror={exc.strerror!r}")
print("str(exc) =", exc)
finally:
for fd in fds:
os.close(fd)
print("closed; rlimit still", resource.getrlimit(resource.RLIMIT_NOFILE))
Output on the test host, unedited:
start: soft=4096 hard=524288
after setrlimit: soft=64 hard=524288
opened 61 fds, then: errno=24 (EMFILE) strerror='Too many open files'
str(exc) = [Errno 24] Too many open files: '/proc/self/status'
closed; rlimit still (64, 524288)
Read that 61 carefully. The limit was 64, and the loop died on descriptor number 62 — the interpreter already held stdin, stdout and stderr, so the available budget was 61, not 64. That is the arithmetic that ruins capacity planning: the limit is a total for the process, and every library, driver and connection pool in it is spending from the same purse before your code runs a single line.
Node reports the same kernel event with a different shape, and it is worth knowing both because the two are routinely grepped together. The probe lowers the descriptor budget to 64 and opens the same file five hundred times:
// Reproduce EMFILE from Node: run under `ulimit -n 64`.
const fs = require('fs');
(async () => {
const handles = [];
try {
for (let i = 0; i < 500; i += 1) {
handles.push(await fs.promises.open('/proc/self/status', 'r'));
}
} catch (err) {
console.error(`name=${err.name} code=${err.code} errno=${err.errno} syscall=${err.syscall}`);
console.error(`message=${err.message}`);
console.error(`opened ${handles.length} handles before failing`);
process.exitCode = 1;
} finally {
await Promise.all(handles.map((h) => h.close().catch(() => {})));
}
})();
$ ulimit -n 64 && node emfile_probe.js
name=Error code=EMFILE errno=-24 syscall=open
message=EMFILE: too many open files, open '/proc/self/status'
opened 47 handles before failing
Same EMFILE, but libuv negates the number: errno=-24 in a Node error means the kernel returned +24. If you have ever built an alert on errno > 0 and wondered why Node's EMFILE alerts never fired, that sign is why.
EMFILE vs ENFILE: the same sentence, two different ceilings
The two errors are one letter apart in the Linux manual and weeks apart in effort. open(2) defines them as follows:
| errno | Manual wording (open(2)) | Scope | Read it with | Measured value here |
|---|---|---|---|---|
EMFILE (24) | "The per-process limit on the number of open file descriptors has been reached (see the description of RLIMIT_NOFILE in getrlimit(2))." | One process | /proc/<pid>/limits, prlimit --pid | 4096 soft |
ENFILE (23) | "The system-wide limit on the total number of open files has been reached." | Whole kernel | /proc/sys/fs/file-max | 65535 |
The practical split: EMFILE is fixed by raising a limit for one process or closing descriptors in one process, and it never affects your neighbours. ENFILE is a host-wide condition — every process on the box is now failing to open files, including your SSH session and the tool you would reach for. If the same error text appears in several unrelated services at the same second, stop tuning one app and look at file-max and at whatever is leaking across the machine.
⚠️ Pitfall: /proc/sys/fs/file-max is not a per-container number when containers share the host kernel. Raising it from inside a container is not possible without host privileges, and exhausting it affects the host. Treat an ENFILE investigation as a host-level incident, not an application bug hunt.
Read the limit the failing process actually has
Before changing anything, read the limit from the process that complained. Two commands, no ambiguity: the kernel's own per-process record, and the tool that reads and writes it.
$ cat /proc/<pid>/limits | grep 'Max open files'
Max open files 4096 524288 files
$ prlimit --pid <pid> --nofile
RESOURCE DESCRIPTION SOFT HARD UNITS
NOFILE max number of open files 4096 524288 files
The same numbers seen from inside Python, plus the descriptor count that produced the EMFILE, look like this — the interpreter's four descriptors at rest are stdin, stdout, stderr and one temporary entry the listdir call itself creates:
resource.getrlimit(RLIMIT_NOFILE) -> soft=4096 hard=524288
open fds right now: 4
--- /proc/self/limits ---
Limit Soft Limit Hard Limit Units
Max processes unlimited unlimited processes
Max open files 4096 524288 files
Max locked memory 8388608 8388608 bytes
And the ceilings above the process limit, which is where ENFILE lives and where a privileged process stops: /proc/sys/fs/file-max was 65535 on this host and /proc/sys/fs/nr_open was 1048576.
| Value | Source | Scope | Changeable by |
|---|---|---|---|
| Soft limit (4096) | /proc/<pid>/limits | Process | The process itself, up to the hard limit |
| Hard limit (524288) | /proc/<pid>/limits | Process | CAP_SYS_RESOURCE, or a parent before exec |
fs.nr_open (1048576) | /proc/sys/fs/nr_open | Kernel | Root, host-wide |
fs.file-max (65535) | /proc/sys/fs/file-max | Kernel | Root, host-wide; ENFILE when exhausted |
Trap 1: you can raise the soft limit, and only the soft limit
The manual is precise about the asymmetry: "The soft limit is the value that the kernel enforces for the corresponding resource. The hard limit acts as a ceiling for the soft limit: an unprivileged process may set only its soft limit to a value in the range from 0 up to the hard limit, and (irreversibly) lower its hard limit." Only a process holding CAP_SYS_RESOURCE "may make arbitrary changes to either limit value."
Because high-level runtimes pre-validate before entering the syscall, the error you see in application code is often not the kernel's. Python's resource.setrlimit raises a ValueError with no errno at all, and the message depends on which ceiling you hit — only one of the three names the real cause:
measured: soft=4096 hard=524288
kernel ceiling /proc/sys/fs/nr_open = 1048576
[soft>hard] ValueError: errno=None (n/a) str(exc)=current limit exceeds maximum limit
[hard raise] ValueError: errno=None (n/a) str(exc)=not allowed to raise maximum limit
[hard>nr_open] ValueError: errno=None (n/a) str(exc)=not allowed to raise maximum limit
process is uid 10000
To see what the kernel itself says, call setrlimit(2) through ctypes and read errno directly — this is the only way to get the real codes, because every wrapper library translates them first:
import ctypes, ctypes.util, errno, os
libc = ctypes.CDLL(ctypes.util.find_library("c") or "libc.so.6", use_errno=True)
RLIMIT_NOFILE = 7
class RLimit(ctypes.Structure):
_fields_ = [("rlim_cur", ctypes.c_ulong), ("rlim_max", ctypes.c_ulong)]
def setrlimit(cur, mx):
lim = RLimit(cur, mx)
ctypes.set_errno(0)
rc = libc.setrlimit(RLIMIT_NOFILE, ctypes.byref(lim))
err = ctypes.get_errno()
return f"rc={rc} errno={err} ({errno.errorcode.get(err, '-')}: {os.strerror(err) if err else 'ok'})"
def getrlimit():
lim = RLimit()
libc.getrlimit(RLIMIT_NOFILE, ctypes.byref(lim))
return lim.rlim_cur, lim.rlim_max
nr_open = int(open("/proc/sys/fs/nr_open").read())
soft, hard = getrlimit()
print(f"getrlimit -> soft={soft} hard={hard}; /proc/sys/fs/nr_open={nr_open}")
for line in open("/proc/self/status"):
if line.startswith(("Uid", "CapEff")):
print(" ", line.rstrip())
print("soft=100, hard=50 ->", setrlimit(100, 50))
print(f"hard={hard + 1} (raise) ->", setrlimit(hard + 1, hard + 1))
print(f"hard={nr_open + 1} ->", setrlimit(nr_open + 1, nr_open + 1))
print("new soft=32 ->", setrlimit(32, hard), "now", getrlimit())
print("restore ->", setrlimit(soft, hard), "now", getrlimit())
Three calls, three different answers — and note that the last two differ by a factor of two in the value requested yet return the same code, because the capability check happens before the value is looked at:
getrlimit -> soft=4096 hard=524288; /proc/sys/fs/nr_open=1048576
Uid: 10000 10000 10000 10000
CapEff: 0000000000000000
soft=100, hard=50 -> rc=-1 errno=22 (EINVAL: Invalid argument)
hard=524289 (raise) -> rc=-1 errno=1 (EPERM: Operation not permitted)
hard=1048577 -> rc=-1 errno=1 (EPERM: Operation not permitted)
new soft=32 -> rc=0 errno=0 (-: ok) now (32, 524288)
restore -> rc=0 errno=0 (-: ok) now (4096, 524288)
CapEff: 0000000000000000 is the whole story: an empty effective capability set, so no CAP_SYS_RESOURCE, so EPERM the moment the requested hard limit exceeds the current one. Lowering the soft limit to 32 succeeded instantly. This is why "just raise the limit" works in a systemd unit that runs as root and silently does nothing in a unit that runs as User=app.
💡 Tip: for a running process, prlimit --pid <pid> --nofile=65536:65536 sets both values without a restart — but only down the same rules, and only if you may signal that process. For a shell session and everything it will spawn, ulimit -n 65536 is enough, provided 65536 is at or below the hard limit you already have.
One warning that survives from 1995 and is still in the manual: systemd's own documentation notes that raising the soft limit above 1024 needs care because select(2) "cannot function with file descriptors above 1023 on Linux". Code using select instead of epoll will break in a way that looks nothing like a resource limit. If the service that broke is old C code, that is the first thing to check after the limit is raised.
Trap 2: a systemd unit never inherits your shell's ulimit
This is the single most common reason a fix "works manually" and fails as a service. A unit is started by PID 1, not by your shell, so the shell's ulimit -n is irrelevant to it. systemd.exec(5) defines LimitNOFILE= among the process properties, and its syntax is the part people get wrong: values are "either as single value to set a specific soft and hard limit to the same value, or as colon-separated pair soft:hard to set both limits individually".
Writing LimitNOFILE=65536 sets both values to 65536. Writing LimitNOFILE=1024:524288 sets the enforced soft limit to 1024 and leaves the ceiling at 524288. Both are legal; only the first is what people usually mean.
# /etc/systemd/system/api.service.d/nofile.conf
[Service]
LimitNOFILE=65536
$ systemctl daemon-reload
$ systemctl restart api.service
$ systemctl show -p LimitNOFILE api.service
LimitNOFILE=65536
Then verify against the running process rather than the unit file, because the unit file is not the kernel's record:
$ PID=$(systemctl show -p MainPID --value api.service)
$ grep 'Max open files' /proc/$PID/limits
Max open files 65536 65536 files
Two documented details decide what happens when you write neither of those files. DefaultLimitNOFILE= "defaults to 1024:524288" (systemd-system.conf(5)) — which is why a fresh unit on a modern distribution starts with a soft limit of 1024 while the hard limit looks enormous, and why a leak of a few sockets per request kills the service in minutes rather than days. And those defaults "are only defaults for units, they are not applied to the service manager process (i.e. PID 1) itself", so PID 1's own limit is not the number your service gets either. Measure, do not infer.
⚠️ Not measured here: the test host runs no systemd (/run/systemd/system does not exist), so the systemd numbers above come from systemd-system.conf(5) and systemd.exec(5), not from a live unit. On any host that does run systemd, the two verification commands in the block above are the ones that count.
Trap 3: containers get their limit from the runtime
Inside a container the limit arrives with the container, and the Docker documentation states why the flag is needed at all: "Since setting ulimit settings in a container requires extra privileges not available in the default container, you can set these using the --ulimit flag." The documented form is the same soft:hard pair, and the manual supplies the verification in its own example:
$ docker run --ulimit nofile=1024:1024 --rm debian sh -c "ulimit -n"
1024
Two behaviours in that page catch people out. First: "If you don't provide a hard limit value, Docker uses the soft limit value for both values. If you don't provide any values, they are inherited from the default ulimits set on the daemon." So --ulimit nofile=65536 is a hard ceiling of 65536 — the container's own code can no longer raise it, and a library that tries will fail with the EPERM measured earlier. Second, nproc is not per container: the docs warn that Linux applies nproc "to a user, not to a container", and their example of four containers sharing --ulimit nproc=3 ends with the fourth failing with [8] System error: resource temporarily unavailable. The same class of confusion applies to every limit you set with a flag: it is a property of the process, and processes share users.
For Compose, the same values are declared per service and are the reason a stack that works in development dies after a deploy:
services:
api:
image: ghcr.io/example/api:1.4.2
ulimits:
nofile:
soft: 65536
hard: 65536
stop_grace_period: 30s
⚠️ Not measured here: the test host's daemon socket is absent — docker info returns Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? — so the container numbers above are quoted from the Docker CLI reference and from the general limit rules verified in the previous sections, not from a container run on this machine. On your own host, one command settles it: docker run --rm --ulimit nofile=65536:65536 debian sh -c 'ulimit -n; cat /proc/self/limits | grep "Max open files"'.
Find the leak, not just the ceiling
Raising a limit buys time; it does not close a descriptor. The difference between a leak and a correctly bounded loop is easy to see once you print the count while it runs. This probe sets a soft limit of 128, then runs the same 300 opens twice: once leaking every descriptor, once closing each one before the next.
soft limit now 128
baseline fds: 4
after 0 opens -> 4 fds
after 30 opens -> 34 fds
after 60 opens -> 64 fds
after 90 opens -> 94 fds
after 120 opens -> 124 fds
leak loop died: errno=24 (EMFILE) 'Too many open files'
after closing the leaked fds: 4 fds
1000 bounded opens OK, 4 fds in flight
The linear ramp — exactly one descriptor per open — is the signature of a leak: nothing is being reclaimed. The second run performed a thousand opens and never exceeded four descriptors in flight. That is the whole fix: not a bigger number, but a bounded number of handles that are always released on every path, including the exception path.
| Question | Command | Note |
|---|---|---|
| How many descriptors does it hold? | ls /proc/<pid>/fd | wc -l | Needs the same uid or root; counts every type of descriptor, not just files |
| What are they? | ls -l /proc/<pid>/fd | Shows socket:[inode], pipe:[inode] and deleted files — the usual leak shapes |
| Is it sockets rather than files? | ss -tanp | grep pid=<pid> | A socket leak is often one missing close() in a retry path |
| Is the ceiling system-wide? | cat /proc/sys/fs/file-max | Compare with total in-use descriptors before blaming the app |
Two patterns produce most of these leaks in services. The first is a retry loop that opens a connection, fails, and retries without closing what it opened — each attempt costs a descriptor and eventually the retries are the load. The second is per-request client construction: creating an HTTP client, a database handle or a TLS context inside the request handler instead of once per worker. Both are invisible at a thousand requests per day and fatal at a thousand per second.
Fix the code: bounded concurrency and scope-based closing
On a runtime that has it, closing at the end of a lexical scope removes the class of bug entirely. In Node 20 and later, a FileHandle is disposable, so await using closes it when the block exits — including on the throw path. Verified on Node v26.5.1: touching the handle after the block fails with EBADF, which is the proof the close happened.
const fs = require('fs');
(async () => {
let leaked;
{
await using fh = await fs.promises.open('/proc/self/status', 'r');
leaked = fh;
const buf = Buffer.alloc(16);
const { bytesRead } = await fh.read(buf, 0, 16, 0);
console.log('read', bytesRead, 'bytes;', JSON.stringify(buf.toString('utf8', 0, bytesRead)));
}
try {
await leaked.read(Buffer.alloc(1), 0, 1, 0);
console.log('BAD: handle still usable after `await using` block');
} catch (err) {
console.log(`closed by scope exit: code=${err.code} message=${err.message}`);
}
})();
read 16 bytes; "Name:\tnode-MainT"
closed by scope exit: code=EBADF message=file closed
Scope-based closing still needs a cap on parallelism, because Promise.all over a directory tree opens one handle per entry before any of them close. A semaphore of 32 slots turns that into a fixed footprint. The semaphore has no dependency:
const fs = require('fs');
const MAX_IN_FLIGHT = 32;
function semaphore(slots) {
const queue = [];
return function acquire() {
return new Promise((resolve) => {
if (slots > 0) { slots -= 1; resolve(); return; }
queue.push(resolve);
}).then(() => function release() {
const next = queue.shift();
if (next) next();
else slots += 1;
});
};
}
function fdCount() {
try { return fs.readdirSync(`/proc/${process.pid}/fd`).length; } catch { return -1; }
}
(async () => {
const acquire = semaphore(MAX_IN_FLIGHT);
const files = Array.from({ length: 400 }, () => '/etc/hostname');
let peak = fdCount();
let bytes = 0;
await Promise.all(files.map(async (file) => {
const release = await acquire();
try {
const handle = await fs.promises.open(file, 'r');
try {
const { size } = await handle.stat();
bytes += size;
peak = Math.max(peak, fdCount());
} finally {
await handle.close();
}
} finally {
release();
}
}));
console.log(`${files.length} files, ${bytes} bytes, at most ${MAX_IN_FLIGHT} handles open`);
console.log(`peak fds during the batch = ${peak}; after the batch = ${fdCount()}`);
})().catch((err) => {
console.error(`failed: ${err.code || err.name}: ${err.message}`);
process.exitCode = 1;
});
Output on the test host with 400 files:
400 files, 5200 bytes, at most 32 handles open
peak fds during the batch = 50; after the batch = 19
One subtlety decides whether the cap holds at all: a released slot must be transferred to a waiting caller, not counted twice. Incrementing the counter and waking a waiter lets the free-slot count drift upward, and a streaming workload — acquire, work, release, repeat — then runs more tasks than the cap allows even though a fully queued batch looked correct. With the transfer above, eight workers issuing fifty tasks each against a two-slot semaphore measure exactly two at a time:
slots=2, 8 workers x 50 tasks -> measured max concurrent = 2
Note the honest gap between 32 and 50: the semaphore bounds your handles, while the runtime keeps its own descriptors — epoll, eventfd, the DNS socket, the readdir call that counts them. Size the limit for the difference, or the pool will be correct and the process will still die.
In Python the equivalent discipline is a with block or an explicit try/finally around every acquisition, plus a worker pool instead of a list comprehension of futures. The rule that matters is not the syntax: it is that every descriptor has exactly one owner and that the owner releases it on all paths.
Verify the fix after the deploy
A limit change is unverified until you have read the number back from the running process. This script does both halves — it reads the ceiling from /proc and the current usage from the descriptor table, and it exits non-zero when the process is close to the wall so it can be wired straight into a post-deploy check.
#!/usr/bin/env python3
"""fd budget check: exit 0 = headroom, exit 1 = at risk, exit 2 = could not read."""
import os
import sys
pid = sys.argv[1] if len(sys.argv) > 1 else "self"
root = "/proc/self" if pid == "self" else f"/proc/{pid}"
try:
used = len(os.listdir(f"{root}/fd"))
with open(f"{root}/limits") as fh:
soft = int([l for l in fh if l.startswith("Max open files")][0].split()[3])
except FileNotFoundError:
print(f"no such process: {pid}", file=sys.stderr)
raise SystemExit(2)
except PermissionError:
print(f"cannot read {root}/fd: not your process (try the same uid or sudo)",
file=sys.stderr)
raise SystemExit(2)
pct = used / soft * 100 if soft else 100.0
verdict = "OK" if pct < 80 else "AT RISK"
print(f"pid={pid} fds={used} soft_limit={soft} used={pct:.1f}% -> {verdict}")
raise SystemExit(0 if pct < 80 else 1)
Both failure paths were exercised, not just the happy one, because a monitor that dies silently is worse than no monitor:
$ python3 fd_budget_check.py
pid=self fds=4 soft_limit=4096 used=0.1% -> OK # exit 0
$ python3 fd_budget_check.py 999999
no such process: 999999 # exit 2
$ python3 fd_budget_check.py 1
cannot read /proc/1/fd: not your process (try the same uid or sudo) # exit 2
Wired into a cron or a post-deploy hook with the service's MainPID, exit code 1 becomes the alert that fires before the EMFILE line appears in the logs, which is the only version of this monitor that saves you an incident.
The checklist, in the order that saves the most time
- Read the limit from the failing process:
cat /proc/<pid>/limits— never from your own shell. - Decide
EMFILEorENFILE: one process or the whole kernel. - If it is a service: set
LimitNOFILE=soft:hardin a drop-in,daemon-reload, restart, then re-read/proc/<pid>/limits. - If it is a container: set
--ulimit nofile=soft:hardor the Composeulimitsblock, then verify inside the container. - If the counter is still climbing linearly, stop tuning numbers: find the descriptor with no owner.
- After the fix, run a budget check against the live PID so the next leak alerts instead of paging.
Related reading on this site: the cgroup and capability side of container limits in Operation not permitted in Docker: the capability wall, a different class of "the file will not open" in Text file busy (ETXTBSY), and the socket-side twin of this error in ECONNREFUSED from another host: the 127.0.0.1 bind trap.
Last verified 2026-10-06 on kernel 6.17.0-1019-oracle, aarch64, uid 10000, Python 3.13.5, Node v26.5.1. All transcripts in this article are captured from that run; the systemd and Docker figures are quoted from systemd-system.conf(5), systemd.exec(5), open(2), getrlimit(2) and the Docker CLI reference, as marked in place.