ENOSPC: System limit for number of file watchers reached
ENOSPC: System limit for number of file watchers reached has nothing to do with free disk space. The kernel returns ENOSPC from inotify_add_watch() when the per-user watch counter is full, and the dev server that reports it is usually the victim, not the cause. On this host the entire budget is two integers — fs.inotify.max_user_instances = 128 and fs.inotify.max_user_watches = 188727 — both charged per real user ID, both shared silently by every process you own, and both easy to measure with nothing but /proc.
Everything numeric below was executed on this machine: Linux 6.17 (arm64), uid 10000, Python 3.13.5, Node v26.5.1, Docker daemon absent. The lab calls inotify_init1(), inotify_add_watch() and read() through ctypes, so the error codes are the ones the kernel actually returned, and every block is pasted from that run rather than paraphrased.
TL;DR — two counters, one UID, measured
- Measured ceilings on this host:
max_user_instances=128,max_user_watches=188727,max_queued_events=16384. All three are per real user ID, which is the detail that explains most of the confusion around this error. - Watches are per inode, not per directory, and a directory watch is not recursive. Adding 1000 file watches took the descriptor's watch count from 1 to 1001 (measured in
/proc/self/fdinfo), and touching a file one level below a watched directory produced exactly zero events. - Instances run out long before watches do. A process opened 122 inotify instances and then failed on number 123 with
errno=24 (Too many open files)—EMFILE. Six instances were already held by other processes of the same uid: 6 + 122 = 128. - Node's fan-out is cheap in instances and expensive in watches. 200
fs.watch()calls by Node v26.5.1 occupied 1 instance and 200 watches, visible as a single fd with 200inotify wd:lines. - Audit before you raise. The budget is readable from
/proc/<pid>/fdinfo/<fd>for every process you own; the auditor at the end of this page prints the per-process table and the percentage of each ceiling you are burning. - Raising the ceiling is a sysctl, and it does not help a leak.
fs.inotify.max_user_watchesis a global kernel knob that every container on the host shares — raising it hides the leak and keeps the process that never closes a descriptor happily burning the new headroom.
Two budgets, and both are charged per real user ID
inotify is not one resource. It is an instance (a file descriptor returned by inotify_init1(), carrying a queue of events) and, inside each instance, a set of watches (one per inode you asked to observe). The kernel meters both against the same subject: your real user ID, not the process, not the cgroup, not the container.
The three knobs and their documented meaning, quoted from inotify(7):
/proc/sys/fs/inotify/max_queued_events
The value in this file is used when an application calls inotify_init(2) to set
an upper limit on the number of events that can be queued to the corresponding
inotify instance. Events in excess of this limit are dropped, but an
IN_Q_OVERFLOW event is always generated.
/proc/sys/fs/inotify/max_user_instances
This specifies an upper limit on the number of inotify instances that can be
created per real user ID.
/proc/sys/fs/inotify/max_user_watches
This specifies an upper limit on the number of watches that can be created per
real user ID.Two consequences follow from "per real user ID" and they are the whole article in miniature. First, every process you own competes for the same two counters — your editor, its language server, three dev servers, a log tailer, a backup agent, and a browser each take a slice, and none of them can see the others. Second, because the limit is enforced by the kernel rather than by a namespace, a container that does not create a user namespace runs with your UID in your kernel and draws from the same counters as everything on the host. That is why a watch failure inside a container can be caused by a process outside it.
The failure modes are distinct and the kernel distinguishes them by error code, which is the fastest way to know which of the two budgets you exhausted: inotify_init1() fails with EMFILE when the instance budget is full, inotify_add_watch() fails with ENOSPC when the watch budget is full. From inotify_init(2): "EMFILE: The user limit on the total number of inotify instances has been reached." From inotify_add_watch(2): "ENOSPC: The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource."
Audit the budget before you raise it
The kernel publishes the accounting per descriptor. For an inotify fd, /proc/<pid>/fdinfo/<fd> contains one line per watch, each beginning with inotify wd: — so counting those lines is counting watches, and counting descriptors whose readlink target is anon_inode:inotify is counting instances. No root, no strace, no guesswork.
Here is the whole auditor. It takes an optional uid (default: yours), walks /proc, and prints a per-process table with the percentage of each ceiling in use:
#!/usr/bin/env python3
"""inotify budget auditor - who is holding the watch and instance budget?
Usage:
inotify_budget.py [uid] # default: your own uid
Reads the kernel's per-descriptor bookkeeping in /proc/<pid>/fdinfo to count
inotify instances and watches actually held by every process of a given uid,
then prints them against the limits in /proc/sys/fs/inotify/.
"""
import os
import sys
def sysctl(name):
try:
with open("/proc/sys/fs/inotify/" + name) as fh:
return int(fh.read().strip())
except OSError:
return None
def proc_uid(pid):
try:
with open("/proc/%s/status" % pid) as fh:
for line in fh:
if line.startswith("Uid:"):
return int(line.split()[1])
except OSError:
pass
return None
def inotify_fds(pid):
"""Return [(fd, watch_count)] for every inotify descriptor of a process."""
out = []
fd_dir = "/proc/%s/fd" % pid
try:
names = os.listdir(fd_dir)
except OSError:
return out
for name in names:
try:
target = os.readlink(os.path.join(fd_dir, name))
except OSError:
continue
if target != "anon_inode:inotify":
continue
watches = 0
try:
with open("/proc/%s/fdinfo/%s" % (pid, name)) as fh:
watches = sum(1 for line in fh if line.startswith("inotify wd:"))
except OSError:
pass
out.append((name, watches))
return out
def audit(uid):
procs = []
for entry in os.listdir("/proc"):
if not entry.isdigit():
continue
if proc_uid(entry) != uid:
continue
fds = inotify_fds(entry)
if not fds:
continue
cmd = ""
try:
with open("/proc/%s/cmdline" % entry, "rb") as fh:
cmd = fh.read().replace(b"\0", b" ").decode(errors="replace").strip()[:60]
except OSError:
pass
procs.append((entry, cmd or "(kernel thread)", fds))
return procs
def main():
uid = int(sys.argv[1]) if len(sys.argv) > 1 else os.getuid()
max_inst = sysctl("max_user_instances")
max_watch = sysctl("max_user_watches")
procs = audit(uid)
print("uid %d - inotify budget" % uid)
print("limits: max_user_instances=%s max_user_watches=%s" % (max_inst, max_watch))
if not procs:
print("no process of this uid currently holds an inotify instance")
return
print("%-8s %-45s %8s %9s" % ("pid", "command", "instances", "watches"))
total_inst = total_watch = 0
for pid, cmd, fds in sorted(procs, key=lambda r: int(r[0])):
watches = sum(w for _, w in fds)
total_inst += len(fds)
total_watch += watches
print("%-8s %-45s %8d %9d" % (pid, cmd, len(fds), watches))
for fd, w in sorted(fds, key=lambda f: int(f[0])):
print(" fd %-5s watches %d" % (fd, w))
print("-" * 74)
print("total: %d instances, %d watches" % (total_inst, total_watch))
if max_inst:
print("instance budget used: %d/%d (%.1f%%)" % (total_inst, max_inst, 100.0 * total_inst / max_inst))
if max_watch:
print("watch budget used: %d/%d (%.1f%%)" % (total_watch, max_watch, 100.0 * total_watch / max_watch))
if __name__ == "__main__":
main()First run, plain and uninteresting — which is itself the diagnostic baseline you want when a watcher is failing. Six headless Chromium processes, each holding one instance and two watches:
$ python3 inotify_budget.py
uid 10000 - inotify budget
limits: max_user_instances=128 max_user_watches=188727
pid command instances watches
676010 /opt/data/.playwright-browsers/chromium-1234/chrome-linux/ch 1 2
fd 24 watches 2
676046 /opt/data/.playwright-browsers/chromium-1234/chrome-linux/ch 1 2
fd 15 watches 2
2411897 /opt/hermes/.playwright/chromium_headless_shell-1234/chrome- 1 2
fd 27 watches 2
2411923 /opt/hermes/.playwright/chromium_headless_shell-1234/chrome- 1 2
fd 16 watches 2
2545653 /opt/data/.playwright-browsers/chromium-1234/chrome-linux/ch 1 2
fd 21 watches 2
2545734 /opt/data/.playwright-browsers/chromium-1234/chrome-linux/ch 1 2
fd 15 watches 2
--------------------------------------------------------------------------
total: 6 instances, 12 watches
instance budget used: 6/128 (4.7%)
watch budget used: 12/188727 (0.0%)Now the same command while a synthetic watcher holds three instances of 200 watches each. Note that nothing else changed: the new process simply appears with its own three descriptors, and the totals move by exactly that much:
holder said: holder pid=2595647 instances=3 watches=600
### while the synthetic watcher holds 3 instances / 600 watches
uid 10000 - inotify budget
limits: max_user_instances=128 max_user_watches=188727
2595647 /usr/bin/python3 /opt/data/c2cz/lab/2026-10-14/watcher_holde 3 600
fd 3 watches 200
fd 4 watches 200
fd 5 watches 200
--------------------------------------------------------------------------
total: 9 instances, 612 watches
instance budget used: 9/128 (7.0%)
watch budget used: 612/188727 (0.3%)And the same measurement taken over a Node process, which is the shape most people meet this error in — 200 files watched by Node v26.5.1, reported as one descriptor with 200 watches:
node said: node pid=2595966 file_watchers=200 version=v26.5.1
uid 10000 - inotify budget
limits: max_user_instances=128 max_user_watches=188727
2595966 node /opt/data/c2cz/lab/2026-10-14/node_watch_demo.js /opt/d 1 200
fd 17 watches 200
--------------------------------------------------------------------------
total: 7 instances, 212 watches
instance budget used: 7/128 (5.5%)
watch budget used: 212/188727 (0.1%)That last table is the important one for web developers. A bundler or editor that reports ENOSPC: System limit for number of file watchers reached is not holding a great many instances; it is holding one instance whose watch count grew with your file tree, plus whatever the rest of your session already spent. When the ceiling is hit, the message identifies the one file whose watch was refused, not the project or the process that consumed the budget.
Watches are per inode, and a directory watch is not recursive
Two properties of the API decide how fast the watch counter fills, and both are measurable in a few lines of ctypes. The first is that a watch is attached to an inode — one per file or directory, no matter how many tools want the same one. The second is that watching a directory covers the entries in it, not the entries inside its subdirectories. inotify(7) states it plainly: "Inotify monitoring of directories is not recursive: to monitor subdirectories under a directory, additional watches must be created. This can take a significant amount time for large directory trees."
Measured, on a tree with one file at the top and one subdirectory below it. One watch is created on the parent, then two files are modified — one in the watched directory, one nested one level down:
== T1: parent-directory watch is NOT recursive ==
wd(parent) = 1
events after touching parent/file0000 and sub/nested: 3
wd=1 mask=0x00000020 name='file0000'
wd=1 mask=0x00000004 name='file0000'
wd=1 mask=0x00000008 name='file0000'
errno of the empty read that ended the drain: 11 (EAGAIN)Three events, all of them for the file that sits directly in the watched directory: mask 0x00000020 is the open, 0x00000004 is the attribute change from the timestamp update, 0x00000008 is the close after writing. The nested file produced nothing, and the read that finally returned no data failed with EAGAIN (11) — the non-blocking descriptor was simply empty. If your watcher behaves as though subdirectories were included, it is because the library created a watch per subdirectory for you, and it paid a watch per subdirectory to do it.
The same run also shows that asking twice is free. A second inotify_add_watch() on the same inode — with or without a trailing slash — returns the descriptor that already exists instead of allocating a second watch:
== T2: duplicate add_watch on the same inode reuses the watch descriptor ==
add_watch(dir) again -> wd=1 ; add_watch(dir + '/') -> wd=1
same descriptor: True
watch count for this descriptor (from fdinfo): 1And confirmed at scale: after adding one watch per file across a 1000-file directory, the descriptor's accounting — the same inotify wd: lines the auditor counts — reads 1001. One for the directory, 1000 for the files:
== T3: per-descriptor watch accounting ==
after adding 1000 file watches -> fdinfo count = 1001So the cost model for anyone watching a source tree is linear in the number of files and directories actually handed to the kernel. Two thousand files under a recursive watcher is two thousand watches; a node_modules tree with a hundred thousand entries is a hundred thousand watches, which is more than half of this host's entire max_user_watches budget — for one project, in one process, before any editor of yours has started.
One more trap sits in inotify(7)'s notes and matches a symptom people report around bind mounts: "If a filesystem is mounted on top of a monitored directory, no event is generated, and no events are generated for objects immediately under the new mount point." A watch created before a mount appears over that path stays silent for the mounted content. It is the same class of failure as the recursion rule — the watch is valid, the events are for an object you are no longer looking at — and it is worth checking before you blame the limit, because the error message is no message at all.
Instances run out long before watches do
Watches are the budget with a big number next to it, so they get the attention. The instance budget is the small one, and on this host it is 128 for everything you run. Exhausting it looks like this: a process opened inotify descriptors until the kernel refused:
== T4: instance budget (fs.inotify.max_user_instances) ==
inotify instances opened before failure: 122
failed at instance #123 with errno=24 (Too many open files)
errno name: EMFILE
instances released after close: 122128 is the ceiling and 122 descriptors were granted. The missing six are the headless Chromium processes from the auditor baseline, each of which held one instance of its own: 6 + 122 = 128, measured twice on the same host by two independent code paths. That is the per-user nature of the limit stated as arithmetic, and it is why the same EMFILE can appear in a process that opened exactly one descriptor and looks entirely innocent.
Confirm the code before you act on it. EMFILE (24) from inotify_init1() is the instance budget, and it is the one error in this article that a plain file-descriptor shortage can also produce — if your process-level RLIMIT_NOFILE is the smaller of the two, you will see the same errno for a different reason. On this host the two are far apart, which is the useful check to run first:
== T5: /proc/self/limits headroom for this process ==
Max processes unlimited unlimited processes
Max open files 4096 524288 filesWith a soft limit of 4096 descriptors, the 123rd inotify descriptor could not have failed on descriptor exhaustion — it failed because the per-uid instance counter was full. Compare that with the fd-budget failure covered in the EMFILE walkthrough: same errno, different meter, and the fix belongs to whichever meter actually ran out.
The practical asymmetry is brutal. One leaked instance out of 128 costs 0.8% of the budget, and there are many more things that open an inotify instance than most people expect: an editor with several windows, a language server per workspace, a file browser, three dev servers, a terminal multiplexer, a sync client, a CI agent. When a process dies without closing its descriptors, the instance returns to the pool — the measured run shows 122 instances released on close — but a process that is merely alive and idle keeps its slice of the budget forever. Published reports of this exact symptom show an IDE process holding tens of thousands of watches with no dev server running, and users discovering that closing terminals does not return the budget — the Cursor forum thread on the extension host exhausting the inotify limit documents both, with the same /proc/<pid>/fdinfo accounting used above as the evidence.
When watchers fail, the sequence that actually resolves the incident is short: run the auditor, identify the process whose instance or watch count is out of proportion with what it does, and only then decide between reducing its footprint and raising the ceiling.
The fix, in order of least damage
1. Find the holder, don't guess. The auditor above is the whole diagnosis. Run it while the failure is happening and read the two totals: if instances are near the limit, one process is holding an unreasonable number of descriptors; if watches are, look at the single descriptor with the largest count. The number to sanity-check against is what the tool is supposed to do. A bundler watching a 400-file project should account for a few hundred watches, not forty thousand.
2. Reduce watches by watching less. Because a watch is per inode, the cheapest fix is to hand fewer inodes to the kernel. Watch a directory instead of every file in it where the API allows it. Exclude the trees you never edit: in a Node project that means the dependency and build directories, expressed in the tool's own ignore mechanism rather than by hoping the library prunes them. The difference is not marginal — it is the difference between a few hundred watches and a hundred thousand, taken straight from the measured linear cost above.
3. Fall back to polling for the trees that are too big. Every major watcher library ships a polling mode precisely because watches are a finite, per-user resource. It trades CPU calls for descriptors, and it is the right answer for a large tree on a machine where the budget is genuinely shared with other users' workloads. Slower detection, zero watch cost, and it also sidesteps the mount-over-directory silence described above.
4. Raise the ceiling only after step 1. The knobs are global sysctls, not per-service settings, and raising them is one file plus a reload:
# /etc/sysctl.d/98-inotify.conf
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512
fs.inotify.max_queued_events = 32768sysctl --system
cat /proc/sys/fs/inotify/max_user_watches
cat /proc/sys/fs/inotify/max_user_instancesThen re-run the auditor and confirm both totals against the new ceilings. Do not expect this to fix a leak: a process that never closes its descriptors will happily consume the larger budget and fail again later, one restart after the change. It also does not scale per container — these are kernel-wide values shared by every workload on the host, so a dev server inside a container is competing with everything else, and a bigger number is a shared resource decision rather than a local one.
What could not be measured here, stated plainly. This sandbox runs as uid 10000 with /proc/sys on a read-only filesystem, so the sysctl values could not be changed and the ENOSPC branch of the watch budget was never reached: filling 188727 watches would have required allocating that many watches in the shared kernel, which is a hostile thing to do to a host you do not own. The ENOSPC wording quoted near the top is therefore taken from inotify_add_watch(2), the documented behaviour, and not from a run on this machine. Every other value, table and error code on this page came from an executed call on this host, and the distinction matters: the instance budget (EMFILE) was measured to the exact descriptor; the watch budget (ENOSPC) is documented and arithmetically bounded by the number printed in /proc/sys/fs/inotify/max_user_watches.
Reproducing every number on this page
The lab is four files and needs no root, no Docker and no third-party packages. The two syscalls are called through ctypes, so the same code runs on any Linux with Python 3:
mkdir -p ~/inotify-lab && cd ~/inotify-lab
mkdir -p tree/sub
for i in $(seq 0 999); do : > "tree/file$(printf %04d $i)"; done
# read the three ceilings
for k in max_user_instances max_user_watches max_queued_events; do
echo "fs.inotify.$k = $(cat /proc/sys/fs/inotify/$k)"
done
# what is holding the budget right now
python3 inotify_budget.pyThe mask values in the event dump are the ones the kernel returns; 0x00000020 for an open, 0x00000004 for an attribute change and 0x00000008 for a close-after-write are worth recognising, because they are how you tell "my watcher is not firing" apart from "my watcher is firing but only for the parent directory". The equivalent failure on the descriptor side is worth knowing too, and it is the subject of the E2BIG walkthrough of per-process kernel limits: same family of hard ceilings, same discipline of measuring the meter before you change a setting.
For completeness, if you are running watchers inside containers and cannot tell whether the ceiling you hit belongs to the host or to the container, the answer is almost always the host kernel, and the way to prove it is to run the auditor on the host while the container is failing. The related case of a container operation failing because the kernel refuses it regardless of what the container can see is documented in the capability wall in Docker — the two problems share the shape of a permission decision made outside the process that reports the error.
The short version: ENOSPC here is a full counter, not a full disk. Two integers define your inotify capacity, both are charged to your user ID, and the accounting is readable in /proc with no privileges at all.