System limit for number of file watchers reached in Docker

By Mara Voss · C2CZ · 12 September 2026 · Last verified: 2026-09-12

ENOSPC: System limit for number of file watchers reached does not mean your disk is full — it means the kernel refused a new inotify watch because your user ID has already spent its entire watch budget. The word that misleads everyone is ENOSPC: on a filesystem it means "no space left on device", but inotify_add_watch(2) reuses the same errno for "the user limit on the total number of inotify watches was reached". Nothing is broken, nothing is leaking, and there is no file to delete.

I meet this in self-hosted stacks rather than on laptops: a hot-reloading dev container that stops noticing edits after a few hours, a Syncthing or filebeat sidecar that goes quiet, a tsc --watch inside a container that works on Monday and throws on Tuesday. The pattern is always the same. The limit is a property of the host kernel and is accounted per real user ID, so every container and every host process running as the same uid is spending from one shared pot. This article is the version of the fix I use on client hosts: first work out which of the two inotify budgets ran out, then raise it on the host where it actually lives. Everything quoted below was executed on a Linux 6.17 host as uid 10000, or taken verbatim from the kernel and Docker documentation.

TL;DR

• ENOSPC here is a watch budget, not disk space. inotify_add_watch(2) returns it when the per-user watch count is exhausted; your filesystem is fine.
• There are two budgets, with two different errnos. Watches (fs.inotify.max_user_watches) fail with ENOSPC; instances (fs.inotify.max_user_instances) fail with EMFILE. I reproduced the second one on demand: 124 instances opened, then errno 24 EMFILE (Too many open files) against a limit of 128.
• Read the real numbers before you change anything: /proc/sys/fs/inotify/max_user_watches, max_user_instances, max_queued_events. The lab host runs 188727 / 128 / 16384.
• A container cannot raise its own limit. The budgets are per real user ID in the kernel, and Docker only accepts namespaced sysctls per container — its own documentation says it "does not support changing sysctls inside a container that also modify the host system". The fix happens on the host, or in the VM that runs the engine.
• Not every missed event is a limit. A watch on a file dies when an editor saves atomically: the kernel sends IN_ATTRIB then IN_IGNORED, and the watcher is deaf forever. Watch the directory instead, or use the library's atomic option.
• Docker Desktop is a different bug with the same symptom. On macOS and Windows the events may never cross the host/container boundary at all — and Docker's own known-issues page states outright that inotify does not work under QEMU emulation.

The three budgets behind "file watchers reached"

inotify is a kernel API, and the kernel accountants for it expose exactly three numbers. You do not need the sysctl binary to read them; /proc is the source of truth and sysctl reads the same file.

for k in max_user_watches max_user_instances max_queued_events; do
  printf '%-20s = %s\n' "$k" "$(cat /proc/sys/fs/inotify/$k)"
done
id -u

Run on the host used for this article, that prints the budgets a single user ID gets for free:

max_user_watches     = 188727
max_user_instances   = 128
max_queued_events    = 16384
10000

The man page is explicit about the scope of both, and the wording matters more than any blog post: max_user_watches "specifies an upper limit on the number of watches that can be created per real user ID", and max_user_instances "specifies an upper limit on the number of inotify instances that can be created per real user ID". Per user ID — not per process, per container, per systemd unit, or per compose project. max_queued_events is different in kind: it caps the queue behind one instance, and when it overflows you do not get an error, you get dropped events plus an IN_Q_OVERFLOW event that most applications never check for.

A watch is not a file descriptor, and an instance is not a watch. One instance can hold thousands of watches, which is why a single Node process can exhaust the watch budget without coming close to its file-descriptor limit. The two failures are distinct and the error text tells you which one you have:

What ran out	Errno	What the tooling prints	Fix
Watches (per uid)	ENOSPC (28)	ENOSPC: System limit for number of file watchers reached (Node/chokidar), inotify_add_watch: No space left on device (C, Python, Go)	Raise fs.inotify.max_user_watches, or watch less
Instances (per uid)	EMFILE (24)	inotify_init: Too many open files, Failed to initialize watcher	Raise fs.inotify.max_user_instances
Queue (per instance)	none	Silence, then stale caches — an IN_Q_OVERFLOW event nobody read	Raise max_queued_events; drain the queue faster

The exact wording of the first row is worth keeping: inotify_add_watch(2) documents ENOSPC as "the user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource". A tool that reports this has not written anything to your disk and has not filled a filesystem.

Why the container reports it and the host owns it

The obvious first instinct is to raise the number inside the container. It cannot work, and Docker's documentation says why in one sentence: "You can only use sysctls that are namespaced in the kernel. Docker does not support changing sysctls inside a container that also modify the host system." The fs.inotify.* family is exactly that kind of sysctl — kernel-wide state, accounted per real user ID — so a docker run --sysctl fs.inotify.max_user_watches=524288 does not give a container a private budget.

Two consequences follow, and both bite in production. First, the numbers you read inside a container are the host's numbers, and there is no per-container knob to raise. Verify that on any host you own with a working daemon:

# the value inside the container is the host kernel's value
docker run --rm alpine cat /proc/sys/fs/inotify/max_user_watches

# and this is not the way to raise it
docker run --rm --sysctl fs.inotify.max_user_watches=524288 alpine true

Second — and this is the part that makes the problem intermittent — the budgets are shared by every process with the same uid. Most self-hosted images run as root by default, which means the file watcher in your compose stack, the container's log shipper, and anything else running as 0 on that host are drawing from one allocation. A container that starts fine and fails a few hours later is often just the last arrival at an exhausted budget.

The operational upshot is the one I put in every build: do not run watchers as root, and do not let unrelated services share a uid. A dedicated user per service is not only a least-privilege measure here, it also gives that service its own watch budget instead of making it a noisy neighbour. If you already isolate with user namespaces, the accounting still follows the real uid unless you also remap it — check before assuming isolation.

Which of the two budgets do you have?

Before changing a kernel setting, prove which budget is exhausted. This probe opens instances and adds watches until the kernel refuses, printing the errno instead of a stack trace. It uses nothing outside the standard library: inotify is driven through ctypes, which also means it works on a minimal host image where inotifywait is not installed.

#!/usr/bin/env python3
"""inotify-probe: tell apart ENOSPC (watch budget) from EMFILE (instance budget).

Usage: python3 inotify-probe.py [--watches N] [--instances N]
Exit status: 0 = both budgets still had room, 1 = at least one budget is exhausted.
"""
import argparse
import ctypes
import ctypes.util
import errno
import os
import shutil
import sys
import tempfile

IN_ALL_EVENTS = 0x00000FFF
IN_NONBLOCK = 0x00000800

_libc = ctypes.CDLL(ctypes.util.find_library("c") or "libc.so.6", use_errno=True)


def limits():
    """Read the three inotify budgets straight from the kernel."""
    out = {}
    for key in ("max_user_watches", "max_user_instances", "max_queued_events"):
        with open("/proc/sys/fs/inotify/" + key, encoding="ascii") as fh:
            out[key] = int(fh.read().strip())
    return out


def errno_name(number):
    return errno.errorcode.get(number, str(number))


def open_instance():
    """One inotify instance = one file descriptor, capped by max_user_instances."""
    fd = _libc.inotify_init1(IN_NONBLOCK)
    if fd == -1:
        number = ctypes.get_errno()
        raise OSError(number, os.strerror(number))
    return fd


def add_watch(fd, path):
    """One watch = one inode, capped by max_user_watches."""
    wd = _libc.inotify_add_watch(fd, os.fsencode(path), IN_ALL_EVENTS)
    if wd == -1:
        number = ctypes.get_errno()
        raise OSError(number, os.strerror(number))
    return wd


def probe_watches(want):
    workdir = tempfile.mkdtemp(prefix="inotify-probe-")
    added, failure, fd = 0, None, None
    try:
        fd = open_instance()
        for index in range(want):
            path = os.path.join(workdir, "watch-%06d" % index)
            open(path, "w", encoding="ascii").close()
            try:
                add_watch(fd, path)
            except OSError as exc:
                failure = exc
                break
            added += 1
    finally:
        if fd is not None:
            os.close(fd)
        shutil.rmtree(workdir, ignore_errors=True)
    return added, failure


def probe_instances(want):
    fds, failure = [], None
    for _ in range(want):
        try:
            fds.append(open_instance())
        except OSError as exc:
            failure = exc
            break
    for fd in fds:
        os.close(fd)
    return len(fds), failure


def main(argv=None):
    parser = argparse.ArgumentParser(description="Which inotify budget ran out?")
    parser.add_argument("--watches", type=int, default=64,
                        help="how many watches to add before giving up (default 64)")
    parser.add_argument("--instances", type=int, default=96,
                        help="how many instances to open before giving up (default 96)")
    args = parser.parse_args(argv)

    budgets = limits()
    print("uid %d -- kernel budgets: %s"
          % (os.getuid(), ", ".join("%s=%d" % item for item in budgets.items())))
    exhausted = False

    count, failure = probe_watches(args.watches)
    if failure is None:
        print("watches   : %d added, still room (tried %d)" % (count, args.watches))
    else:
        print("watches   : %d added, then errno %d %s (%s)"
              % (count, failure.errno, errno_name(failure.errno), failure.strerror))
        exhausted = True

    count, failure = probe_instances(args.instances)
    if failure is None:
        print("instances : %d opened, still room (tried %d)" % (count, args.instances))
    else:
        print("instances : %d opened, then errno %d %s (%s)"
              % (count, failure.errno, errno_name(failure.errno), failure.strerror))
        exhausted = True

    return 1 if exhausted else 0


if __name__ == "__main__":
    sys.exit(main())

Run with defaults, the host is comfortable. Run with an instance count above the budget and the second failure appears — the exact string you would see when a container image starts one watch process per log file or per worker:

$ python3 inotify-probe.py --instances 200
uid 10000 -- kernel budgets: max_user_watches=188727, max_user_instances=128, max_queued_events=16384
watches   : 64 added, still room (tried 64)
instances : 124 opened, then errno 24 EMFILE (Too many open files)
$ echo $?
1

Note the arithmetic: the budget is 128 and the probe stopped at 124, because four instances were already held by other processes running as the same uid. That gap is the whole story of the shared budget, in one run. The watch side did not fail at 64 or at 500, and I am not going to find its edge on a shared host: proving ENOSPC for watches means consuming all 188,727 watches of that uid, which would break every other watcher on the machine — including the ones I need to finish this article. So the watch-limit branch of the probe is quoted from the kernel documentation and from the tooling that surfaces it, not from a deliberate exhaustion run, and I would give the same advice about your own host: raise the number, do not test it.

The failure that is not a limit at all

A limit is only one reason a watcher goes silent, and in development containers it is not the most common one. inotify is inode-based: the man page states that "when monitoring a file (but not when monitoring the directory containing a file), an event can be generated for activity on any link to the file". Watch a path to a file, and you have watched that one inode. Editors that save atomically — write a temporary file, then rename it over the original — replace the inode, so the watch you own belongs to a file that no longer has a name.

I reproduced it with fs.watch(), the same primitive most JavaScript dev servers sit on, watching one file and its directory at the same time while an editor-style atomic save happens:

'use strict';
/*
 * node fs.watch(): file watcher vs directory watcher across an atomic save.
 * Run:  node node_watch_lab.js
 */
const fs = require('fs');
const os = require('os');
const path = require('path');

const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'node-inotify-'));
const file = path.join(dir, 'app.log');
fs.writeFileSync(file, 'start\n');

let fileEvents = 0;
let dirEvents = 0;
let fileLast = '-';
const fw = fs.watch(file, (event, name) => { fileEvents += 1; fileLast = `${event}${name ? ' ' + name : ''}`; });
const dw = fs.watch(dir, () => { dirEvents += 1; });

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

(async () => {
  const steps = [
    ['in-place write', () => fs.appendFileSync(file, 'appended in place\n')],
    ['atomic save (write .tmp, rename over)', () => {
      const tmp = `${file}.tmp`;
      fs.writeFileSync(tmp, 'written by the editor\n');
      fs.renameSync(tmp, file);
    }],
    ['in-place write after the rename', () => fs.appendFileSync(file, 'post-rename write\n')],
    ['unrelated file created in the watched directory', () => fs.writeFileSync(path.join(dir, 'other.txt'), 'x\n')],
  ];

  for (const [label, fn] of steps) {
    const f0 = fileEvents;
    const d0 = dirEvents;
    fn();
    await sleep(300);
    console.log(
      `${label.padEnd(46)} file-watch: +${fileEvents - f0}  dir-watch: +${dirEvents - d0}`
    );
  }
  console.log(`last event seen by the file watcher: ${fileLast || 'none'}`);
  fw.close();
  dw.close();
})();

The file watcher never errors. It simply stops mattering after the rename — and it keeps consuming its watches while doing so:

in-place write                                 file-watch: +1  dir-watch: +1
atomic save (write .tmp, rename over)          file-watch: +3  dir-watch: +4
in-place write after the rename                file-watch: +0  dir-watch: +1
unrelated file created in the watched directory file-watch: +0  dir-watch: +2
last event seen by the file watcher: rename app.log

At the syscall level the same experiment explains the silence precisely: the kernel sends IN_ATTRIB, then IN_IGNORED, and the watch descriptor is gone. The directory watch, by contrast, sees IN_CREATE for the temporary file, IN_CLOSE_WRITE for the write, and IN_MOVED_TO for the rename — which is why every well-behaved watcher watches directories, and why chokidar ships an atomic option for exactly this case. If your dev server reloads on the first edit of a file and then goes quiet, you have this failure, not the limit, and raising max_user_watches will change nothing.

The man page adds a second trap for bind mounts: "when monitoring a directory, events are not generated for the files inside the directory when the events are performed via a pathname (i.e., a link) that lies outside the monitored directory." If one service writes to a host path and another watches the same data through a different path — a bind mount, a symlink, a second mount namespace — the events can legitimately never arrive, no matter the budget.

The host-side fix

Once you know the budget is the problem, raise it where it lives: on the host that runs the kernel, which for self-hosted Docker means the Docker host, and for Docker Desktop means the VM that runs the engine, not your laptop's shell. Set all three values deliberately — the instance and queue numbers matter as soon as a stack has several watcher processes.

# 1. current values
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances fs.inotify.max_queued_events

# 2. inspect what is already configured (file wins over file; last wins over first)
grep -r fs.inotify /etc/sysctl.conf /etc/sysctl.d/ 2>/dev/null

# 3. persist the raise (one file, one owner, no duplicates)
cat <<'EOF' | sudo tee /etc/sysctl.d/99-inotify.conf
# inotify budgets: watches cap recursive watchers, instances cap watcher processes
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 1024
fs.inotify.max_queued_events = 65536
EOF

# 4. apply and verify from the kernel, not from the tool
sudo sysctl --system
cat /proc/sys/fs/inotify/max_user_watches

On a host without the procps package — some minimal images and appliances — sysctl does not exist, and writing the /proc file directly is the same operation: echo 524288 | sudo tee /proc/sys/fs/inotify/max_user_watches. That form is not persistent, which is the point: use it to test, use /etc/sysctl.d/ to make it survive a reboot. On systems using systemd, sysctl --system reloads every drop-in and is the correct way to apply the file without a reboot.

Sizing is the one place people copy magic numbers. Do the arithmetic instead of trusting a blog: inotify directory monitoring is not recursive, so one watch per directory in the trees you watch, times the number of watcher processes, times a safety factor for editor temp files. Count the directories in the repository you care about and multiply — find /srv/myapp -type d | wc -l tells you the floor for a single recursive watcher. Half a million watches is a reasonable ceiling for a busy host precisely because it is far above what an honest recursive watcher needs; if you find yourself wanting a million, the real problem is that you are watching node_modules or a data directory that should be excluded.

Docker Desktop and WSL2: when the host is not the host

Everything above assumes Linux containers on a Linux kernel. Docker Desktop inserts a VM, and there the failure mode changes: the limit must be raised inside that VM, and in some configurations it is not a limit problem at all.

• Windows with the WSL2 backend. The engine runs inside a WSL distribution, so that distribution's kernel settings are the ones that matter. Persist them with /etc/wsl.conf — a [boot] command entry runs as root each time the instance starts — and remember Microsoft's rule that a WSL change only takes effect after the distribution has fully restarted (wsl --shutdown). The [boot] section is documented as available on Windows 11 and Server 2022 only.
• Bind mounts that never deliver events. On Docker Desktop, host-to-container event propagation is a known gap: the engine issue docker/for-win#12766 reports create, delete and modify events not reaching an Alpine container through a bind mount at all. No sysctl fixes that — the events are not crossing the boundary.
• Emulated architectures are worse. Docker's own known-issues page states that for Intel images on Apple silicon, "filesystem change notification APIs (inotify) do not work under QEMU emulation". If you run --platform linux/amd64 on an M-series Mac, inotify is unavailable by construction; use an arm64 image or watch from the host.
• The reliable fallback is polling, and it is a tax. chokidar, Vite, webpack and most dev servers can poll instead of watching (CHOKIDAR_USEPOLLING=1, or the equivalent option in your tool). It works everywhere and costs CPU proportional to file count, so it belongs on the smallest possible subtree — never the repository root of a monorepo.

If you run WSL2 as your Docker host, the diagnostics are the same as on a Linux server, and so are the failure modes around it: WSL2 networking surprises and iptables failures at container start come from the same place — a VM whose state you forgot was separate from the host you were configuring.

Verification checklist

• Read it inside the container that failed: docker exec <container> cat /proc/sys/fs/inotify/max_user_watches must show the new value. If it does not, you edited a different kernel.
• Check where the value comes from: grep -r fs.inotify /etc/sysctl.conf /etc/sysctl.d/ — two files setting the same key means the load order decides, and the last one wins.
• Prove the fix with load, not with a restart: run the probe at a count just above the old budget, or start the full watcher set, and confirm no ENOSPC/EMFILE in the logs after a few hours of normal work.
• Re-check after a reboot: a raise that is not in /etc/sysctl.d/ is gone, and the error returns at the worst possible time.
• Separate the two failures in your runbook: ENOSPC = watches per uid, EMFILE = instances per uid, silence with no error = usually an inode-level watch that died on an atomic save, or events that never crossed a Docker Desktop file-sharing boundary.
• Shrink the demand before raising the ceiling: exclude node_modules, build output and data directories; give each service its own uid; prefer directory watchers over per-file watchers.

Related work on this blog

• Docker "request canceled while waiting for connection" — the same discipline applied to a registry failure: establish which layer refused before changing settings.
• network_mode: host not working on Docker WSL2 — why the WSL2 VM, not your shell, is the host you must configure.
• Docker driver failed programming external connectivity — the container-start failures that keep iptables and kernel state in the picture.

All kernel limits, errno values and event sequences quoted in this article were reproduced on a Linux 6.17 host (uid 10000, Python 3.13, Node 26.5) on 2026-09-12; the container and Docker Desktop statements are quoted from the Linux man-pages, Docker's documentation and the linked engine issues.