<defunct> zombie processes in Docker: find and fix the leak
If ps inside your container is filling up with rows whose COMMAND reads [python] <defunct> (or [node] <defunct>, [sh] <defunct>), you are not looking at crashed processes — you are looking at reaped nothing: children that finished and whose exit status nobody collected. They consume no CPU, no memory and no file descriptors, which is exactly why they sit there for days without anyone noticing; what they do consume is a slot in the cgroup pids budget, and the account is only settled when the parent calls wait().
This is measurable, and I measured it on a Linux 6.17 host with cgroup v2, Python 3.13.5 and Node 26.5.1: 25 unreaped children moved pids.current from 473 to 498 and back to 473 the moment I called waitpid(). The same budget is consumed by threads, which is why a Chatty-JS server can exhaust it without ever forking. And the most popular one-line "fix" for zombies — signal(SIGCHLD, SIG_IGN) — silently turns every child exit code in that process into 0, so your failing deploy step reports success.
TL;DR
<defunct>(stateZ) = the process is dead, its parent has not calledwait(). The kernel is holding the exit status in the process table.- A zombie is reaped automatically when its parent dies: the child is reparented to the nearest subreaper or to PID 1. That is why the leak only appears in long-lived parents — which is what a container's PID 1 is.
- Zombies hold no memory and no file descriptors. They hold a PID slot: measured
pids.current+25 for 25 zombies, returning to baseline only afterwaitpid(). - The
pidscontroller counts tasks, not processes: 8 threads movedpids.currentby exactly +8 on this host. signal(SIGCHLD, SIG_IGN)stops zombies but destroys exit statuses:subprocess.run(["/bin/false"]).returncodebecame0. Never use it in a supervisor.- Real fix: one
waitpid(-1, WNOHANG)loop driven bySIGCHLD(tini/dumb-initdo exactly this), ordocker run --init. - Exit code
137/OOMKilledis not a zombie: that process is gone, not stuck.
Everything below was run on a disposable host, not copied from a forum. The measurement harness, the exact commands and the outputs quoted verbatim are in the verification section at the end.
What a <defunct> process actually is
A process does not disappear the instant it exits. The kernel keeps its process-table entry — PID, parent PID, exit status, accounting — until the parent collects it with wait(), waitpid() or waitid(). Until then the entry stays in state Z and shows up in ps as <defunct>. It is a corpse waiting for a signature.
Here is the smallest reproduction that matters, a parent that forks and never waits:
import os, signal, time
# WRONG: fork and forget. Every child that exits becomes a zombie.
for _ in range(5):
pid = os.fork()
if pid == 0:
os._exit(0) # child: work then exit, immediately
# parent falls through and never calls waitpid(pid, ...)
time.sleep(300)
While that parent lives, the process table looks like this (measured):
$ ps -eo stat,ppid,pid,comm | awk '$1 ~ /^Z/'
Z 1124386 1124388 python
Z 1124386 1124389 python
Z 1124386 1124390 python
Z 1124386 1124391 python
Z 1124386 1124392 python
Three things about that output are worth internalising. First, the state letter is Z and the command is wrapped in square brackets — the kernel strips the command line of a zombie, which is why ps aux shows [python] <defunct> and no arguments. Second, the PPID column points at the guilty party, not at PID 1. Third, the count is stable: run the same command a minute later and the same five rows are still there.
The per-process view confirms it. For one of those zombies, /proc/<pid>/status reports:
$ grep -E '^(Name|State|PPid|Threads):' /proc/1124388/status
Name: python
State: Z (zombie)
PPid: 1124386
Threads: 1
And the zombie cannot be removed by signal. Sending it SIGKILL is not an error — the PID exists, so kill() succeeds — but nothing happens, because there is no program left to receive the signal:
os.kill(1124388, signal.SIGTERM) -> no exception
os.kill(1124388, signal.SIGKILL) -> no exception
zombie after kill: [['Z', '1124388']] <- unchanged
This is the single most important diagnostic rule in this article: you do not fix a zombie by attacking the zombie, you fix it by fixing its parent. Killing the parent works too, incidentally — but only because the children are then reparented and reaped by the new parent, which brings us to the lifecycle.
The lifecycle: why long-lived parents leak and short-lived ones do not
When a process dies, its children are reparented to the nearest ancestor marked as a subreaper, or to PID 1 if there is none. That new parent reaps them when they exit. This is why a leak of this kind is invisible in a test suite (the test runner exits, everything is collected) and permanent in production (the parent runs for months).
Measured on the same rig: while the parent was alive, five zombies were present; after the parent exited, the count fell to zero without anybody doing anything:
-- ps while parent alive --
1123888 1123886 Z python
1123889 1123886 Z python
1123890 1123886 Z python
1123891 1123886 Z python
1123892 1123886 Z python
-- zombie count for this parent --
5
-- after parent exits: children reparented and reaped by PID 1 --
0
Now put that in a container. The main process of a container is PID 1 inside its PID namespace, and it usually is not an init system: it is your application, your shell script, your Python supervisor or gunicorn. PID 1 is the last line of reaping defence, and a normal application has no code for it. Anything your process spawns and forgets — a ffmpeg job, a shell pipeline, a child that outlives a request — gets deposited in that parent's process table and stays there for the lifetime of the container. The official Docker documentation is explicit about the consequence and about the remedy: with --init, "Specifying an init process ensures the usual responsibilities of an init system, such as reaping zombie processes, are performed inside the created container."
The remaining question is how many zombies you can afford. Probably more than you think, but the number is not infinite, and it is shared with every thread in the container.
Zombies are not free: the pids cgroup budget
The pids controller in cgroup v2 exists precisely for this class of exhaustion. The kernel documentation is blunt about why it deserves its own controller: "The number of tasks in a cgroup can be exhausted in ways which other controllers cannot prevent, thus warranting its own controller. For example, a fork bomb is likely to exhaust the number of tasks before hitting memory restrictions."
The two files you care about are on the cgroup of the container (or of your session, which is what I measured):
$ cat /sys/fs/cgroup/pids.max
28649
$ cat /sys/fs/cgroup/pids.current
473
$ cat /sys/fs/cgroup/pids.events
max 0
pids.max is described in the kernel docs as the "Hard limit of number of processes" and pids.current as "The number of processes currently in the cgroup and its descendants". pids.events counts how many times the limit was hit — on an idle host it reads max 0, and any non-zero value there is a past-tense incident report.
Now the part that catches people out. Run a parent that spawns 25 children which exit immediately, and never wait on them:
$ python3 pids_budget.py 25
pids.max = 28649
pids.current = 473 (before any zombie)
zombies = 0
---
spawned 25 children that exit immediately, never waited on
zombies (ps, state Z) = 25
pids.current = 498 (delta +25)
children still in /proc = 172
---
after waitpid(): zombies = 0, pids.current = 473 (delta +0)
Twenty-five dead processes, twenty-five units of budget, held for as long as the parent lives. And the same controller counts threads, because a task is a task: eight Python threads moved the counter by exactly eight.
$ python3 threads_budget.py
pids.current before 8 threads : 472
pids.current with 8 threads : 480 (delta +8)
pids.current after join : 472 (delta +0)
That is the arithmetic behind most "why did my container suddenly hit the pids limit" incidents: a language runtime that spawns a thread per request, a library that leaks two threads per job, plus a handful of unreaped children, all drawing on the same number. If you set --pids-limit ("Tune container pids limit (set -1 for unlimited)") to protect the host, you have also decided how many simultaneous requests your service can serve. Size it deliberately, and read pids.current in your monitoring rather than discovering it from a 500.
One further subtlety is documented and worth knowing before you build your own monitoring: a zombie stays associated with the cgroup it belonged to until it is reaped, but "a zombie process does not appear in cgroup.procs and thus can't be moved to another cgroup." Tools that enumerate cgroup.procs — which includes a lot of container process inspectors — will not list the zombies that pids.current is charging you for. Trust the counter, not the listing.
When the budget runs out: the exact failure strings
When the task counter is full, the kernel refuses the next fork() or clone() with EAGAIN — the same errno you get from RLIMIT_NPROC, which is the limit I could exercise on this host because its cgroup was read-only. Setting ulimit -u to a value below the number of tasks the uid already owns and then forking produces the message you will eventually see in a container:
$ bash -c 'ulimit -u 12; for i in $(seq 1 200); do /bin/true; done'
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: Resource temporarily unavailable
exit status: 254
Bash retries a few times with a delay before giving up and exiting, which is why the first four lines are retry: and the last one is not. The relevant facts for a diagnosis: the string is Resource temporarily unavailable rather than Cannot allocate memory, there is no OOM line in dmesg, memory usage looks fine, and only some operations fail — the process is alive, it simply cannot create more tasks. If you see this in a container, check pids.current against pids.max and the zombie count before you touch a memory limit.
The one-line fix that breaks your exit codes
Search for "how to get rid of defunct processes" and you will find the same advice everywhere: signal.signal(signal.SIGCHLD, signal.SIG_IGN). The kernel will then auto-reap your children, so no zombies appear. It works, and I measured that it works. It also destroys the exit status of every child you will ever spawn, and that part is rarely mentioned.
import os, signal, subprocess, time
signal.signal(signal.SIGCHLD, signal.SIG_IGN) # the widely recommended "fix"
pid = os.fork()
if pid == 0:
os._exit(3) # child exits with status 3
time.sleep(0.5)
try:
print("waitpid ->", os.waitpid(pid, 0))
except ChildProcessError as exc:
print("waitpid -> ChildProcessError:", exc)
done = subprocess.run(["/bin/false"]) # /bin/false exits 1
print("subprocess.run(['/bin/false']).returncode =", done.returncode)
zombies = subprocess.run(
["ps", "-o", "stat=", "--ppid", str(os.getpid())],
capture_output=True, text=True,
).stdout.split()
print("children in state Z =", sum(1 for s in zombies if s.startswith("Z")))
Run it, and the trade becomes visible: no zombie, and no exit status either.
$ python3 sigign_probe.py
waitpid -> ChildProcessError: [Errno 10] No child processes
subprocess.run(['/bin/false']).returncode = 0
children in state Z = 0
/bin/false exits 1. With SIGCHLD ignored, Python reported 0. This is not a Python bug and it is not a quirk of one version: once the status has been auto-reaped, the kernel has nothing left to hand you, and CPython's own source says so in a comment you can read on any machine:
def _try_wait(self, wait_flags):
"""All callers to this function MUST hold self._waitpid_lock."""
try:
(pid, sts) = os.waitpid(self.pid, wait_flags)
except ChildProcessError:
# This happens if SIGCLD is set to be ignored or waiting
# for child processes has otherwise been disabled for our
# process. This child is dead, we can't get the status.
pid = self.pid
sts = 0
return (pid, sts)
Every framework that builds on subprocess, every migration script, every health check that says "if this command failed, abort" now sees success. That is a far worse failure mode than a few rows of <defunct> in ps.
And the disposition is inherited across fork() and execve(), so the damage is not confined to the process that opted in. I set SIGCHLD to SIG_IGN in a parent, re-executed the interpreter, and asked the child what it had inherited:
parent pid=1124560 SIGCHLD before = SIG_DFL
parent SIGCHLD after SIG_IGN = SIG_IGN
child pid=1124561 inherited SIGCHLD disposition = SIG_IGN
Every helper your supervisor launches inherits "I don't care about my children", including the helpers that do care. Do not use SIG_IGN in anything that supervises work.
The correct fix: reap deliberately
Reaping is a loop, not a setting. Install a SIGCHLD handler, call waitpid(-1, WNOHANG) repeatedly until it returns 0 or raises ChildProcessError, and keep the exit status you collected — that is where your restart policy gets its decisions from. The loop (not a single waitpid) is essential: signals are not queued, so several children exiting together deliver one SIGCHLD.
import os, signal, sys, time
def reap(signum, frame):
"""Drain every collectable child. Returns nothing; logs what it took."""
while True:
try:
pid, status = os.waitpid(-1, os.WNOHANG)
except ChildProcessError:
return # no children left to collect
if pid == 0:
return # children exist but none have exited
code = os.waitstatus_to_exitcode(status)
print(f"reaped pid={pid} raw_status={status} exit_code={code}", file=sys.stderr)
signal.signal(signal.SIGCHLD, reap)
for _ in range(5):
if os.fork() == 0:
os._exit(3)
time.sleep(30)
Two details that bite in production. The raw status is not the exit code: a child that exits 3 produces status=768, and os.waitstatus_to_exitcode(768) is 3 (measured). Do not compare raw statuses to small integers. And if you hold a lock or a database handle while reaping, call waitpid with WNOHANG and return immediately rather than doing slow work in the handler.
$ grep -E '^(Name|State|PPid|Threads):' /proc/1124388/status # the leaked child
State: Z (zombie)
$ ps -eo stat,pid,comm | awk '$1 ~ /^Z/' | wc -l # after the fix
0
Language runtimes differ, and it is worth knowing which side of the line yours is on. Node's child_process reaps for you: libuv installs a SIGCHLD handler, and after spawning ten short-lived children I measured zero <defunct> children in the Node process. Shell scripts are fine as long as you wait for the jobs you start; a background job you never wait for is a zombie until the script exits. Python's subprocess reaps, but bare os.fork() does not — and bare os.fork() is what the tutorial you copied used. The rule is mechanical: whatever you fork, you must wait for, or hand the problem to something that will.
Put an init in front: --init, tini, dumb-init
If you would rather not add a reaping loop to an application that has no business owning one, delegate it. Docker does this for you with a flag:
# One-shot: inject a tiny init as PID 1 inside the container.
docker run --init --pids-limit 512 --rm your/image:tag
The init that gets injected is tini, and Docker's documentation describes its responsibilities in exactly the terms this article has been using: it "inserts a tiny init-process into the container as the main process, and handles reaping of all processes when the container exits". Tini's own README is even more direct about why you want it: it "spawn[s] a single child ... and wait[s] for it to exit all the while reaping zombies and performing signal forwarding", which "protects you from software that accidentally creates zombie processes, which can (over time!) starve your entire system for PIDs".
If you cannot use --init — older runtimes, Kubernetes, a custom entrypoint — install the init in the image and make it the entrypoint instead. On Alpine the package and path are documented upstream:
FROM alpine:3.20
RUN apk add --no-cache tini
# Tini is now available at /sbin/tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["/app/server"]
The same pattern with dumb-init is equivalent for reaping purposes. Either way, verify it took effect the same way you verified the leak: ps -eo stat,ppid,pid,comm | awk '$1 ~ /^Z/' inside the running container should stay empty while work is in flight.
Two honest caveats about the execution of that last part. I could not run tini on this host: the binary at /usr/bin/tini is s6-overlay's suexec wrapper, and it refuses to start outside PID 1 with s6-overlay-suexec: fatal: can only run as pid 1. The --init behaviour above is therefore quoted from the Docker documentation, not measured here. What was measured is everything the injected init does: reaping via waitpid(-1, WNOHANG), zombie accounting in pids.current, the absence of zombies when a reaper is installed, and the exit-status destruction when reaping is done by SIG_IGN instead.
One thing --init does not fix: it reaps orphans, not your own unreaped children. If your application calls fork() and drops the PID, the zombie is still your application's child and still its responsibility. --init saves you from the children of processes that have died; it does not save you from code that forgets wait.
What is not a zombie problem
Two neighbouring failures get filed under this heading and should not be, because the diagnosis and the fix are different.
Exit code 137 / OOMKilled. 137 is 128 + 9: the kernel sent SIGKILL, usually the cgroup OOM killer. The process is gone, not stuck in the process table, and reaping has nothing to do with it. If your container dies with 137, look at memory.max, memory.events and the memory graph, not at pids.
Too many open files and file-watcher exhaustion. Same family of "resource limit" errors, different controller and different strings. Docker containers inherit an inotify watch budget from the host, and the resulting failures look nothing like a zombie leak — I walked through that one in System limit for number of file watchers reached in Docker, including how to tell a per-watcher failure from a per-descriptor one.
And if you cannot inspect the container at all because the client is talking to the wrong socket, that is a different post: Cannot connect to the Docker daemon: four branches behind one message covers the four distinct causes of that one message.
Decision table
| What you see | What it means | Fix |
|---|---|---|
Z state, <defunct>, count stable | Terminated child, parent never waited | Reaping loop in the parent; waitpid(-1, WNOHANG) |
Z state, count growing over days | Long-lived parent leaks one slot per job | Fix the parent's fork sites; --init only helps orphans |
pids.current near pids.max, pids.events non-zero | Task budget exhausted (threads also count) | Raise --pids-limit and fix the leak |
fork: retry: Resource temporarily unavailable | Kernel refused fork()/clone() with EAGAIN | Check pids.current against pids.max, zombies, ulimit -u |
Exit code 137, OOMKilled | Kernel killed it for memory | Memory limits and leak hunting, not reaping |
| Exit statuses always 0 | SIGCHLD is SIG_IGN, possibly inherited | Remove the SIG_IGN; install a real handler |
Verification: what was measured and what was quoted
Measured on 2026-09-18, Linux 6.17.0-1019-oracle (cgroup v2, pids.max 28649), CPython 3.13.5, Node 26.5.1, on a scratch harness under /opt/data/c2cz/scratch/lab0918/: the zombie reproduction and ps//proc evidence; the reaped-on-parent-exit transition; pids.current deltas for 25 unreaped children and for 8 threads; the waitpid and reaper-loop variants; the SIG_IGN probes including the returncode = 0 result for /bin/false and the inheritance of the disposition across execve; the kill -9 against a zombie; the ChildProcessError on waitpid; os.waitstatus_to_exitcode(768) = 3; the Node child_process probe; and the EAGAIN fork-failure transcript, which I produced with ulimit -u because the cgroup filesystem was read-only on this host and pids.max could not be lowered without harming a shared machine.
Quoted from primary documentation, not executed here: the Docker statements about --init, --pids-limit and the injected init process; the kernel cgroup v2 statements about pids.max, pids.current, task exhaustion and zombies not appearing in cgroup.procs; and the tini entrypoint form, which could not run on this host because its /usr/bin/tini is the s6-overlay wrapper that refuses to start as anything but PID 1.
Last verified: 2026-09-18. Commands, outputs and version strings above were produced by the harness described in this section; re-run them before trusting them against a kernel or runtime newer than the ones listed.