Linux for production engineers

Why "too many open files" is not about disk

When a service under load starts logging EMFILE: too many open files, the word "files" misleads people. Every open thing your process holds is a file descriptor: an open file, yes, but also every TCP socket, every pipe, every epoll instance. A socket is a file descriptor. So a busy server with 20,000 live connections is holding at least 20,000 fds, and the kernel caps how many one process may have.

That cap is RLIMIT_NOFILE. It has a soft limit (what is enforced right now) and a hard limit (the ceiling you can raise the soft limit to without privileges). Cross the soft limit and the next open, socket, accept, or pipe fails with EMFILE.

ulimit -n                 # your process's soft limit (often 1024 by default)
ls /proc/<pid>/fd | wc -l # how many fds this process holds right now
cat /proc/<pid>/limits    # the actual soft/hard NOFILE for a running process

Wrong "It ran out of memory, add RAM." An fd is a small entry in a kernel table, not a buffer, so more memory changes nothing. Right raise the soft limit (ulimit -n, or LimitNOFILE= in the systemd unit) to buy headroom, then find what is not being closed. A steadily climbing fd count under steady traffic is a leak: a client you open per request and never close, a connection pool with no cap. Raising the limit only moves the deadline.

Reading ps and top without fooling yourself

ps and top show two memory numbers per process and they mean different things. VSZ (virtual size) is the whole address space the process has mapped, including memory it reserved but never touched and shared libraries it barely uses. Resident set size (RSS) is the physical RAM actually backing the process right now. VSZ is often many times RSS and is mostly noise; RSS is what you watch.

The STAT column tells you what a process is doing:

R  running or runnable          S  sleeping (waiting on something)
D  uninterruptible sleep        Z  zombie (exited, not yet reaped)
   (usually blocked on I/O)     T  stopped (e.g. by SIGSTOP)

A process stuck in D is blocked in the kernel, normally on disk or network I/O, and you cannot even SIGKILL it out of that state until the I/O returns. A pile of D processes usually means a slow disk, not a CPU problem.

Defunct processes and why PID 1 is special

When a child process exits, it does not vanish. The kernel keeps a stub (its exit code and some accounting) until the parent calls wait() to collect it. Between exit and that collection the child is a zombie, shown by ps as state Z and <defunct>. A zombie uses no CPU and almost no memory; it holds one slot in the process table. You cannot kill it, because it is already dead. It leaves only when its parent reaps it.

If a parent dies before reaping its children, those orphans are handed to PID 1, the first process on the system, which is expected to reap them. This is where containers bite. Inside a container your app is usually PID 1, and most app runtimes never reap stray children. So if your service shells out to other processes, their leftovers can accumulate as zombies until the process table fills. The fix is to run a tiny init as PID 1 (docker run --init, or bake in tini) whose whole job is to reap and to forward signals.

What exit code 137 means

A process that dies from a signal reports exit code 128 + signal number. That decodes the two you will see most:

137 = 128 + 9   killed by SIGKILL   (often the OOM killer, or a grace period that expired)
143 = 128 + 15  killed by SIGTERM   (a normal stop request the process did not survive)

So a container that keeps restarting with 137 was force-killed, and the usual culprit is the out-of-memory killer hitting the container's memory limit. A 143 means something asked it to stop with SIGTERM and it exited on that signal rather than draining first. The number is a fact about how the process ended, and it points you straight at whether to look at memory, at your shutdown handler, or at a deploy that timed out.

What fork and exec actually do

Creating a process on Linux is two mechanisms, not one. fork() makes a child that is a near-copy of the parent, and exec() replaces the current program image with a new one. A shell running node server.js forks itself, then the child execs into node.

fork() does not copy the parent's memory. Linux uses copy-on-write: parent and child share the same physical pages read-only, and a page is duplicated only when one side writes to it. The man page is explicit that the cost is "the time and memory required to duplicate the parent's page tables," not the whole heap. Two consequences seniors trip over: forking a process with a large resident heap can briefly need real memory as writes fault in copies (this is the latency spike behind a Redis BGSAVE or a heavy fork() under write load), and every open fd is inherited, with the child's fd pointing at the same open file description as the parent, so they share the file offset. A child that writes to an inherited log fd advances the same offset the parent sees.

exec() throws away that memory image and loads a new one, but the fd table survives the exec (minus any marked close-on-exec). Signal dispositions are treated in between: a fork() child inherits the parent's signal handlers, and across exec() handled signals reset to their default while ignored signals stay ignored. That reset is why a freshly exec'd program starts with default signal behavior regardless of what its launcher installed.

How PID 1 changes signal delivery

Signals have default actions the kernel takes when no handler is installed: Term (terminate), Core (terminate and dump core), Ign (ignore), Stop, and Cont. SIGTERM and SIGINT default to Term; SIGCHLD defaults to Ign. Two signals, SIGKILL and SIGSTOP, can never be caught, blocked, or ignored.

PID 1 is the exception that breaks container shutdowns. For the init process of a PID namespace, the kernel disables the default dispositions: per pid_namespaces(7), "only signals for which the 'init' process has established a signal handler can be sent to the 'init' process by other members of the PID namespace," and SIGKILL and SIGSTOP from an ancestor namespace are the only ones "forcibly delivered." So a container whose PID 1 never installed a SIGTERM handler simply drops the SIGTERM that docker stop or a rolling deploy sends, waits out the whole grace period, and then eats the SIGKILL the runtime sends to end it.

Wrong "Switch the Dockerfile to the exec form so the app is PID 1 and it shuts down gracefully." Exec form is necessary but not sufficient. If the app is PID 1 and has no SIGTERM handler, the kernel still drops SIGTERM and the grace period still expires. Right make the app PID 1 with the exec form AND install a SIGTERM handler that stops accepting, drains in-flight work, and exits, or run a real init (tini, docker run --init, dumb-init) at PID 1 that forwards signals to your app and reaps orphaned children. If PID 1 itself exits, the kernel kills every other process in the namespace with SIGKILL, so the init has to outlive the workers.

RSS, virtual memory, and what the OOM killer counts

/proc/<pid>/status splits memory into VmSize (the whole virtual address space) and VmRSS (resident physical pages), and since Linux 4.5 breaks RSS into RssAnon (heap and stack), RssFile (file-backed mappings, including the executable), and RssShmem (shared and tmpfs memory). VmSize is mostly reservations and shared libraries and tells you little. RssAnon is the number that gets a process OOM-killed.

Two facts matter under a memory limit. First, RSS counts shared pages in full for every process that maps them, so adding up the RSS of ten workers that share a large read-only heap wildly overcounts real usage; PSS (in /proc/<pid>/smaps_rollup) divides shared pages by their sharers if you need the honest number. Second, the page cache is not in any process's RSS. Files the app reads sit in the kernel's page cache, counted in the cgroup's memory.current but reclaimable: under pressure the kernel evicts clean cache before it OOM-kills anything.

Wrong "Reading big files raised RSS and OOM-killed the container." Page cache is reclaimable and does not count as RSS; the kernel drops it first. What actually drives an in-cgroup OOM is anonymous memory that cannot be reclaimed: a real leak, an unbounded in-memory cache, a spike in live objects. Watch RssAnon, not the resident total.

CPU throttling versus memory kills

Both limits are cgroup v2 knobs, and os-fundamentals already covers the mechanisms: memory.max fires the in-cgroup OOM killer while memory.high only reclaims, and cpu.max caps bandwidth in microseconds so a tight cap adds latency even when average CPU looks low. What this module adds is reading the interface files that say which failure happened, because the two demand opposite fixes.

Throttling is counted in cpu.stat:

cpu.stat:      nr_throttled 8123        # periods where the group was parked at its cap
               throttled_usec 41290000  # total time parked waiting for the next period

Climbing throttled_usec against idle-looking CPU is the signature: the group is fine on average but stalls at the cap inside each period, and that stall is the tail latency.

A memory kill leaves a different fingerprint. The victim exits 137, which decodes as 128 + 9: the shell adds 128 to the terminating signal number, and 9 is SIGKILL. Confirm it came from this cgroup rather than the host in memory.events:

memory.events: oom 3          # allocations that hit the limit and could not reclaim
               oom_kill 1     # processes the OOM killer actually killed here

So the diagnostic split is clean: slow but stable with climbing throttled_usec is CPU throttling, and you loosen cpu.max or cut the work; a restart with exit 137 and a nonzero oom_kill is a memory kill, and you raise memory.max or fix the leak. Raising the CPU limit does nothing for OOM restarts, and raising memory does nothing for throttling latency. Confusing the two sends you tuning the wrong knob for days.

The 60-second triage, then strace and lsof

Brendan Gregg's first-60-seconds checklist is the fastest way to place a problem on a resource before you theorize: uptime (load averages, remembering that Linux load counts uninterruptible-sleep tasks so a high load with idle CPUs means I/O waiting, not CPU saturation), dmesg | tail (OOM kills and driver errors land here), vmstat 1, mpstat -P ALL 1, pidstat 1, iostat -xz 1, free -m, sar -n DEV 1, sar -n TCP,ETCP 1, and top. It feeds the USE method: for each resource check utilization, saturation, and errors.

Once you know the resource, two tools go deeper. lsof lists open files, so lsof -p <pid> counts and categorizes a process's fds (the fastest way to confirm an fd leak) and lsof -i :8080 names what holds a port. strace shows the syscalls a process makes: strace -f -p <pid> attaches to a running process and its threads, and strace -f -e trace=network narrows to one class. It answers "what is this stuck process actually waiting on" when a stack trace will not. The caution is real: strace uses ptrace and stops the target on every syscall, so it can slow a hot process by an order of magnitude. Attach it to a sick process you are debugging, never leave it on a healthy one under load. When even that is too invasive, /proc/<pid>/ is the zero-overhead ground truth: status for memory and state, fd/ for open descriptors, limits for the real RLIMIT values, and maps for the address-space layout.