Diagnose network mbuf exhaustion

Synopsis #

FreeBSD uses mbufs and several cluster sizes for network data. The kern.ipc.nmbclusters loader tunable limits one of those pools. A timeout, slow transfer, dropped packet, or busy firewall does not by itself show that the limit is too low.

Diagnose mbuf exhaustion from allocation counters collected during the failing workload. Change the loader tunable only when the relevant cluster limit is reached and denied or delayed requests increase with the incident. An arbitrary large value can reserve excessive kernel memory or prevent a successful boot.

Capture evidence before changing anything #

Record the release, uptime, configured limit, and complete mbuf report:

$ freebsd-version -kru
$ uptime
$ sysctl kern.ipc.nmbclusters
$ netstat -m

netstat -m reports current, cached, total, and maximum counts for ordinary and jumbo clusters. It also reports denied and delayed allocation requests separately for mbufs, clusters, combined mbuf-and-cluster objects, and jumbo clusters.

Save one report before the workload, another while the failure is reproducible, and a third after it subsides. The counters are more useful as a change over the observation interval than as an isolated number.

Interpret the counters #

Evidence of ordinary mbuf-cluster exhaustion normally has all of these properties:

ordinary cluster use approaches its reported maximum;
the cluster or combined allocation-denial counter increases during the incident;
the increase coincides with an application error, packet loss, or throughput collapse;
reducing the workload stops the increase.

A nonzero denial counter in one later snapshot proves that a denial occurred, but not that it caused the present failure. A high-water mark near the maximum is also insufficient when current use has fallen and denial counters remain unchanged.

The counter category matters. A jumbo-cluster denial, an sfbuf shortage, a full socket listen queue, and an ordinary cluster denial are different resource limits. Raising kern.ipc.nmbclusters is not a demonstrated correction for every line in netstat -m.

When the denial counters remain unchanged, continue with Diagnose DNS, routing, and firewall paths . Interface errors, route selection, PF state, socket queues, and application limits can produce similar symptoms without exhausting network memory.

Relate pressure to the workload #

Repeat the observation with one controlled change in load. Record at least:

concurrent connections and expected socket-buffer sizes;
traffic rate and packet size;
enabled firewalls, bridges, jails, VPNs, and packet-capture processes;
the exact netstat -m counter that grows;
available physical memory and other kernel-memory consumers.

This record provides a basis for capacity planning. It also prevents a larger pool from hiding an unbounded connection leak or an unexpectedly amplified traffic path.

Plan a loader-tunable change #

kern.ipc.nmbclusters is set by the boot loader and cannot be safely evaluated as an ordinary writable runtime sysctl. Derive a proposed limit from measured peak use, observed denied allocations, expected growth, and the host’s memory budget. There is no release-independent value suitable for every firewall, storage server, hypervisor, or desktop.

After review, place the single assignment in /boot/loader.conf and schedule a reboot with console access. The assignment has the form kern.ipc.nmbclusters="value", where value is the reviewed numeric limit rather than a copied example. See Make a system setting persistent for the distinction between loader tunables and runtime sysctls.

Preserve the previous file and record the exact rollback before rebooting. At the loader prompt, a problematic override can be bypassed for one boot:

unset kern.ipc.nmbclusters
boot

After the system starts, remove or correct the assignment in /boot/loader.conf before the next reboot.

Verify the result #

After reboot, confirm the active limit and repeat the same workload and observation interval:

$ sysctl kern.ipc.nmbclusters
$ netstat -m

Success means that the relevant denial counter no longer increases under the accepted workload, memory remains within the planned budget, and the original application-level failure is absent. A larger displayed maximum alone is not an acceptance test.