Synopsis #
DTrace can observe kernel and process behavior while the system is running. It is most useful after ordinary counters have narrowed a performance symptom to a question about latency, call paths, scheduling, I/O, networking, or a particular process.
Begin with a bounded observation, a narrow probe description, and an aggregation rather than printing every event. Record the exact kernel, workload, duration, probe set, and D program with the output. Do not convert the first trace into a loader tunable or kernel change without a repeatable baseline and an explanation of the causal mechanism. If the objective is durable attribution of security events, use security event auditing instead.
Use a simpler tool first when it answers the question #
DTrace is not required to establish that a host is busy or a filesystem is full. Begin with base-system summaries:
$ uptime
$ top -SH
$ vmstat -w 1
$ iostat -x -w 1
$ systat -vmstat 1
Select the tool that matches the symptom and stop it after a representative interval. Useful first distinctions include:
| Symptom | First evidence | DTrace becomes useful when |
|---|---|---|
| High CPU | top, systat, and per-process state | The hot process, kernel path, or scheduler behavior remains unclear |
| Slow storage | iostat, pool status, and application latency | Latency must be associated with operations, devices, or call paths |
| Memory pressure | vmstat, swap activity, and process size | Allocation or reclaim behavior needs event-level observation |
| Network failure or pressure | netstat, sockstat, interface counters | Packet, protocol, or socket behavior needs correlation |
| One application stalls | Application logs and process state | System calls or userland function boundaries need tracing |
For mbuf allocation failures, follow the mbuf exhaustion guide before considering a loader value. DTrace can refine a diagnosis, but it should not replace the allocator’s own counters.
Form one traceable question #
A useful question names an event, subject, and interval. Examples include:
- which executable names consumed on-CPU samples during a 30-second latency event;
- which system call returned a specific error for one process;
- how long one I/O operation class remained in flight;
- whether a process was runnable but waiting for CPU;
- which kernel stacks appeared while one reproducible workload ran.
“Why is the server slow?” is too broad. Split it after collecting ordinary counters. A trace should confirm or reject one hypothesis, not collect every available probe in case an answer appears later.
Record context before tracing:
$ date -u
$ freebsd-version -kru
$ uname -m
$ uptime
Also record the workload start and stop time, relevant application version, jail identity, and any configuration change preceding the symptom. Output without this context is difficult to compare after a reboot or upgrade.
Understand providers and probes #
A DTrace probe description has four fields:
provider:module:function:name
Providers expose different stable and implementation-specific boundaries:
| Provider family | Suitable question | Caution |
|---|---|---|
profile | Periodic on-CPU sampling and bounded timers | Sampling frequency and stack collection add cost |
sched | Scheduling, run queue, on-CPU, and off-CPU behavior | Probe names and arguments must be checked locally |
syscall | System call entry, return, error, and latency | A wildcard with per-event printing can be extremely noisy |
io | Block I/O lifecycle and latency | Relate device-level events to the actual storage stack |
vfs | Filesystem operations | It does not by itself identify physical device latency |
proc | Process lifecycle | Filtering by process or identity is normally required |
tcp, udp, and ip | Protocol and packet behavior | Captured addresses and process data can be sensitive |
pid | Function boundaries in a selected user process | Symbols, argument types, and support vary by binary and release |
fbt | Kernel function boundary investigation | Kernel implementation details are unstable and broad matches are costly |
FreeBSD provider availability and argument types are not guaranteed to match Solaris or another DTrace implementation. A script copied from another system must be reviewed against the local provider manual page and probe listing.
Discover the local probe surface #
On FreeBSD, DTrace use is restricted to root. Running dtrace(1) loads the necessary modules on supported GENERIC kernels. List rather than enable probes first:
# dtrace -l | less
# dtrace -l -n 'profile:::'
# dtrace -l -n 'sched:::'
The list identifies the probe ID, provider, module, function, and name available on the running kernel. If a provider expected from a document is absent, confirm the architecture, kernel configuration, loaded modules, and the manual page for that release. Do not solve an absent optional provider by immediately building a custom production kernel.
Review scripts shipped in /usr/share/dtrace as FreeBSD-specific examples:
$ find /usr/share/dtrace -type f -print
Read a script completely before running it. D programs execute with substantial observation privilege, and their output can expose kernel addresses, paths, process arguments, network endpoints, and workload data.
Run a bounded first observation #
The profile provider is a practical first step when ordinary counters show CPU activity but not its distribution. This example samples executable names at 97 Hz and exits after 30 seconds:
# umask 077
# capture_dir=$(mktemp -d /var/tmp/dtrace.XXXXXXXX)
# dtrace -q -n 'profile-97 { @[execname] = count(); } tick-30s { exit(0); }' > "${capture_dir}/profile.txt"
The odd sampling frequency reduces the chance of synchronizing with a periodic workload. The tick-30s clause provides an explicit stop condition. The aggregation counts in memory and prints at exit instead of writing one line for every sample. mktemp creates an unpredictable private directory, while umask 077 protects its capture from other local users. A predictable filename directly under a world-writable directory is unsafe for root output because an existing file or symbolic link can redirect the write.
Treat the result as a distribution of samples, not an exact accounting ledger. A process appearing often was observed on CPU often during that interval. The trace does not by itself prove why it consumed CPU or whether that activity caused the reported latency.
If kernel stacks are required, the dtrace_profile(4) manual page provides a bounded stack-sampling pattern. Stack collection increases overhead and output sensitivity. First reduce the subject, frequency, and interval, then compare the same workload with and without tracing.
Keep the probe set and actions narrow #
The cost of DTrace depends on probe firing rate, enabled probe count, predicates, actions, stack unwinding, copying, and output volume. The fact that disabled probes have low cost does not make an enabled wildcard inexpensive.
Prefer:
- a stable provider that directly represents the event;
- an exact process, UID, jail, function, or error predicate;
count(),sum(),avg(), or a distribution aggregation;- a timer probe that exits after a stated duration;
- one capture file with restrictive permissions;
- a repeated control run to estimate observer effect.
Avoid beginning with:
- every
fbtorsyscallprobe; printf()ortrace()on a high-frequency event;- full kernel and user stacks for every event;
- an unbounded script left attached to a production workload;
- destructive actions or the option that permits them;
- an unreviewed script downloaded for another operating system or release.
DTrace denies destructive actions by default. That protection should remain in place for performance investigation.
Protect privilege and captured data #
DTrace is root-only on FreeBSD because its observations can cross process and kernel boundaries. The trace operator can see data that application users cannot. Keep the command, script, and capture under the same change-control and access rules as other privileged diagnostics.
Before transferring a capture, inspect it for:
- usernames and numeric identities;
- executable arguments and environment-derived data;
- file paths and dataset names;
- network addresses and ports;
- kernel symbols and stack addresses;
- customer or application data copied by a probe action.
Retain the raw capture only as long as the investigation requires. A pasted excerpt should include the probe description and context needed to interpret it, but omit unrelated sensitive records.
Separate observation from remediation #
A useful investigation produces:
- the original symptom and user-visible measurement;
- the ordinary counters that selected a subsystem;
- the exact D program and local probe listing;
- the bounded capture and an untraced control interval;
- a hypothesis that explains both sets of evidence;
- a reversible change and a measurement that can disprove the hypothesis.
Do not add loader.conf or sysctl.conf values during the discovery run. A setting belongs in persistent configuration only after the owning manual page identifies the interface, a measured limit is reached, and a controlled change improves the original outcome. Review Make a system setting persistent
after that evidence exists.
When the trace points to application code, reproduce it in a staging environment and use application profiling or debugging tools where possible. When it points to unstable kernel implementation details, preserve the exact FreeBSD revision and ask for review with the smallest reproducer rather than turning an FBT symbol into a permanent monitoring contract.