When a Linux server starts behaving strangely, the first tools administrators usually reach for are familiar ones: top to identify CPU-hungry processes, vmstat to inspect memory and scheduling, iostat to analyze storage, and strace to follow system calls. The problem comes when all of them show that something is wrong, but none of them explains why. bpftrace lets you go one level deeper and observe what is happening inside the kernel and applications in real time, without recompiling programs or rebooting the server.

The key points about bpftrace in 30 seconds

bpftrace uses eBPF to run controlled observability programs inside the Linux kernel.
It can trace system calls, kernel functions, processes, networking, memory, storage, and user-space functions.
Its syntax is reminiscent of awk, and many investigations can be solved with a single command.
Tracepoints, kprobes, uprobes, maps, and histograms help move from observing symptoms to identifying root causes.
In production, it is best to filter events early, avoid excessive printf(), and prefer stable interfaces such as tracepoints.

eBPF has become one of the most interesting technologies in the Linux ecosystem in recent years. Tools such as Cilium, Falco, BCC, and Pixie rely on it to observe or control systems without introducing permanent kernel modifications.

bpftrace makes those capabilities much more accessible to systems administrators. Instead of writing eBPF programs in C, compiling them, and manually managing how they are loaded, it allows many investigations to be expressed as small scripts.

A minimal example makes the philosophy clear:

sudo bpftrace -e 'BEGIN { printf("Hello from bpftrace!\n"); }'Code language: JavaScript (javascript)

It may not look like much, but behind that line bpftrace generates the corresponding program, loads it into the kernel, and prepares it for execution.

What eBPF Brings to Linux Troubleshooting

BPF was originally created for packet filtering. Its evolution into eBPF, extended Berkeley Packet Filter, dramatically expanded its possibilities and turned it into an event-driven execution engine inside the kernel.

An eBPF program can be triggered when, among many other things:

a system call runs;
a kernel function starts or returns;
a file is opened;
a network connection is created;
a process starts or terminates;
memory is allocated;
a hardware performance counter fires.

This makes it possible to answer questions that traditional monitoring systems often struggle with.

For example:

Which process is deleting this file?

Which application is opening thousands of files?

Which kernel function is consuming CPU?

How long does a read() call really take?

Which processes are creating the most connections?

Which execution path is causing unexpected latency?

eBPF also includes a verifier that checks programs before allowing them to run. Once accepted, they can be JIT-compiled into native code and executed only when the event they are attached to occurs.

That does not mean every script is automatically safe to run carelessly in production. A trace attached too broadly, or to an event that fires millions of times per second, can still add load. The important difference is that eBPF provides an infrastructure specifically designed for this type of observability.

bpftrace Syntax Is Surprisingly Simple

Programs usually follow this structure:

probe /predicate/
{
    action
}

The probe defines which event to observe.

The predicate filters when the action should run.

The action defines what to do when the event occurs.

For example:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat
/pid == 1234/
{
    printf("Process %s opened a file\n", comm);
}'Code language: PHP (php)

This observes openat(), but only when the call comes from process ID 1234.

Filtering early is especially important on busy production systems.

Tracepoints vs. kprobes

One of the first decisions when using bpftrace is choosing where to attach the probe.

Tracepoints are events deliberately exposed by the kernel. Their main advantage is stability.

For example:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat
{
    printf("%s %s\n", comm, str(args.filename));
}'Code language: PHP (php)

This lets you observe files that processes are trying to open.

kprobes, by contrast, attach directly to internal kernel functions:

sudo bpftrace -e '
kprobe:do_sys_openat2
{
    printf("%s PID=%d\n", comm, pid);
}'Code language: PHP (php)

They provide very deep visibility, but come with a trade-off: internal kernel function names and arguments can change between versions.

That is why, when an equivalent tracepoint exists, it is usually the safer option.

Available probes can be listed with:

sudo bpftrace -l 'tracepoint:*'Code language: JavaScript (javascript)

or:

sudo bpftrace -l 'kprobe:*open*'Code language: JavaScript (javascript)

You Can Trace Applications Without Changing Their Code

bpftrace does not stop at the kernel.

uprobes let you attach to functions inside user-space applications and shared libraries.

For example, to monitor calls to malloc():

sudo bpftrace -e '
uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc
{
    printf("%s PID=%d called malloc()\n", comm, pid);
}'Code language: PHP (php)

The corresponding return probe, uretprobe, lets you inspect what the function returns.

Used together, they can measure latency:

sudo bpftrace -e '
uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc
{
    @start[tid] = nsecs;
}

uretprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc
{
    @latency = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
}'Code language: PHP (php)

This can be very useful when investigating production applications that cannot easily be recompiled with custom instrumentation.

There are also USDT, User Statically Defined Tracing, probes intentionally added by application developers to expose important internal events.

Maps Turn Millions of Events into Useful Information

Printing every event is usually a bad idea.

If a server performs thousands of system calls per second, the terminal quickly becomes noise, and the printf() itself can create unnecessary overhead.

That is where maps and aggregations become valuable.

To count which programs call openat() most frequently:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat
{
    @[comm] = count();
}'Code language: PHP (php)

The output might look like this:

@["bash"]: 42
@["vim"]: 18
@["nginx"]: 156Code language: JavaScript (javascript)

Other useful aggregation functions include:

Function	Purpose
`count()`	Count events
`sum()`	Sum values
`avg()`	Calculate averages
`min()`	Record the minimum
`max()`	Record the maximum
`hist()`	Build logarithmic histograms
`lhist()`	Build linear histograms
`stats()`	Report count, total, and average

Histograms are especially useful for latency analysis because they avoid reducing a complex distribution to a single average.

Twelve Useful Investigations for Linux Administrators

One of the best ways to learn bpftrace is by using it against real problems.

To observe file opens:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat
{
    printf("%-16s %s\n", comm, str(args.filename));
}'Code language: PHP (php)

To identify which processes generate the most system calls:

sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter
{
    @[comm] = count();
}'Code language: PHP (php)

To monitor new processes:

sudo bpftrace -e '
tracepoint:sched:sched_process_fork
{
    printf("%s -> %s\n",
           str(args.parent_comm),
           str(args.child_comm));
}'Code language: PHP (php)

To identify which process is deleting files:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_unlinkat
{
    printf("%s PID=%d\n", comm, pid);
}'Code language: PHP (php)

To measure how long read() calls take:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_read
{
    @start[tid] = nsecs;
}

tracepoint:syscalls:sys_exit_read
/@start[tid]/
{
    printf("%s took %llu ns\n",
           comm,
           nsecs - @start[tid]);
    delete(@start[tid]);
}'Code language: PHP (php)

To see which applications create the most network connections:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_connect
{
    @[comm] = count();
}'Code language: PHP (php)

To get a rough view of top CPU consumers:

sudo bpftrace -e '
profile:hz:99
{
    @[comm] = count();
}'Code language: PHP (php)

And to identify the most frequent kernel stack traces:

sudo bpftrace -e '
profile:hz:99
{
    @[kstack] = count();
}'Code language: PHP (php)

That last approach can be particularly useful as a starting point for profiling and understanding where CPU time is really being spent.

From One-Liners to Reusable Scripts

Once an investigation grows beyond a few lines, keeping everything inside -e becomes awkward.

Programs can instead be stored in .bt files.

For example:

tracepoint:syscalls:sys_enter_openat
{
    @[comm] = count();
}

interval:s:5
{
    printf("=== File Opens ===\n");
    print(@);
    clear(@);
}Code language: PHP (php)

Then run it with:

sudo bpftrace opens.btCode language: CSS (css)

This makes it possible to build an internal troubleshooting library, version it in Git, and share it across operations teams.

An organization can eventually maintain dedicated scripts for PostgreSQL, Nginx, Kubernetes, networking, storage, or specific internal applications.

bpftrace Does Not Replace top, strace, or perf

It would be a mistake to think of bpftrace as a replacement for traditional Linux tools.

Each tool answers different questions well.

top remains excellent for quickly identifying processes with high CPU usage.

vmstat provides an immediate view of CPU, memory, process scheduling, and wait states.

iostat remains highly useful for storage analysis.

strace is still extremely effective for examining the system calls of a specific process.

perf offers advanced profiling capabilities.

bpftrace becomes especially useful when an investigation requires asking a very specific question about system behavior that standard metrics do not directly answer.

That is where it stands out.

How to Use bpftrace Without Making Troubleshooting Worse

The power of bpftrace requires discipline.

In production, a few basic rules are worth following:

Prefer tracepoints when a stable interface is available.
Use predicates to filter events as early as possible.
Aggregate with maps instead of printing every event.
Understand how frequently a probe can fire before attaching to it.
Delete temporary map entries when they are no longer needed.
Use BTF where available to work with kernel types.
Validate new scripts in development or staging.
Attach only to the probes needed for the investigation.

It is also important to remember that dynamic instrumentation is not completely free. A poorly designed script attached to an extremely frequent event can affect the system you are trying to diagnose.

That matters even more on critical servers.

bpftrace represents a shift in mindset for Linux administrators. Instead of only reading predefined metrics, they can ask their own questions about the system while it is still running.

When top shows the symptom, iostat confirms that something is wrong, and strace still does not fully explain the cause, eBPF and bpftrace provide a much deeper window into what Linux is actually doing.

Frequently Asked Questions

What is the difference between eBPF and bpftrace?

eBPF is the Linux kernel technology that allows event-driven programs to run safely. bpftrace provides a high-level language that makes writing tracing programs on top of eBPF much easier.

Is bpftrace safe to use in production?

It can be used in production, but scripts should be designed carefully. Filtering events, avoiding massive output, and testing complex traces beforehand is recommended.

Should I use tracepoints or kprobes?

When a suitable tracepoint exists, it is generally preferable because it provides a more stable interface. kprobes can reach deeper into kernel internals, but those internals may change between kernel versions.