Skip to content

Probes (kprobes/uprobes)

Linux kernel probes (kprobes and kretprobes) are mechanisms for dynamically instrumenting kernel functions to trace execution paths and measure performance. These probes are used by eBPF-based tools like BCC (Berkeley Packet Filter Compiler Collection), enabling low-overhead analysis of system behavior without modifying kernel source code. kprobes intercept kernel function entry points, while kretprobes capture exit points, allowing precise measurement of function execution time and call chains.

What are kprobes and kretprobes?

kprobes allow inserting probes at arbitrary kernel function entry points. When a function is called, the probe executes a user-defined handler, enabling tasks like counting invocations, logging arguments, or measuring latency. kretprobes operate similarly but are attached to function return addresses, making them ideal for tracking the time spent inside a function. Together, they provide visibility into kernel execution flow, critical for diagnosing performance bottlenecks or understanding system call behavior.

Mechanism of Interception

kprobes leverage the kernel's built-in infrastructure to redirect execution to user-space handlers. When a probe is attached to a function, the kernel replaces the function's first instruction with a jump to the probe handler. After the handler completes, execution resumes at the original function. kretprobes use a similar approach but modify the return address, redirecting control to the handler before resuming the function's execution.

It is important to note that kprobes and kretprobes are distinct kernel tracing mechanisms, separate from eBPF itself. eBPF-based tools like BCC abstract these low-level details, providing high-level utilities like trace and bpftrace to manage probes. For example, BCC's trace tool automatically handles probe registration, data collection, and output formatting, reducing the complexity of raw eBPF programming.

Example Tracing with BCC

Here are examples of using BCC tools to trace kernel functions:

Counting Function Invocations

# Count how many times the do_sys_open function is called
sudo trace -p $(pidof systemd) -e do_sys_open

Measuring Function Latency

# Measure the time spent in the vfs_read function
sudo trace -p $(pidof systemd) -e vfs_read --latency

Combining kprobes and kretprobes

# Trace entry and exit of the sys_open function
sudo bpftrace -e 'kprobe:sys_open { printf("Enter: %s\n", ctx->args->filename); } kretprobe:sys_open { printf("Exit: %s\n", ctx->ret->filename); }'

These examples demonstrate how BCC simplifies the use of kprobes and kretprobes for performance analysis, abstracting the kernel's internal mechanics.

Use Cases in Performance Analysis

  • Identifying Bottlenecks: Measure the time spent in critical kernel functions (e.g., vfs_read, do_sys_open) to pinpoint slow paths.
  • Call Chain Analysis: Track how functions are invoked, revealing dependencies or unexpected interactions.
  • System Call Monitoring: Monitor the frequency and arguments of system calls to diagnose resource contention or misuse.
  • Latency Profiling: Use kretprobes to calculate the duration of function execution, helping optimize kernel modules or drivers.

Key takeaways

  • kprobes and kretprobes enable dynamic tracing of kernel functions for performance analysis.
  • BCC tools like trace and bpftrace abstract the complexity of kernel probe management.
  • kretprobes are particularly useful for measuring function execution time and latency.
  • These mechanisms are essential for diagnosing bottlenecks, analyzing call chains, and optimizing system behavior without kernel modifications.