eBPF for Application Developers: Debug Production Issues Without Restarting Anything

# eBPF for Application Developers: Debug Production Issues Without Restarting Anything

Here's a scenario every senior engineer has lived through: it's 2:17 AM, production is degraded, and your metrics dashboard is a wall of red. CPU is spiking, p99 latency has tripled, and you have no idea why. The code hasn't changed in three days. Logs show nothing unusual. Your on-call rotation partner suggests "let's just restart the pod and see if it goes away."

That's not debugging. That's superstition.

eBPF gives you a better option. It lets you attach safe, sandboxed programs to kernel events — syscalls, network packets, function entry/exit points — without modifying your application _at all_. No restart, no redeploy, no printf debugging in production. And in 2026, the tooling has matured enough that you don't need to be a kernel developer to use it.

This post covers four real-world scenarios where eBPF saved my team from the restart-and-pray approach, with the exact commands and programs we used. No vague "you could do this" — these are copy-paste-ready recipes.

---

What eBPF Actually Is (In One Paragraph)

eBPF is a technology that runs sandboxed programs inside the Linux kernel in response to events. Your program is compiled to eBPF bytecode, verified for safety (no infinite loops, no out-of-bounds memory access, bounded execution time), and then attached to a hook — a syscall, a kernel function, a network socket, a tracepoint, or a USDT probe in your application. When that event fires, your program runs. It can read kernel and userspace memory, compute statistics, and write data to ring buffers that userspace tools consume.

The key words are sandboxed and safe. You cannot crash the kernel with an eBPF program. The verifier rejects anything dangerous. This is what makes it viable for production use.

---

Prerequisites: What You Need

Before diving into the scenarios, here's what the tooling landscape looks like in mid-2026. You have three tiers:

For the scenarios below, we'll mostly use bpftrace — it's the sweet spot between power and complexity. One-liners that solve real problems without writing C.

Install it once on your bastion/debug host:

# Ubuntu/Debian
sudo apt install bpftrace

# Amazon Linux 2023 / Fedora
sudo dnf install bpftrace

# Verify kernel support
bpftrace -e 'BEGIN { printf("eBPF is ready\n"); exit(); }'

If that works, you're good. If it doesn't, your kernel is probably too old — you need 4.15+ for basic support, 5.8+ for BTF (which you want for the easier usage patterns).

---

Scenario 1: Find Which Process Is Saturating Disk I/O

The situation: A database server is crawling. iostat shows await times in the hundreds of milliseconds. But SHOW PROCESSLIST shows nothing unusual — the DB isn't the one hammering the disk. Something else is.

Before eBPF, you'd run iotop, hope it was installed, and squint at a real-time feed hoping to catch the culprit. With bpftrace, you get a histogram of I/O latency per process in one line:

sudo bpftrace -e 'kprobe:blk_account_io_done { 
    @latency[comm] = hist((nsecs - @start[tid]) / 1000); 
} 
kprobe:blk_account_io_start { 
    @start[tid] = nsecs; 
}'

This attaches to the block layer I/O start and completion functions and builds a latency histogram grouped by process name. Let it run for 30 seconds, then hit Ctrl+C. The output looks like:

@latency[logrotate]: 
[0]                 1023 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@          |
[1]                  523 |@@@@@@@@@@@@@                             |
[2, 4)                89 |@@                                         |
[4, 8)                12 |                                           |
[8, 16)                5 |                                           |
[16, 32)               2 |                                           |

@latency[postgres]: 
[0]                  812 |@@@@@@@@@@@@@@@@@@@@@@@@@                 |
[1]                 1234 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  |
[2, 4)               456 |@@@@@@@@@@@@@@@                           |
[4, 8)               234 |@@@@@@@@                                   |
[8, 16)              189 |@@@@@@                                     |
[16, 32)             134 |@@@@                                       |
[32, 64)              45 |@                                          |
[64, 128)             12 |                                           |
[128, 256)             4 |                                           |
[256, 512)             1 |                                           |

Turns out logrotate is compressing 50GB of logs mid-day because someone's cron schedule was misconfigured. Postgres is fine — it's just waiting on the same disk. You'd never catch this with application-level tooling.

One step deeper — which files?

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read {
    @bytes[comm, str(args->buf)] = count();
}'

This traces every read() syscall. The str(args->buf) won't actually give you the filename directly from the buffer pointer, but you can pair it with:

# Better approach: trace vfs_read for filenames
sudo bpftrace -e 'kprobe:vfs_read {
    @reads[comm, str(((struct file *)arg0)->f_path.dentry->d_name.name)] = count();
}'

---

Scenario 2: TLS Handshake Failures Are Silent Killers

The situation: Your gRPC services intermittently return "UNAVAILABLE" to each other. The connection pool metrics look fine. Network pings are fast. MTU is correct. But every few minutes, a batch of requests fails.

TLS handshake issues are notoriously hard to debug because:

• They happen before your application code runs
• Most frameworks log them at DEBUG level (if at all)
• Timeouts look identical to network failures in metrics

eBPF can trace the connect() and SSL/TLS handshake functions directly:

sudo bpftrace -e '
kprobe:tcp_v4_connect {
    @conn_start[tid] = nsecs;
    @conn_dst[tid] = args->daddr;
}
kretprobe:tcp_v4_connect {
    if (@conn_start[tid]) {
        $duration_ms = (nsecs - @conn_start[tid]) / 1000000;
        if ($duration_ms > 100) {
            printf("SLOW CONNECT: %s -> %d.%d.%d.%d took %d ms\n",
                comm,
                @conn_dst[tid] & 0xff,
                (@conn_dst[tid] >> 8) & 0xff,
                (@conn_dst[tid] >> 16) & 0xff,
                (@conn_dst[tid] >> 24) & 0xff,
                $duration_ms);
        }
        @conn_hist = hist($duration_ms);
        delete(@conn_start[tid]);
        delete(@conn_dst[tid]);
    }
}'

This tells you _which_ connections are slow to establish. In our case, we discovered that 10% of connections to a specific upstream took 2-5 _seconds_ to connect — not to respond, just to establish the TCP session. That pointed straight at a misconfigured firewall doing deep packet inspection on the TLS ClientHello.

Going further — TLS-specific tracing:

If your application links against OpenSSL (most do), you can trace the actual TLS handshake functions:

# On a system where your app links OpenSSL dynamically
sudo bpftrace -e '
uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.3:SSL_do_handshake {
    @tls_start[tid] = nsecs;
}
uretprobe:/usr/lib/x86_64-linux-gnu/libssl.so.3:SSL_do_handshake {
    if (@tls_start[tid]) {
        $ms = (nsecs - @tls_start[tid]) / 1000000;
        @tls_hist = hist($ms);
        if ($ms > 500) {
            printf("SLOW TLS HANDSHAKE in %s (pid %d): %d ms\n", comm, pid, $ms);
        }
        delete(@tls_start[tid]);
    }
}'

This revealed that our Go services (which use the standard library's crypto/tls, not OpenSSL) were fine, but a Python service using an older OpenSSL was negotiating TLS 1.2 with a slow cipher — adding 300ms to every connection. Switching to TLS 1.3 eliminated the delay entirely.

---

Scenario 3: Memory Leaks Without Restarting

The situation: A Node.js service grows from 200MB to 2GB over 24 hours. The heap profiler shows nothing unusual. GC is working. But RSS keeps climbing.

Classic symptom of native memory leaks — memory allocated through C++ addons or the runtime itself that the JS heap profiler can't see. Before eBPF, you'd enable malloc debugging (which slows everything by 10x) or just add a nightly restart cron job and move on with your life.

Instead, trace mmap and brk calls — these are how processes request memory from the kernel:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_mmap {
    @mmap_bytes[comm, pid] = sum(args->len);
}
tracepoint:syscalls:sys_enter_brk {
    @brk_bytes[comm, pid] = sum(args->brk);
}
interval:s:30 {
    print(@mmap_bytes);
    print(@brk_bytes);
    clear(@mmap_bytes);
    clear(@brk_bytes);
}'

This prints the total mmap and brk allocations per process every 30 seconds. If a process is leaking, you'll see a steady climb. But to find _where_ in the code, stack traces are what you need:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_mmap /comm == "node"/ {
    @allocations[ustack(perf)] = count();
}
END {
    print(@allocations);
}'

The ustack(perf) gives you a userspace stack trace at the point of allocation. The output points directly to the C++ addon doing the allocation:

@allocations[
    node::mmap_allocate(unsigned long)+12
    v8::internal::Heap::AllocateRaw(...)+0x45
    ...
]: 45231

In one memorable case, a native addon for image processing was allocating tile buffers and never freeing them because an error path skipped the free(). The JavaScript GC had no idea. eBPF caught it in 5 minutes — something that would have taken days of heap profiling and addon source diving.

---

Scenario 4: syscall-Level Profiling for "Why Is This Slow?"

The situation: A Go service handles 5,000 req/s with p50 latency of 3ms. Suddenly p50 jumps to 45ms. No code changes. No traffic spike. The application profiler shows most time spent in "external" — which is a black box.

Time to profile at the syscall boundary. Every I/O operation, every network read, every futex wait — it all goes through syscalls. If you can measure which syscalls are slow, you usually know what's broken.

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_* /comm == "my-go-service"/ {
    @start[tid] = nsecs;
    @syscall[tid] = args->__syscall_nr;
}
tracepoint:syscalls:sys_exit_* /comm == "my-go-service" && @start[tid]/ {
    $duration_us = (nsecs - @start[tid]) / 1000;
    @latency[@syscall[tid]] = hist($duration_us);
    delete(@start[tid]);
    delete(@syscall[tid]);
}
END {
    print(@latency);
}'

This gives you a latency histogram for every syscall your process makes. On a healthy system, read and write dominate. When things are broken, you see different patterns — futex latency spiking means lock contention, epoll_wait spiking means the event loop is starved, connect spiking means DNS or network issues.

In this particular case, futex was ballooning to 10-50ms latencies. Tracing deeper:

sudo bpftrace -e '
kprobe:futex_wait /comm == "my-go-service"/ {
    @futex_waiters[kstack] = count();
}
interval:s:10 {
    print(@futex_waiters);
}'

The kernel stack trace pointed directly at a sync.Mutex that was being held across an HTTP call to a slow downstream service. The fix was a one-line change: move the mutex unlock before the external call. Found in 15 minutes. Without eBPF, this would have been a week of "it's probably the database" dead ends.

---

The Tooling Decision Matrix

Not every problem requires custom bpftrace scripts. Sometimes the existing tools are enough. Here's my personal decision matrix:

---

Production Safety: What Can Go Wrong

eBPF programs are safe _by design_ — the verifier rejects anything dangerous before it runs. But there are still practical concerns:

1. Performance overhead

Tracing every syscall on a high-throughput service will add overhead. For a service handling 100k req/s, tracing all syscalls might add 5-15% CPU. The solution: use filters aggressively.

# Bad: traces every process on the system
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @[comm] = count(); }'

# Good: only traces your process, and only read() syscalls
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read /comm == "my-service"/ { @[pid] = count(); }'

2. Map memory limits

bpftrace allocates BPF maps for aggregations. Running for hours with high-cardinality keys (like per-tid) can exhaust map space. Set bounds:

# Limit to top 100 entries
sudo bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }
END { print(@, 100); clear(@); }'

3. Kernel version compatibility

CO-RE (Compile Once, Run Everywhere) solves most portability issues as of kernel 5.8+, but some tracepoints and kprobes vary between kernel versions and distributions. Always test your scripts on a staging host running the same kernel as production before running them during an incident.

4. Security considerations

bpftrace requires CAP_BPF or root. On a locked-down production host, you may need your security team to grant this capability to your debug user. Frame it as: "This lets us debug without restarting, which means lower downtime and faster incident resolution." Most security teams get it when you put it that way.

---

When NOT to Use eBPF

This is important. eBPF is powerful, but it's not always the right tool:

• If your APM already shows the answer, use your APM. eBPF is for the gaps.
• If you can reproduce locally, use a debugger. eBPF shines in production when local reproduction is impossible.
• If you've never used it before and it's 3 AM, don't experiment during an incident. Build muscle memory during peacetime.
• If the problem is in your application logic, use your language's profiler and debugger. eBPF is for the boundary between your code and the system.

---

What to Do Tomorrow

eBPF isn't something you learn during a Sev-1 at 2 AM. Here's the concrete plan:

1. Install bpftrace on a staging host today. Run the examples in this post against a test workload. Get comfortable with the syntax.

1. Create a runbook entry with 3-5 bpftrace one-liners for your most common failure modes. The templated versions above are ready to adapt.

1. Add eBPF to your observability stack properly. Tools like Pixie (now part of New Relic) and Cilium's Hubble run eBPF programs continuously and expose the data through dashboards — no ad-hoc scripts needed for routine monitoring.

1. Learn the bpftrace one-liner cheat sheet. Print these out. They'll save you hours:

# What files are being opened most?
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[str(args->filename)] = count(); }'

# What processes are spawning child processes?
sudo bpftrace -e 'tracepoint:sched:sched_process_exec { printf("%s (pid %d) exec'd %s\n", comm, pid, str(args->filename)); }'

# Per-process disk I/O size distribution
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @bytes[comm] = hist(args->bytes); }'

# TCP retransmit rate per destination
sudo bpftrace -e 'kprobe:tcp_retransmit_skb { @retrans[comm, kstack] = count(); }'

# What's calling fsync? (Often the hidden disk latency culprit)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_fsync { @fsync[comm, ustack] = count(); }'

eBPF won't solve every problem, but it answers the hardest question in production debugging: "What is the system actually doing right now?" — without changing anything, without restarting anything, and without guessing. In a world of increasingly complex distributed systems, that's not just useful. It's essential.