A Practical Guide to Using eBPF for Safer Deployments: Lessons from GitHub

Introduction

GitHub itself runs on github.com, creating a classic circular dependency: if the site goes down, engineers can't access the code they need to fix it. While a mirror helps, deployment scripts can still introduce subtle circular dependencies—for example, a script that downloads a tool from GitHub during an outage. To break this cycle, GitHub turned to eBPF (extended Berkeley Packet Filter). This guide shows you how to apply the same technique: monitor and block deployment scripts from making unsafe network or system calls that could create circular dependencies.

A Practical Guide to Using eBPF for Safer Deployments: Lessons from GitHub — Source: github.blog

What You Need

A Linux system with kernel 4.4+ (for eBPF support)
BCC (BPF Compiler Collection) or bpftrace installed
Basic familiarity with C or Python (for custom eBPF programs)
Access to your deployment scripts and host machines
A staging environment to test safely

Step-by-Step Guide

Step 1: Identify Circular Dependencies in Your Deployment Scripts

Before you can block dangerous calls, you need to know what to look for. Circular dependencies fall into three categories:

Direct dependencies: The script downloads a binary or configuration from a service that may be down (e.g., GitHub).
Hidden dependencies: An existing tool on the machine checks for updates every time it runs—even if you didn't ask it to.
Transient dependencies: The script calls an internal API that, in turn, fetches something from an external source.

Review each step in your deployment scripts and note every external network call or system file read. Pay special attention to package managers, update checkers, and API calls.

Step 2: Set Up eBPF Monitoring to Trace Calls During Deployments

Use existing eBPF tools to capture what your scripts actually do. For example:

# Trace network connections
sudo opensnoop -p $(pgrep -f deploy-script)
# Trace executed commands
sudo execsnoop -p $(pgrep -f deploy-script)

Run these in parallel with a test deployment in staging. Collect a list of all connections (IP addresses and ports) and executables launched. This becomes your baseline of potential circular dependencies.

Step 3: Define Policies to Block Unsafe Connections

Based on your monitoring data, decide which calls are unsafe during an outage. For example:

Block all outbound HTTP/HTTPS to github.com or api.github.com.
Block access to internal services that themselves have external dependencies.
Block file reads of cached update-check scripts that trigger network calls.

Write these policies as simple IP/domain allow lists or deny lists. eBPF lets you enforce them at the kernel level, so even a well-hidden call can’t escape.

Step 4: Write a Custom eBPF Program to Intercept and Filter Calls

With BCC, you can create a small C program that runs inside the kernel. Here’s a skeleton that attaches to the connect syscall and blocks connections to a specific IP:

from bcc import BPF

bpf_text = """
#include <uapi/linux/ip.h>
#include <net/sock.h>

int block_connect(struct pt_regs *ctx, struct sockaddr_in *addr)
{
    if (addr->sin_addr.s_addr == blocked_ip) {
        return -EPERM;  // block
    }
    return 0;
}
"""

blocked_ip = 0x0a000001  # 10.0.0.1 for example
b = BPF(text=bpf_text)
b.attach_kprobe(event="tcp_v4_connect", fn_name="block_connect")
# alternative: use cgroup/connect4 for container-level filtering

Test this program in isolation—first log, then gradually enforce denials.

Step 5: Test the eBPF Program in a Staging Environment

Simulate an outage scenario (e.g., add a firewall rule to block GitHub temporarily). Run your deployment script with the eBPF program active. Verify that:

The script completes without accessing blocked resources.
All legitimate local operations (file reads, internal APIs on allowed IPs) proceed normally.
Error messages are clear so engineers know what was blocked.

If you detect false positives, refine your policy. Repeat until the script works perfectly under the simulated outage.

Step 6: Integrate eBPF into Your Deployment Pipeline

Once stable, incorporate the eBPF programs into your deployment process:

Start the eBPF program at the beginning of the deployment script (using a wrapper or systemd unit).
Ensure it only applies to the current deployment session (e.g., using cgroup-based filtering).
Add fallback logic: if the eBPF program fails to load, warn but proceed (or abort based on risk tolerance).
Log all blocked calls for post-deployment analysis.

Tips and Conclusion

Start small: monitor before you block. Use eBPF to log all network connections for a week to identify unexpected dependencies. Then progressively enforce blocks. Remember that even hidden dependencies—like a tool checking for updates—can cause failures during an incident.

GitHub’s approach shows that eBPF provides a low-overhead, fine-grained way to enforce deployment safety without modifying application code. By following these steps, you can break circular dependencies and make your own deployments more resilient.

Tags: