5 Key Lessons from GitHub's Use of eBPF for Safer Deployments

Imagine this: GitHub, the platform that hosts millions of repositories, runs its own code on github.com. This self-reliance means GitHub tests changes internally before releasing them to users. But there's a critical flaw—if github.com goes down, GitHub engineers can't access the source code needed to fix it. That's a classic circular dependency. To solve this, GitHub adopted eBPF (Extended Berkeley Packet Filter), a powerful Linux kernel technology, to monitor and block dangerous network calls during deployments. In this article, we'll explore five key lessons from GitHub's approach, diving into the types of circular dependencies they faced and how eBPF provides a robust safety net.

1. The Circular Dependency Dilemma

GitHub's worst nightmare is an outage that prevents access to its own source code. To mitigate this, the team maintains a mirror of the codebase for fixing forward and built assets for rollbacks. But that's only the beginning. Deployment scripts themselves can introduce new circular dependencies—for example, by downloading a binary from GitHub or calling an internal service that depends on GitHub. These hidden risks can turn a simple fix into a cascading failure. The core challenge is ensuring that deployment code never creates a dependency on the very platform it's trying to repair. Traditional solutions relied on manual reviews, but that's error-prone. eBPF offers a systematic way to enforce safety without human oversight.

5 Key Lessons from GitHub's Use of eBPF for Safer Deployments — Source: github.blog

2. Direct Dependencies in Deploy Scripts

Consider a MySQL outage that makes GitHub unable to serve release data. To resolve it, engineers need to run a deploy script on each MySQL node. If that script tries to pull the latest release of an open source tool from GitHub, it will fail because GitHub is down. That's a direct circular dependency—the fix requires GitHub, but GitHub is broken. Before eBPF, teams had to manually audit scripts for such patterns. With eBPF, GitHub can programmatically block any outbound HTTP requests to github.com during deployment, preventing the script from even attempting the download. The result: deployments that work even when the platform is compromised.

3. Hidden Dependencies at Runtime

Sometimes a dependency isn't obvious. A deploy script might use a servicing tool already on disk, but that tool checks GitHub for updates when it runs. If GitHub is down, the tool might hang or fail, even though the script doesn't directly call GitHub. This hidden dependency can derail an entire deployment. eBPF solves this by monitoring system calls at the kernel level. It can detect when any process spawned by the deployment attempts to contact specific domains or services, and either log or block those calls. This gives engineers a safety net without modifying the tools themselves.

4. Transient Dependencies Through Other Services

Deployment scripts often call other internal services via APIs. For example, a migrations service might be used during a MySQL fix. If that migrations service itself needs to fetch a binary from GitHub to run, the failure propagates back to the deploy script. This transient dependency chain is hard to predict and even harder to prevent manually. eBPF can inspect all network traffic generated by the deployment process, including calls to intermediate services. By setting eBPF filters to block known problematic endpoints, GitHub breaks the chain before it causes a failure.

5. How eBPF Provides a Robust Safety Net

GitHub's new host-based deployment system uses eBPF to selectively monitor and block network calls. eBPF runs sandboxed programs in the Linux kernel, allowing extreme performance and low overhead. In practice, GitHub writes small eBPF programs that attach to network socket operations. These programs check the destination IP or domain against a blacklist (e.g., github.com itself or internal services that depend on it). If a match is found, the call is rejected or logged. This mechanism catches all three types of circular dependencies—direct, hidden, and transient—without requiring changes to deployment scripts. The result is safer, faster deployments with fewer incidents.

GitHub's journey highlights the power of eBPF for infrastructure safety. By systematically blocking circular dependencies at the kernel level, they've turned a manual, error-prone process into an automated safety net. Whether you're managing a small cluster or a global platform, these lessons can help you design more resilient deployment systems. Start small: identify your own circular dependencies, and consider using eBPF to enforce boundaries. Your future self—and your users—will thank you.

Tags:

5 Key Lessons from GitHub's Use of eBPF for Safer Deployments

1. The Circular Dependency Dilemma

2. Direct Dependencies in Deploy Scripts

3. Hidden Dependencies at Runtime

4. Transient Dependencies Through Other Services

5. How eBPF Provides a Robust Safety Net

Related Articles

Recommended

Discover More