Enhancing Deployment Safety at GitHub with eBPF: A Deep Dive into Circular Dependency Prevention

Introduction

At GitHub, we host our own source code on github.com, making us our own biggest customer. This practice allows us to test changes internally before releasing them to users. However, it creates a critical circular dependency: if github.com goes down, we lose access to our own source code, making it impossible to deploy fixes. While we maintain a mirrored repository and built assets for rollback, other circular dependencies persist, especially during deployments. This is where eBPF (extended Berkeley Packet Filter) comes into play, enabling us to monitor and block problematic calls in deployment scripts. In this article, we’ll explore the types of circular dependencies we encountered and how eBPF helps ensure deployment safety.

Types of Circular Dependencies

Consider a scenario where a MySQL outage causes GitHub to fail serving release data, requiring a configuration roll out via a deploy script on each affected node. Let’s break down the circular dependencies that could arise:

Direct Dependency

The MySQL deploy script attempts to pull the latest release of an open source tool from GitHub. Because GitHub cannot serve release data during the outage, the script fails to complete. This is a straightforward circular dependency—the deployment process depends on the very service it is meant to fix.

Hidden Dependency

The deploy script uses a servicing tool already present on the machine’s disk. However, when the tool runs, it checks GitHub for an update. Due to the outage, the check fails or hangs, depending on error handling. This hidden dependency is not obvious from the script itself but can cause deployment failures.

Transient Dependency

The deploy script calls an internal API (e.g., a migrations service), which in turn tries to fetch the latest binary from GitHub. The failure propagates back to the deploy script, creating a chain of dependencies that ultimately loops back to GitHub. Transient dependencies are especially tricky because they involve intermediate services.

Previous Solutions and Their Limitations

Before adopting eBPF, the responsibility fell on each team owning stateful hosts to manually review their deployment scripts and identify circular dependencies. This process was error-prone and time-consuming. Many dependencies remained undetected until an outage occurred, leading to failed deployments and extended incident resolution times. Clearly, a more systematic approach was needed.

How eBPF Solves Circular Dependencies

eBPF provides a powerful mechanism to safely and efficiently monitor and influence system behavior at the kernel level. In our new host-based deployment system, we use eBPF to intercept specific system calls made by deployment scripts and their children. By attaching eBPF programs to syscalls like connect, open, and execve, we can:

• Monitor all outbound network connections and file accesses during deployment.
• Block any calls that would create a circular dependency (e.g., trying to reach github.com or internal services that depend on GitHub).
• Log blocked attempts for auditing and debugging.

For example, when a deploy script tries to curl a binary from GitHub, the eBPF program inspects the destination IP or domain. If it matches our defined blacklist (e.g., github.com domains or internal service endpoints known to create circular dependencies), the syscall is denied, and an error is returned to the script. The script can then fail gracefully or fall back to a local cache.

Implementation Details

Our eBPF programs are written in C and compiled into BPF bytecode. They are loaded into the kernel using bpf() syscalls or through a library like libbpf. We attach them to tracepoints or kprobes associated with networking and file I/O syscalls. For security, we ensure that only privileged processes (our deployment agent) can load and manage these programs. The agent also handles the generation of eBPF programs specific to each deployment context—e.g., which dependencies to block based on the service being deployed.

One key challenge was ensuring that eBPF programs do not introduce performance overhead. We optimized by using maps to pre-check allowed destinations and by limiting the scope to only deployment-related processes. The overhead measured is negligible (<1% CPU) under typical deployment loads.

Benefits and Conclusion

With eBPF, GitHub now has a runtime safety net that catches circular dependencies automatically, without relying on manual script reviews. This has reduced deployment failures during outages and accelerated incident response. The same eBPF infrastructure can be extended to other use cases, such as security sandboxing or network policy enforcement.

In summary, eBPF provides a safe, efficient, and flexible way to prevent deployment scripts from creating circular dependencies. By embedding these checks at the kernel level, we ensure that even if scripts evolve or are written by different teams, deployment safety remains intact. If you’re facing similar challenges, we encourage you to explore eBPF—it might be the missing piece in your reliability toolkit.

For more information on eBPF programming, see our introductory guide.