Fail-Open vs. Fail-Close Security Strategies

Hello! Welcome back to our course on designing security products.

In our last lesson, we explored the trade-offs between agent-based and agentless architectures. We concluded that agent-based systems are often necessary for inline prevention, as they can directly intervene in a system's operations. This, however, introduces a critical architectural challenge: the security product itself becomes a single point of failure.

This brings us directly to today's topic. When an inline security product fails, what should happen? Should it allow traffic to pass uninspected, or should it block everything? This is a fundamental decision that pits security against availability.

This lesson addresses the learning outcome: Evaluate fail-open vs. fail-close strategies for inline security products and their availability implications. We will analyze this classic trade-off from a first-principles perspective, explore the mechanisms used to implement these strategies, and examine how to design systems that are both secure and resilient.

1. The Inline Security Dilemma

Inline security products, like a Web Application Firewall (WAF) or an Intrusion Prevention System (IPS), act as gatekeepers. They sit directly in the path of network traffic to inspect and, if necessary, block malicious requests before they reach their target.

While powerful, this inline position introduces significant risk. A bug, a crash, or a simple power failure in the security appliance can bring down the application it's supposed to protect.

To see this in action, let's consider the difference between detection and prevention.

As the video highlights, if an inline system makes a mistake or fails, it can disrupt business operations. This forces us, as architects, to make an explicit choice about the system's default failure state.

2. The Core Strategies: Fail-Open vs. Fail-Close

When an inline security tool fails, we have two primary strategies:

• Fail-Close (or Fail-Shut): The system defaults to a secure state. If the security tool is not operational, it blocks all traffic. This prioritizes security over availability. No traffic gets through, but you are certain that no uninspected, potentially malicious traffic reaches the application.
• Fail-Open: The system defaults to an available state. If the security tool is not operational, it allows all traffic to pass through uninspected. This prioritizes availability over security. The application stays online, but it is temporarily left unprotected.

To get a formal understanding of these concepts and their use cases, please read the following brief chapters from the e-book "The ABCs of Inline Security."

The choice between these two modes is not just theoretical; it has profound, real-world consequences.

3. A Real-World Case Study: Cloudflare's "Fail-Closed" Design

On June 12, 2025, Cloudflare experienced a significant outage. The root cause was the failure of a critical dependency, Workers KV, a key-value store used by many of their services. This incident provides a powerful, large-scale example of a "fail-close" philosophy in action.

In this case, Cloudflare's Zero Trust services could not retrieve the user identity and policy information needed to make an authorization decision. Instead of defaulting to "allow," which would have granted users access without security checks, the system failed closed. Users were blocked from logging in.

This decision prioritizes security at the cost of availability. For a critical security function like authentication, this is almost always the correct choice. The alternative—failing open—would have meant completely bypassing the security policies the customer had put in place, rendering the product ineffective and potentially leading to a breach.

4. Architectural Implementation

So, how do you actually build a system that can fail-open or fail-close? You don't want the security appliance itself to be responsible for this logic, as it's the very component that might be failing.

The most common implementation is an external device called a bypass switch (also known as a network bypass tap).

Let's return to "The ABCs of Inline Security" to understand this mechanism.

As you read, the core architectural pattern is:

1. A bypass switch is placed in the network path, in front of the inline security appliance (e.g., a WAF or IPS).
2. The bypass switch continuously sends tiny "heartbeat" packets to the security appliance and waits for a response.
3. If the appliance stops returning heartbeats, the switch declares it "dead."
4. The switch then executes its pre-configured failover logic:
   • Fail-Close: The switch shuts the network link, blocking all traffic.
   • Fail-Open (or Fail-Safe): The switch physically reroutes traffic around the dead appliance, allowing it to continue to the application uninspected.

5. Mitigating the Availability Impact

A fail-close strategy seems brutal—a single appliance failure causes a complete outage. As a system designer, your job is to get the security benefit of failing closed without paying the full price in availability. This is achieved through redundancy and failover.

Instead of simply blocking traffic, a smart bypass switch or load balancer can redirect it to a backup security appliance.

The key architectural patterns for mitigating the impact of a fail-close design are:

• Active-Standby (Failover): A primary appliance handles all traffic. A secondary, identical appliance is idle but ready. Upon failure of the primary, the bypass switch redirects all traffic to the standby appliance. This minimizes downtime from minutes/hours to seconds.
• Active-Active (High Availability): Two or more appliances actively process traffic simultaneously, sharing the load. If one fails, the remaining appliance(s) instantly absorb its share of the traffic. This provides the fastest failover (sub-second) and better resource utilization during normal operation, but is more complex to configure.

By designing a fail-close system with an active-standby or active-active failover strategy, you achieve the best of both worlds: you ensure no traffic ever bypasses security inspection, and you minimize downtime in the event of a component failure.

Conclusion

Today we explored a fundamental trade-off in the design of any inline security product. Your decision to fail-open or fail-close defines the product's core priorities when faced with an unexpected failure.

Key Takeaways:

• Fail-Close prioritizes security by blocking traffic when a tool fails. This is the standard for critical functions like authentication and access control.
• Fail-Open prioritizes availability by allowing traffic to bypass a failed tool. This may be acceptable for non-critical monitoring tools or during initial testing phases.
• The choice is a business and risk decision. The architect's job is to implement the chosen strategy reliably.
• Bypass switches are a common hardware mechanism for implementing this logic, using heartbeat packets to detect appliance failures.
• The availability impact of a fail-close design is mitigated with high-availability architectures, such as active-standby or active-active failover, which redirect traffic to redundant appliances.

Preview of the Next Lesson

We have now discussed how to handle failures in inline prevention tools that form the control path of a security architecture. Next, we will turn our attention to the reliability of the data path.

Security products generate and consume vast amounts of telemetry—logs, events, and metrics. How do you design a system to reliably collect and transport this data from thousands of sources without loss, even under heavy load or during network disruptions? In our next lesson, we will begin to answer this by looking at how to design the collection and transport layer of a security telemetry pipeline, addressing backpressure, buffering, and delivery guarantees.