SYN Proxy and Flow Analysis for DDoS Mitigation

Hello! Welcome back to our course on designing security products.

In our previous lesson, we designed the initial detection layer of our DDoS scrubbing center. We focused on high-speed, often stateless, mechanisms like eBPF/XDP for UDP floods and SYN cookies for TCP SYN floods. These are excellent for identifying and dropping large volumes of malformed or illegitimate traffic.

However, SYN cookies, being stateless, offer limited visibility. After the initial handshake, the connection bypasses our defense logic. What if we need to maintain control over the entire connection to perform deeper analysis or apply more nuanced policies?

That brings us to today's lesson, where we'll architect a more powerful, stateful mitigation system. We'll be tackling the learning outcome: Design a SYN proxy and flow analysis system for mitigating Layer 3/4 attacks while preserving legitimate connections. We will explore how to build a system that can fully terminate and inspect TCP connections, giving us the visibility needed for advanced flow analysis and a foundation for future Layer 7 defenses.

1. Beyond Stateless: The Case for a SYN Proxy

In the last lesson, we saw how SYN cookies ingeniously avoid storing state on the server for unverified connections. The server sends a cryptographically signed SYN-ACK and essentially says, "Prove you're legitimate by solving this puzzle, then I'll create a connection for you."

A SYN proxy takes a different, more hands-on approach. It's a full-fledged TCP middlebox that acts as an intermediary, terminating the client's TCP connection and, if the client is deemed legitimate, initiating a separate TCP connection to the backend server.

Why accept this added complexity?

• Deep Flow Analysis: By terminating the connection, the proxy can reassemble the entire TCP stream. This allows it to analyze the flow for protocol anomalies, track connection rates, and enforce stateful firewall rules that are impossible to implement with a stateless approach.
• Origin Protection: The proxy completely hides the backend server from the public internet. The server's TCP/IP stack characteristics, open ports, and IP address are never exposed. Attackers can only interact with the hardened proxy.
• Foundation for L7 Inspection: A SYN proxy is the foundational L4 component upon which Layer 7 security products, like a Web Application Firewall (WAF), are built. You can't inspect HTTP traffic if you don't first terminate the TCP connection it's carried over.

2. Architecting a Classic SYN Proxy

Let's design the traditional SYN proxy. It sits between the client and the server and manages two distinct TCP connections.

To understand the mechanics and inherent challenges, please review the following resource.

As the paper illustrates, the core operation involves "stitching" two connections together. This process creates two significant engineering challenges:

Challenge 1: Sequence Number Translation

The client-to-proxy connection and the proxy-to-server connection are independent. This means they will have different, randomly chosen Initial Sequence Numbers (ISNs).

• Client-Proxy Connection: Client ISN = $IS{N}_C$, Proxy ISN = $IS{N}_P$
• Proxy-Server Connection: Proxy ISN = $IS{N}_{P^{\prime }}$, Server ISN = $IS{N}_S$

When the server sends data to the client, the proxy must intercept the packet, which has sequence numbers relative to $IS{N}_S$, and rewrite them to be relative to $IS{N}_P$. The same translation must happen in the other direction for acknowledgment numbers.

The proxy must therefore store a sequence number delta for every active connection:

$${\Delta }_{C\to S}=IS{N}_S-IS{N}_P$$ $${\Delta }_{S\to C}=IS{N}_{P^{\prime }}-IS{N}_C$$

For every single packet traversing the proxy, it must look up the connection's state, fetch the correct delta, and perform addition/subtraction on the SEQ and ACK fields in the TCP header. This is a computationally intensive, per-packet operation that must be maintained for the life of the connection.

Challenge 2: Initial Data Buffering and Latency

There's a race condition during setup. A client might complete its handshake with the proxy and immediately send a packet with data (e.g., an HTTP GET request). However, the proxy's own handshake with the backend server might not be complete yet.

What can the proxy do?

1. Drop the packet: This forces the client to retransmit after a timeout (e.g., 200ms), introducing significant latency.
2. Buffer the packet: The proxy can store the data packet in memory. Once the proxy-server connection is established, it can perform the sequence number translation and forward the buffered data. This avoids the retransmission delay but requires the proxy to manage memory buffers for potentially thousands of new connections, adding state and complexity.

These two challenges—constant sequence number translation and initial data buffering—are the primary drawbacks of the classic SYN proxy design, impacting both performance and scalability.

3. A Modern Approach: The Split-Proxy Architecture

Given your background in system design, you'll appreciate that the classic proxy's limitations are ripe for architectural innovation. The "SMARTCOOKIE" paper presents a brilliant solution: a split-proxy architecture that divides the labor between a high-speed, but simple, network device and a more flexible software component.

Let's break down this modern design.

This design elegantly solves the two main problems of the classic proxy:

• Solving the Buffering Problem: The high-speed "switch agent" doesn't buffer anything. When it receives the final ACK from the client (packet 3 in Figure 4), it doesn't wait. It immediately forwards that ACK to the "server agent" with a special tag (packet 4). This setup packet acts as a notification to the server agent to initiate the backend connection. The switch's job is done in microseconds, and no buffering is needed.

• Offloading Sequence Number Translation: The stateful, complex task of sequence number translation is moved away from the high-speed switch agent. It's now handled by the server agent, which is implemented as an eBPF program running on the server itself. Since the server has far more memory and compute flexibility than a network switch, it's the perfect place for this task. The eBPF program intercepts packets before they hit the main kernel network stack, applies the sequence number delta, and then passes them on. This is a fantastic example of architectural offloading.

The result is a system that gets the best of both worlds: a fast, stateless-acting frontend that can handle massive traffic volumes, and an intelligent, stateful backend that performs the complex connection stitching transparently.

4. Flow Analysis with a Split-Proxy

With this architecture, how do we perform the "flow analysis" mentioned in the learning outcome?

1. At the Switch Agent (Proxy): The switch agent is the first line of defense. It can perform basic flow analysis:

   • Connection Rate Limiting: It can track the rate of new SYNs per source IP and block sources that exceed a threshold.
   • Stateful Packet Filtering: It knows which connections are legitimate (even if approximately, as we'll see). It can drop any non-SYN packets that don't belong to an established flow, mitigating ACK floods or other junk traffic.

2. At the Server Agent (eBPF): The server agent maintains the exact state for every connection. It can perform more detailed analysis:

   • TCP State Machine Enforcement: The eBPF program can track the TCP state for each flow and drop packets that are invalid for the current state (e.g., receiving data after a FIN).
   • Connection Counting: It maintains an exact count of active connections per source IP, which can be used for more precise rate-limiting policies.

Connecting to the Previous Lesson: Approximate State

You might wonder: how does the fast switch agent know which packets belong to an "established flow" without keeping a massive state table? This is where the concepts from our last lesson come back into play.

By using a Bloom filter, the switch agent can check if a flow has been verified with very high memory efficiency. This has one important consequence: false positives. A small fraction of attack packets might match the Bloom filter by chance and be forwarded to the server agent.

The system handles this gracefully. As described in Section 6.4 of the paper, the server agent acts as the final verifier. If it receives a packet that it doesn't have state for (a false positive from the switch), it performs its own final cookie check before dropping it or creating a new connection. This multi-layered defense makes the system robust.

Conclusion

Today we've designed a sophisticated, stateful SYN proxy system, moving from a classic design to a modern, high-performance architecture. You can now articulate the core trade-offs between stateless and stateful L4 mitigation and design a system that balances performance, scalability, and security.

Key Takeaways:

• A SYN proxy is a stateful L4 middlebox that terminates TCP connections, enabling deep flow analysis and origin protection at the cost of complexity.
• The classic SYN proxy architecture suffers from latency and state overhead due to initial data buffering and per-packet sequence number translation.
• A modern split-proxy design solves these issues by offloading the complex state management (like sequence number translation) to a flexible server-side component (e.g., an eBPF program), leaving the high-speed proxy to handle attack blocking.
• Flow analysis in such a system is layered: the proxy performs high-level checks (e.g., connection rates), while the server agent can enforce fine-grained TCP state machine rules.
• Probabilistic data structures, like Bloom filters, can be used to make the proxy's state tracking more memory-efficient and scalable.

Preview of the Next Lesson:

We have now built robust defenses for volumetric attacks at Layers 3 and 4. Our system can handle UDP floods, SYN floods, and other TCP-based attacks. But what happens when the attacker uses a perfectly valid TCP connection and sends a flood of application-layer requests, like thousands of HTTP GETs per second? Our current system would see this as legitimate traffic.

In our next lesson, we will move up the stack to address this threat: Design Layer 7 DDoS detection using behavioral baselining and adaptive rate limiting to distinguish floods from legitimate traffic.