Detecting Layer 3/4 DDoS Attacks

Hello! Welcome back to our third module on architecting DDoS mitigation systems.

In the last lesson, we designed the global circulatory system for our DDoS protection service. We architected a traffic ingestion network using BGP Anycast and distributed scrubbing centers, ensuring we could absorb massive, geographically distributed attacks without any single point of failure.

Now that we have a way to get all this traffic to our scrubbing centers, we need to build the brain of the operation. Today's lesson will focus on exactly that, as we work to achieve this learning outcome: Design detection mechanisms for common Layer 3/4 DDoS attack vectors such as SYN and UDP floods.

We'll move from the network-level routing architecture to the packet-level analysis that happens inside a scrubbing center, designing the high-speed systems that distinguish legitimate users from a torrent of malicious traffic.

1. The Detection Challenge: Finding Needles in a Haystack Fire

At its core, DDoS detection is a real-time data analysis problem on a massive scale. A large volumetric attack can exceed millions of packets per second. Your detection system must be able to:

1. Operate at Line Rate: Make decisions as fast as packets arrive, without introducing significant latency.
2. Differentiate Good from Bad: Accurately identify attack patterns while minimizing the impact on legitimate traffic (i.e., reducing false positives).

To achieve this, system designers typically employ a multi-layered approach, moving from coarse, fast analysis to more granular, resource-intensive methods. We'll explore two primary philosophies:

• Aggregate Analysis: Looking at the overall traffic statistics (e.g., total packets per second). This is computationally cheap and great for spotting the onset of a huge volumetric flood.
• Per-Flow Analysis: Tracking metrics for individual source IPs or connections. This is more expensive but provides the precision needed to identify specific attackers and apply surgical mitigations.

A robust system will start with aggregate analysis and escalate to per-flow analysis as needed.

2. Detecting UDP Floods

Let's start with the simpler case: the UDP flood. Since UDP is a connectionless protocol, attackers can easily generate and send vast numbers of UDP packets from spoofed source IP addresses.

Detection Mechanism 1: Aggregate Thresholding

The most basic detection method is to monitor the total volume of UDP traffic.

As the presentation shows, the logic is straightforward:

1. A baseline for normal UDP traffic is established.
2. A threshold is set (e.g., 1 million packets/sec).
3. If the incoming UDP packet rate exceeds this threshold, an alarm is triggered.

This method is extremely fast but lacks precision. It tells you that you're under a volumetric UDP attack, but not who is attacking or what the specific characteristics of the attack are. It's a blunt instrument.

Detection Mechanism 2: High-Performance Packet Inspection

A more intelligent approach involves inspecting the packets themselves to find anomalies. Attack tools are often simple and generate packets with tell-tale signs: all packets have the same size, an empty payload, or unusual header values.

To perform this inspection at multi-gigabit speeds, we can't rely on traditional application-level code. We need to operate much closer to the hardware. This is where the Berkeley Packet Filter (BPF), and its modern incarnation eBPF, become critical design components.

BPF allows you to run a small, sandboxed program directly within the Linux kernel—or even better, directly on the network interface card (NIC) driver layer via the eXpress Data Path (XDP). This is the earliest possible point to inspect and drop a packet, making it incredibly efficient.

For our system design, this means we can create BPF programs that:

• Check for valid protocol formats (e.g., is this a well-formed DNS query?).
• Match against signatures of known attack tools.
• Filter based on packet length, payload contents, or other header fields.

By using eBPF/XDP, our scrubbing center can discard a huge percentage of malicious UDP traffic with minimal CPU overhead, preserving resources for more complex analysis.

3. Detecting SYN Floods with SYN Cookies

SYN floods are a classic L4 attack. They exploit the stateful nature of the TCP three-way handshake. By sending a flood of SYN packets from spoofed IPs and never completing the handshake, an attacker can fill the server's SYN backlog (its queue for half-open connections), preventing legitimate users from establishing new connections.

A simple detection method would be to monitor the size of this SYN backlog or the rate of incoming SYN packets. However, there is a much more elegant solution that combines detection and mitigation: SYN Cookies.

The SYN cookie mechanism is a brilliant piece of system design. Let's architect how it works inside our scrubbing appliance:

1. Trigger Condition: The SYN cookie mechanism is not always on. It's activated when the system detects that the SYN backlog is nearing capacity, which is our primary indicator of a potential SYN flood.

2. Stateless SYN-ACK Generation: When activated, for each incoming SYN, the server (our scrubbing appliance) does not create a state entry. Instead, it crafts a special SYN-ACK packet where the TCP sequence number is a "cookie" calculated as a cryptographic hash of connection details:

   $$\text{cookie}=f(\text{src\_ip},\text{src\_port},\text{dst\_ip},\text{dst\_port},\text{secret\_key})$$

   This cookie also encodes a timestamp and the client's Maximum Segment Size (MSS).

3. Validation:

   • If the SYN came from a spoofed IP, no ACK will ever be received. Since we didn't store any state, no resources were wasted.
   • If the SYN came from a legitimate client, it will respond with an ACK packet containing sequence number + 1. Our server receives this ACK, re-runs the calculation on the connection details, and validates that the ACK number matches the expected cookie value. If it matches, the server reconstructs the connection and allows it to proceed.

The Cloudflare talk (LINK, section from 15:19 to 19:49) further elaborates on the practical implementation in Linux, highlighting the importance of enabling net.ipv4.tcp_syncookies and net.ipv4.tcp_timestamps for this to work effectively with modern TCP options.

By using SYN cookies, our system becomes resilient to SYN floods by design. It stops wasting state on unverified connections, effectively filtering out the attack traffic.

4. Advanced Detection: Per-Flow Analysis with Sketches

The techniques above are excellent for specific, high-volume floods. But what about more subtle attacks, like a botnet where thousands of real machines each send a small amount of traffic? Aggregate thresholding won't catch this, and SYN cookies won't help if the bots are completing connections.

We need a way to count traffic on a per-source-IP basis. The naive approach—a hash table mapping every source IP to a counter—would require an enormous amount of memory. A more scalable solution is to use probabilistic data structures, often called "sketches".

Let's architect a detection system based on these principles:

• Heavy Hitter Detection: We can use a Count-Min Sketch to get an approximate packet count for every source IP. It works by hashing an IP address to several different arrays of counters and incrementing them. To query the count for an IP, you hash it again and take the minimum of the values from the counters it maps to. This gives a slightly overestimated but very memory-efficient count. Our system can use this to quickly identify IPs sending an unusually high volume of traffic.

• Superspreader Detection (Botnet Detection): As the OpenSketch paper describes, we can build a more sophisticated detector. A "superspreader" is a source IP that connects to a large number of unique destinations, a classic behavior of scanning worms or botnets. We can design this by creating a Count-Min sketch where each "counter" isn't a simple integer but is itself another, smaller sketch (like a bitmap) that tracks the number of unique destination IPs seen from that source. This allows our system to ask, "Which source IPs are exhibiting bot-like behavior?", and flag them for mitigation.

These sketch-based systems give our scrubbing center the ability to spot complex, distributed attack patterns that would be invisible to simple aggregate thresholding.

Conclusion

In this lesson, we designed the core detection mechanisms for a Layer 3/4 DDoS scrubbing center. We've moved from simply absorbing traffic to intelligently analyzing it at line rate.

Key Takeaways:

• Layered Detection: A robust detection architecture combines fast, aggregate analysis (like packet rate thresholds) with more precise, per-flow analysis.
• High-Performance Inspection for UDP: UDP floods are detected by simple rate-limiting and, more effectively, by using eBPF/XDP to perform deep packet inspection at the earliest point in the network stack.
• Elegant Mitigation for SYN: SYN floods are handled by the SYN cookie mechanism, a stateless approach that validates clients cryptographically before committing any server resources, effectively combining detection and mitigation.
• Scalable Per-Flow Analysis with Sketches: To detect distributed attacks from botnets, we use probabilistic data structures like the Count-Min sketch to identify heavy hitters and superspreaders without requiring massive amounts of memory.

Preview of the Next Lesson:

We have designed systems to detect attacks and, in the case of SYN cookies, to mitigate them in a specific way. But what about other attacks? And what if we need more than just rate-limiting or dropping traffic? In our next lesson, we will design a more powerful mitigation component: Design a SYN proxy and flow analysis system for mitigating Layer 3/4 attacks while preserving legitimate connections. This will allow our system to fully terminate and inspect TCP connections, giving us much deeper visibility and control.