Designing Distributed Crawling Components

From a Single Worker to a Distributed Pipeline

In a simple design, a single "crawler" process might do everything: take a URL, fetch the page, parse the text, extract new links, and repeat.

Let's watch a brief segment that illustrates this initial, high-level concept.

This monolithic worker design is a good starting point, but as you've likely experienced in your backend work, such a design has significant drawbacks at scale:

• Fragility: If any step fails (e.g., parsing fails due to malformed HTML), the entire process for that URL might be lost, potentially wasting the successful (and expensive) fetch operation.
• Inefficient Scaling: Fetching a web page is I/O-bound (network-limited), while parsing HTML is CPU-bound. Lumping them together means you can't scale the resources for these distinct tasks independently. You might be over-provisioning CPU for an I/O-bound task, or vice-versa.

The standard solution is to break the process into a multi-stage pipeline, with each stage handled by a dedicated pool of workers.

As the article explains, we'll split our system into at least two main services:

1. Fetcher Service: A pool of workers responsible only for fetching raw HTML from the web.
2. Parser Service: A separate pool of workers that takes the raw HTML, extracts the required text, and discovers new URLs to be sent back to the frontier.

These services will be decoupled by message queues, which not only coordinate the work but are also central to building a fault-tolerant system.

Worker Coordination and Fault Tolerance

With thousands of workers operating in parallel, things will inevitably go wrong. Servers will crash, networks will fail, and websites will be unavailable. A robust design anticipates and handles these failures gracefully.

The core of our coordination and fault tolerance strategy will rely on the properties of our message queues. Let's explore exactly how this works.

Let's consolidate the key fault-tolerance mechanisms discussed in the video.

Mechanism 1: Decoupling with Queues

• Fetcher workers consume URLs from the URL Frontier Queue.
• Upon successfully fetching and storing the HTML in blob storage (like S3), they enqueue a message onto a Parsing Queue. This message contains a pointer to the S3 object.
• Parser workers consume messages from the Parsing Queue, retrieve the HTML from S3, parse it, and store the final text data.

Mechanism 2: Handling Worker Crashes

If a worker process crashes, we must ensure the URL it was processing isn't lost. The message queue guarantees this.

• Using AWS SQS: When a worker picks up a message, the message becomes invisible for a configured "visibility timeout." If the worker crashes without explicitly deleting the message, the timeout expires, and the message becomes visible again for another worker to process. If it succeeds, it deletes the message. This provides an "at-least-once" delivery guarantee.
• Using Apache Kafka: Messages are stored in a persistent, ordered log. Consumers (our workers) belong to a consumer group and track their progress by an "offset" (a pointer to the last message processed). If a worker crashes, another worker from the same group will resume processing from the last committed offset.

Mechanism 3: Handling Task Failures & Retries

A website being temporarily unavailable is a task failure, not a worker crash. We shouldn't immediately give up.

• Retry with Exponential Backoff: If a fetch fails, the worker instructs the queue to make the message available again after a delay. This delay should increase exponentially with each subsequent failure to avoid overwhelming a struggling server.
• Dead-Letter Queue (DLQ): After a set number of retries (e.g., 5), the message is considered a "poison pill." To prevent it from being retried indefinitely and blocking the system, it's moved to a DLQ. This is a separate queue where failed messages are stored for later analysis by engineers.

SQS provides built-in support for visibility timeouts, configurable retries with exponential backoff, and DLQs, making it a very common and effective choice for this type of architecture.

Scaling the Distributed Components

Now that we have a robust, fault-tolerant design, how do we ensure it's fast enough to crawl 10 billion pages in under 5 days? We need to scale our worker fleets.

Because we separated the pipeline, we can scale our Fetcher and Parser services independently based on their specific bottlenecks.

To summarize the scaling strategy:

• Scaling Fetchers (I/O-Bound): The bottleneck is network I/O and politeness delays. We can perform a rough calculation based on target page count, time limit, and the network capacity of a single machine to estimate the number of parallel fetcher instances needed. As the video showed, even a handful of powerful, network-optimized machines can provide immense crawling throughput.
• Scaling Parsers (CPU-Bound): The bottleneck is CPU. The number of parsers needed depends on the rate at which fetchers are producing HTML content. The best strategy here is autoscaling. We can monitor the depth of the Parsing Queue and automatically add or remove parser instances to keep the queue length within a healthy range. This is a classic use case for technologies like AWS Auto Scaling Groups or serverless functions (AWS Lambda).

Alternative: Stateful Stream Processing

While the SQS/stateless worker model is very common, it's worth knowing about alternative patterns. For a senior role, showing awareness of different trade-offs is valuable.

Another approach is to use a stateful stream processing framework like Apache Flink.

In this model:

• Kafka acts as the durable message log.
• Flink jobs act as the stateful workers. Flink can maintain state in memory (e.g., DNS caches, robots.txt rules, as mentioned in the video) for high performance.
• Fault tolerance is achieved through checkpointing. Flink periodically saves its state and the current Kafka offset to durable storage (like S3). If a worker crashes, a new one can be started, load the last checkpoint, and resume processing from the exact same point, providing strong consistency guarantees.

This is a more complex but powerful pattern, often used when processing logic requires significant internal state.

Conclusion

In this lesson, we designed the core engine of our web crawler—the distributed system of workers that fetch and process web pages at scale. We moved from a naive monolithic design to a robust, scalable, and fault-tolerant pipeline.

Key Takeaways:

• Pipelined Architecture: Breaking down the work into distinct stages (e.g., Fetching, Parsing) connected by message queues is fundamental. It enables independent scaling, separation of concerns, and better fault tolerance.
• Worker Coordination via Queues: Message queues (like SQS or Kafka) are the backbone of the system, decoupling services and buffering workload. The best practice is to pass pointers to data, not large payloads, in the messages.
• Comprehensive Fault Tolerance: A robust system handles both worker crashes and task failures.
  • Worker Crashes: Handled by queue mechanics like SQS's visibility timeouts or Kafka's consumer offsets.
  • Task Failures: Handled by a retry strategy (e.g., exponential backoff) and a Dead-Letter Queue (DLQ) to isolate permanently failing jobs.
• Resource-Based Scaling: Scale different components based on their specific bottlenecks—Fetchers are I/O-bound, while Parsers are CPU-bound and can be autoscaled.

Next Lesson Preview:

In both this lesson and the last, we've mentioned the need to rate-limit requests to be a "polite" web citizen. But how do you build a rate limiter that works correctly across a distributed fleet of workers? In our next lesson, we will do a deep dive and compare different rate limiting algorithms (token bucket, leaky bucket, sliding window) and their trade-offs before designing a distributed rate limiter service.