Resilient Crawler State Management

Hello and welcome back.

In our last few lessons, we've built a concurrent web crawler that is fast, polite, and resilient. It can handle network timeouts gracefully and avoids getting trapped in infinite redirect loops. However, it has a critical weakness for any long-running task: its state—the queue of URLs to visit (the "frontier") and the set of already visited URLs—exists only in memory. If the process crashes or is stopped, all that progress is lost.

This lesson addresses that final piece of the puzzle, focusing on the learning outcome: Design a persistence layer for the crawler's state (frontier, visited set) to allow for resuming after a crash.

This challenge is fundamental in distributed systems. In your work on Bedrock's infrastructure, ensuring stateful services like Rudi can recover from failures without losing track of ongoing inference jobs is paramount. We'll apply the same principles of fault tolerance here, making our crawler production-ready.

1. What to Persist: Identifying the Crawler's State

To make our crawler resumable, we must first identify the essential pieces of information that define its current state. At any given moment, the crawler's progress is defined by:

1. The Frontier: The collection of URLs that have been discovered but not yet crawled. In our previous code, this was the asyncio.Queue.
2. The Visited Set: The collection of URLs that have already been crawled (or are currently in the queue) to prevent redundant work. In our code, this was a Python set.

Persisting these two data structures is the minimum requirement for successful crash recovery. Upon restart, a crawler can load this state and pick up exactly where it left off.

2. High-Level Design for a Fault-Tolerant Crawler

To design a persistence layer, we need to move beyond simple in-memory objects and integrate external storage. This naturally leads to breaking our monolithic crawler worker into a more robust, pipelined architecture.

Let's watch a segment from a system design interview that outlines this exact transformation.

This discussion introduces two core components for persistence: a persistent message queue for the frontier and a metadata database for the visited set and other URL-specific information.

3. Strategy 1: The Managed Services Approach (SQS + DynamoDB)

Given your background with AWS, one of the most natural and robust approaches is to use managed services.

• Frontier (URL Queue): Amazon SQS is a perfect fit. It's a durable, scalable, and fully managed queue. A key feature for us is the visibility timeout. When a worker retrieves a URL from SQS, the message isn't deleted but becomes invisible. If the worker processes the URL successfully, it explicitly deletes the message. If the worker crashes, the visibility timeout expires, and the message reappears in the queue, ready to be picked up by another worker. This gives us "at-least-once" delivery and solves the frontier persistence problem out of the box.
• Visited Set (URL Metadata): Amazon DynamoDB is an excellent choice for tracking visited URLs. You can create a table with the normalized_url as the primary key. Before crawling a URL, you query DynamoDB. To perform an atomic "check-and-add," you can use a ConditionExpression that ensures the item doesn't already exist.

This SQS + DynamoDB architecture is extremely resilient and scalable, offloading the complexity of persistence and concurrency control to managed AWS services.

4. Strategy 2: The In-Memory Data Store Approach (Redis)

Another powerful and common approach, especially for systems requiring very low latency, is to use a high-performance in-memory data store like Redis. While often used as a cache, Redis can persist its data to disk, making it a viable primary data store for certain use cases.

Crucially, Redis provides server-side data structures that map perfectly to our crawler's needs.

As you saw, Redis isn't just a simple key-value store. It gives us:

• Redis Lists: A direct replacement for our asyncio.Queue to manage the frontier.
• Redis Sets: A direct replacement for our Python set to manage visited URLs.

Operations on these structures are atomic, which is a major advantage when building concurrent systems.

5. Implementing Redis-Based Persistence

Let's refactor our crawler to use Redis. The core logic remains the same, but we'll replace the in-memory asyncio.Queue and set with calls to a Redis server.

To make this concrete, let's look at documentation from a real-world crawling library that implements this exact feature.

Inspired by this pattern, we can now write our own implementation. Instead of saving the entire state blob on every change, we'll leverage Redis's atomic operations on its List and Set data structures directly for a more granular and efficient approach.

Here is the refactored code using the redis.asyncio library.

import asyncio
import aiohttp
import redis.asyncio as redis
from aiohttp import ClientTimeout
from urllib.parse import urljoin, urlsplit
from bs4 import BeautifulSoup




# --- CONSTANTS ---
MAX_REDIRECTS = 10



# --- Redis Keys ---
FRONTIER_KEY = "crawler:frontier"
VISITED_KEY = "crawler:visited"




# normalize_url function remains the same
def normalize_url(current_page_url: str, link: str) -> str | None:



    # (Implementation from previous lesson)
    try:
        abs_link = urljoin(current_page_url, link)
        parts = urlsplit(abs_link)
        if not parts.scheme or not parts.netloc:
            return None
        return parts._replace(fragment="").geturl()
    except ValueError:
        return None

async def worker(
    name: str,
    session: aiohttp.ClientSession,
    redis_client: redis.Redis,
    domain: str,
    semaphore: asyncio.Semaphore,
):
    """A worker that fetches URLs from Redis, and adds new links back."""
    print(f"[{name}] Starting...")
    timeout_config = ClientTimeout(total=10, connect=5)

    while True:
        current_url = None
        try:



            # --- CHANGE 1: Fetch URL from Redis List (blocking pop) ---
            # We use a tuple in `brpop` to make it blocking. Timeout of 1s.
            # `brpop` returns a tuple: (key, value)
            _, url_data = await redis_client.brpop([FRONTIER_KEY], timeout=1)
            if not url_data:



                # Timed out, maybe the queue is empty. Let's loop and try again.
                continue




            # In a real system, you'd serialize/deserialize a struct (e.g., JSON)
            # For simplicity, we'll decode a string.
            current_url = url_data.decode('utf-8')
            



            # --- CHANGE 2: Atomically check and add to Redis Set ---
            # SADD returns 1 if the element was added, 0 if it already existed.
            if await redis_client.sadd(VISITED_KEY, current_url) == 0:



                # This URL was already visited or is being processed by another worker
                # that was faster.
                continue

            async with semaphore:
                print(f"[{name}] Crawling: {current_url}")

```grasp
{
  "type": "exercise",
  "id": "43fe749e-1a3d-4e10-82aa-64ae5bf5b2c8"
}

            # We'll omit the redirect logic for clarity, but it would be here.
            # It would involve putting the new URL back on the Redis frontier list.

            html = None
            try:
                async with session.get(current_url, timeout=timeout_config) as response:
                    if response.status != 200 or "text/html" not in response.headers.get("Content-Type", ""):
                        continue
                    html = await response.text()
            except (asyncio.TimeoutError, aiohttp.ClientError) as e:
                print(f"[{name}] Error crawling {current_url}: {e}")
                continue




        # --- CHANGE 3: Add new links to Redis List ---
        if html:
            soup = BeautifulSoup(html, "html.parser")
            new_links = []
            for link_tag in soup.find_all("a", href=True):
                normalized_link = normalize_url(current_url, link_tag["href"])
                if normalized_link and urlsplit(normalized_link).netloc == domain:



                    # Check if it's already in the visited set before adding to frontier
                    if not await redis_client.sismember(VISITED_KEY, normalized_link):
                       new_links.append(normalized_link)
            
            if new_links:



                # LPUSH adds all elements to the left of the list (our queue)
                await redis_client.lpush(FRONTIER_KEY, *new_links)

    except asyncio.CancelledError:
        print(f"[{name}] Stopping...")
        break
    except Exception as e:
        url_ref = current_url if current_url else "unknown URL"
        print(f"[{name}] UNEXPECTED Error for {url_ref}: {e}")

async def main():
start_url = "https://xkcd.com"
domain = urlsplit(start_url).netloc

redis_client = redis.from_url("redis://localhost", decode_responses=False)




# --- CHANGE 4: Seed the crawler only if it's a fresh start ---
# Check if the frontier AND visited sets are empty.
if not await redis_client.exists(FRONTIER_KEY) and not await redis_client.exists(VISITED_KEY):
    print("Starting a fresh crawl...")
    await redis_client.lpush(FRONTIER_KEY, start_url)
else:
    print("Resuming previous crawl...")

semaphore = asyncio.Semaphore(10)

async with aiohttp.ClientSession() as session:
    tasks = [
        asyncio.create_task(
            worker(f"Worker-{i}", session, redis_client, domain, semaphore)
        )
        for i in range(5)
    ]
    



    # In a real resumable system, you need a different shutdown signal.
    # For this example, we'll run for a fixed time.
    await asyncio.sleep(60) # Run for 60 seconds
    
    print("Shutting down workers...")
    for task in tasks:
        task.cancel()
    
    await asyncio.gather(*tasks, return_exceptions=True)

await redis_client.close()

if name == "main":

# Ensure you have a Redis server running locally for this to work
asyncio.run(main())

### 6. Discussion of Trade-offs

Choosing a persistence strategy is a classic engineering trade-off. Let's compare the two approaches.

| Feature | Redis | SQS + DynamoDB |
| :--- | :--- | :--- |
| **Performance** | **Extremely high**. In-memory operations offer sub-millisecond latency. | **Lower**. Network hops to multiple managed services introduce higher latency per operation. |
| **Data Structures** | **Ideal fit**. Native `LIST` and `SET` types map directly to the problem. | **Good fit**. SQS is a natural queue. DynamoDB works well for a set, but requires careful use of conditional writes. |
| **Operational Burden**| **Higher**. You must manage the Redis instance, including clustering for high availability, backups, and persistence configuration (AOF vs. RDB). | **Very low**. AWS manages scaling, durability, and availability. |
| **Atomicity/Recovery**| Requires careful implementation. A simple `RPOP` can lose a URL if a worker crashes. A more robust pattern (`BRPOPLPUSH`) is needed to move items to a "processing" list. | **Built-in**. SQS's visibility timeout handles worker crashes automatically, making the URL available again. This is a significant advantage. |
| **Cost** | Can be cheaper if self-hosted, but managed Redis can be expensive. | Pay-per-request model can be very cost-effective, especially for intermittent workloads. |

For a large-scale, enterprise system where operational robustness and high availability are paramount—like the services you build at AWS—the **SQS + DynamoDB** pattern is often superior. For projects where raw performance is the absolute priority and you have the operational capacity to manage the infrastructure, **Redis** is an excellent choice.


```grasp
{
  "type": "exercise",
  "id": "3539ccf4-0abf-4ddc-a44e-e79161c049b8"
}

Conclusion

In this lesson, we elevated our crawler from a transient script to a robust, fault-tolerant application by designing a persistence layer. This is a non-negotiable feature for any serious, long-running data processing task.

Key Takeaways:

• The essential state of a crawler consists of its frontier (URLs to visit) and visited set.
• This state can be persisted using various technologies, with two common patterns being a managed queue/DB stack (like SQS + DynamoDB) or a high-performance in-memory store (like Redis).
• Redis offers excellent performance and data structures that are a perfect fit for the problem (LIST for the frontier, SET for visited URLs).
• SQS + DynamoDB offers superior operational simplicity and built-in fault tolerance mechanisms (like visibility timeouts), which is often preferred in large-scale cloud environments.
• The choice between these strategies involves critical trade-offs between performance, operational complexity, and cost.

Next Lesson Preview

We have now completed our deep dive into building a production-quality web crawler. You've seen how to handle concurrency, rate-limiting, error handling, and persistence.

Now, we will pivot from the infrastructure of data acquisition to the systems that use this data. Our next module, System Design: LLM Inference Service, takes us directly into the core of a company like Anthropic. In the first lesson, we'll start with the most fundamental building block of any Large Language Model: the token. We will explore how tokenization works and why it is the basic unit of workload for LLM serving systems.