HDFS Architecture and Fault Tolerance

1. The HDFS Master-Slave Architecture

HDFS employs a straightforward master-slave architecture. This design consists of two primary types of components, or daemons, that run on the nodes in the cluster:

• NameNode (The Master): There is typically only one NameNode in a cluster. It is the brain of the file system, responsible for orchestrating all operations. It holds all the filesystem metadata.
• DataNodes (The Slaves): There are many DataNodes in a cluster, usually one per machine. These are the workhorses that store the actual application data.

This architectural separation is key: the NameNode manages the directory tree and knows where data is located, while the DataNodes store the data itself.

To begin, let's watch a short video that introduces these core components and their primary functions.

2. Storing Data: Files, Blocks, and Replicas

So, how does HDFS store a massive, terabyte-sized file on DataNodes that only have a few terabytes of disk space each? It breaks the file down into smaller, manageable pieces called blocks.

An HDFS block is a large, fixed-size sequence of bytes. A typical block size in modern Hadoop is 128 MB. Any file stored in HDFS is split into these 128 MB chunks. For example, a 300 MB file would be stored as three blocks: two 128 MB blocks and one 44 MB block.

The key to HDFS's reliability is replication. HDFS creates multiple copies (replicas) of each block and distributes them across different DataNodes. The default replication factor is 3, meaning for every block of data, there are two extra copies. If a DataNode fails, the data is not lost because its replicas exist on other nodes.

This design choice of using large blocks is deliberate. It helps to minimize the amount of metadata the NameNode has to manage. Instead of tracking billions of small 4 KB blocks (as in a traditional filesystem), it only needs to track millions of large 128 MB blocks, which is far more efficient for RAM usage and lookup times.

3. The NameNode's Brain: Managing Metadata

As we've established, the NameNode holds all the filesystem metadata. This includes:

1. The Namespace: The complete directory and file hierarchy (e.g., /users/data/input.txt).
2. Block Mapping: For each file, a list of the blocks that comprise it and, crucially, the list of DataNodes where each block's replicas are stored.

To serve file lookup requests quickly, the NameNode keeps this entire metadata structure in its main memory (RAM). This is a critical performance feature, but it also presents a challenge: what happens if the NameNode machine reboots or crashes? The RAM is volatile and all metadata would be lost.

To solve this, the NameNode uses two persistent files stored on its local disk:

• FSImage: This is a complete snapshot of the filesystem namespace at a specific point in time. It's like a full backup.
• EditLog: This is a transaction log that records every change made to the namespace since the last FSImage was created (e.g., file creation, deletion, or rename). It's a write-ahead log.

When the NameNode starts up, it loads the most recent FSImage into memory and then applies all the transactions from the EditLog to bring the in-memory state fully up-to-date. This process ensures the metadata is durable. A helper process, the Secondary NameNode, periodically merges the EditLog into the FSImage to create a new, updated snapshot, a process called checkpointing.

4. Intelligent Replication for Robust Fault Tolerance

Simply having three copies of data isn't enough. If all three replicas of a block were on DataNodes plugged into the same network switch (a "rack"), a single switch failure or power outage to that rack would make the data unavailable.

To prevent this, HDFS uses a rack-aware replication policy. The NameNode tries to place replicas strategically across the cluster's physical topology. The default policy for a replication factor of 3 is:

1. Replica 1: Placed on a DataNode on the same rack as the client writing the data (if the client is inside the cluster). This minimizes network latency for the first write.
2. Replica 2: Placed on a DataNode on a different rack. This is the crucial step for rack-level fault tolerance.
3. Replica 3: Placed on a different DataNode but on the same rack as the second replica. This provides redundancy without the performance cost of writing to a third, separate rack.

This policy provides a balance between fault tolerance and performance. The system can survive the failure of any single node, and even the failure of an entire rack, without losing any data.

The NameNode continuously monitors the health of DataNodes through periodic signals called heartbeats. If a DataNode stops sending heartbeats, the NameNode declares it "dead." It then consults its metadata to see which blocks were on that failed node. It identifies these blocks as being under-replicated and schedules new replicas to be created on other live DataNodes, automatically restoring the desired replication factor.

Conclusion

In this lesson, we dissected the architecture of HDFS, the storage foundation of the Hadoop ecosystem. You now have a solid understanding of how HDFS provides scalable and reliable storage for massive datasets.

Key Takeaways:

• Master-Slave Architecture: HDFS is coordinated by a central NameNode (master) that manages metadata and multiple DataNodes (slaves) that store the actual data.
• Data Blocks: Files are broken into large, fixed-size blocks (e.g., 128 MB), which are the basic units of storage and replication.
• NameNode Metadata: The NameNode keeps the entire filesystem namespace and block locations in RAM for fast access, using a persistent FSImage and EditLog for durability.
• Fault Tolerance: HDFS achieves high reliability through block replication. A rack-aware placement policy ensures that replicas are distributed across different physical racks to protect against both node and rack failures.
• Self-Healing: The NameNode detects DataNode failures via missed heartbeats and automatically re-replicates the blocks that were on the failed node.

In our next lesson, we will put all the pieces from the last two lessons together. We will trace the data flow and component interaction during the execution of a full MapReduce job, seeing how YARN, the MapReduce engine, and HDFS work in concert to process data at scale.