Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 16

Server Storage Device Replacement in a RAID Array

Recommended resources

Hello! In our last lesson, we played the role of a hardware detective, learning to interpret logs with tools like smartctl and dmesg to predict component failures before they become critical. We identified the what and the where of a problem.

Today, we shift from diagnosis to action. This lesson directly addresses the learning outcome: Diagnose a failed storage device in a server, safely replace it in a RAID array, and verify the rebuild process. We will take the knowledge from our last lesson—that a drive is failing—and execute the procedure to fix it. This is a fundamental, hands-on skill for any server administrator or data center technician and a process you should be able to explain confidently in a job interview.

We will cover the two main ways storage is managed in servers, as the replacement process differs for each:

  • Hardware RAID: Managed by a dedicated controller card.
  • Software RAID: Managed by the operating system (Linux mdadm).

Let's get started.

1. The Scenario: A Degraded RAID Array

Our monitoring has paid off. We've received an alert: a drive in one of our servers has failed. Because the server uses a redundant RAID array (like RAID 1, 5, or 10), it is still running, but it's in a degraded state. This means it has lost its fault tolerance. If another drive were to fail now, we would face data loss. The clock is ticking.

A RAID management tool would show a status similar to this:

RAID Storage Manager Showing Degraded Array and Failed Drive
This RAID manager interface clearly indicates a 'Failed' physical drive and a 'Degraded' logical array. This is the starting point for our maintenance task.

Our goal is to replace the failed drive and restore the array to a healthy, redundant state without any downtime, a process known as a hot-swap.

2. Path A: Replacing a Drive in a Hardware RAID Array

Hardware RAID controllers (like a Dell PERC or a Broadcom/LSI MegaRAID card) present a single logical volume to the operating system. To manage the individual drives, we need to use specialized tools that communicate directly with the controller. The most common command-line tool for this on Linux is storcli.

The process follows three main phases: Identify, Replace, and Verify.

Replace a failed hard drive in a RAID array

This article by Kamil Slowikowski is an excellent, real-world guide to replacing a failed drive in a hardware RAID array using storcli. We will use it as our primary reference for this section.

As we go through the steps, refer to this article. Focus on the commands and their outputs. We will walk through the sections sequentially: Confirm that we have a failed drive Get information about our system & Install storcli Display RAID array and drive information with storcli Locate the failed drive Replace the failed drive

Step 1: Identify the Failed Drive

First, we need to be 100% certain which drive has failed, both logically and physically. A blinking amber LED on the drive carrier is a good physical clue, but you must confirm it with software.

As shown in the article "Replace a failed hard drive in a RAID array," we use storcli to get the details.

  1. Check Array Status: Run sudo /opt/MegaRAID/storcli/storcli64 /c0 show. The output will show the array in a Dgrd (Degraded) state and often identifies the specific drive as Fail, Offln (Offline), or Rbld (Rebuilding, if a hot spare has already taken over).
  2. Get Physical Drive Details: The PD LIST (Physical Drive List) in the output maps the enclosure ID and slot number to each drive.
  3. Pinpoint the Drive: The most crucial step is to get the serial number of the failed drive. The article shows this command: sudo /opt/MegaRAID/storcli/storcli64 /c0/e252/s0 show all. The output provides the SN (Serial Number). You can then take this serial number to the server room and match it to the label on the physical drive you need to pull. This prevents a catastrophic mistake of pulling the wrong drive.

Step 2: Safely Replace the Drive (Hot-Swap)

With the correct drive identified, we can proceed with the replacement. To ensure the system handles the removal gracefully, we first inform the RAID controller.

The article provides the exact sequence of commands under the section "Replace the failed drive":

  1. Set the drive offline: This tells the controller to stop trying to use the drive. 
    


# Replace cx, ey, and sz with your controller, enclosure, and slot numbers
sudo storcli64 /cx/ey/sz set offline
  2. Mark the drive as missing: This removes it from the RAID configuration. 
    sudo storcli64 /cx/ey/sz set missing
  3. Physically Swap: Now it is safe to unlatch and remove the drive carrier from the server. Place the new, identical replacement drive into the carrier and insert it firmly back into the same slot.

This video demonstrates the physical process, though it uses the BIOS for identification. The physical act of replacing the drive in the chassis is the same.

FAQ - Common RAID Configuration Tasks with Dell PowerEdge Servers

This video from STI shows the process for a Dell PowerEdge server. Pay attention to the physical actions of identifying the failed drive by its light and performing the hot-swap.

Watch from 01:50 to 02:30. Notice how the presenter identifies the failed drive, removes the carrier, and inserts the new drive. This is the physical action that follows the storcli commands we just learned.

Step 3: Verify the Rebuild

Once the new drive is inserted, a modern RAID controller with a proper configuration (auto-rebuild enabled) will detect it and automatically begin the rebuild process. The controller reads the data and parity from the other drives in the array to reconstruct the data onto the new drive.

  1. Monitor Progress: You can watch the rebuild status using storcli.

    


# Check the rebuild status for the new drive in slot z
sudo storcli64 /cx/ey/sz show rebuild

    This command will show a progress percentage and an estimated time remaining. This process can be I/O intensive and may take many hours for large drives.

  2. Confirm Health: After the rebuild completes, run sudo storcli64 /c0 show again. The virtual drive state should now be Optl (Optimal) and the new physical drive state should be Onln (Online). The array is now healthy and redundant again.

3. Path B: Replacing a Drive in a Software RAID Array (mdadm)

In Linux software RAID, the mdadm utility is your primary tool. The OS manages the RAID array directly, so the process involves different commands, but the logical flow—Identify, Replace, Verify—is the same.

Step 1: Identify the Failed Drive

Software RAID status is transparently available in the OS.

  1. Check Array Status: The /proc/mdstat file is the source of truth for mdadm. 
    cat /proc/mdstat
    A healthy two-drive RAID1 array shows [UU]. A degraded array will show [U_] or [_U], indicating which drive has failed.
  2. Check System Logs: The kernel will log I/O errors related to the failing device (e.g., /dev/sdc), often visible with dmesg.

Replacing A Failed Hard Drive In A Software RAID1 Array

This HowToForge article provides a clear, command-line walkthrough for replacing a drive in a software RAID1 array. It's a great textual reference for the mdadm process.

Read sections 2. How Do I Tell If A Hard Disk Has Failed? and 3. Removing The Failed Disk. Note the use of cat /proc/mdstat to see the degraded state and the mdadm --fail and --remove commands.

Step 2: Safely Replace the Drive

The process involves telling mdadm that the drive is failed, removing it from the array, and then performing the physical swap.

This video provides a great, practical demonstration of the entire mdadm workflow.

How To Rebuild and Resync From Failed Hard Drive Raid 5 on mdadm

This Novaspirit Tech video is a perfect practical guide. The presenter simulates a failure and walks through the exact commands needed to replace the drive and rebuild the array.

Watch the following segments to see the process in action: Simulating Failure (03:44 - 05:02): The presenter uses mdadm --set-faulty to mark a drive as failed. This is the first step in a planned replacement. Removing the Drive (05:36 - 07:11): He then uses mdadm --remove to detach the drive from the array, making it safe to replace. Adding the New Drive (08:12 - 09:21): After the physical swap (which is assumed to happen off-screen), he adds the new drive to the array with mdadm --add, which automatically starts the rebuild.

To summarize the steps from the video and article:

  1. Mark as Failed: sudo mdadm --manage /dev/md0 --fail /dev/sdc1
  2. Remove from Array: sudo mdadm --manage /dev/md0 --remove /dev/sdc1
  3. Physically Swap: Power down the server (if not hot-swappable) and replace the disk.
  4. Partition New Drive: This is a critical step in software RAID. The new drive is blank. You must create an identical partition layout. A powerful shortcut is to copy the partition table from a healthy drive in the array: 
    sudo sfdisk -d /dev/sdb | sudo sfdisk /dev/sdc
  5. Add New Drive to Array: Add the newly created partition to the RAID device. 
    sudo mdadm --manage /dev/md0 --add /dev/sdc1

Step 3: Verify the Rebuild

With software RAID, mdadm starts the rebuild as soon as the new partition is added.

  1. Monitor Progress: The cat /proc/mdstat command is used for monitoring. While a rebuild is in progress, you will see a status line showing the percentage complete and estimated time. 
    md0 : active raid1 sdb1[0] sdc1[2]
      24418688 blocks [2/1] [U_]
      [=>...................]  recovery =  9.9% (2423168/24418688) finish=2.8min speed=127535K/sec
  2. Confirm Health: Once the rebuild is finished, cat /proc/mdstat will show the array as active and the drive status as [UU], confirming the array is healthy and redundant again.

Conclusion

You have now walked through one of the most common and critical maintenance tasks in a server environment: replacing a failed RAID drive. Understanding the difference between the hardware (storcli) and software (mdadm) procedures is essential and demonstrates a practical depth of knowledge that employers look for.

Key Takeaways:

  • A failed drive in a redundant RAID array leads to a degraded state, which is urgent but not catastrophic.
  • The general process is always Identify -> Replace -> Verify.
  • For Hardware RAID, use vendor-specific tools like storcli to manage the array, set drives offline, and monitor the rebuild. The key is to get the correct slot number and serial number.
  • For Software RAID, use mdadm to manage the array. The key steps are marking the drive as failed, removing it, partitioning the new drive, and adding it back to the array.
  • Monitoring the rebuild process is crucial to confirm that the array has returned to a healthy, fault-tolerant state.

In our previous lessons, we assembled PCs, configured servers, and diagnosed hardware. Now you've performed a critical repair on a server's storage subsystem. With this solid foundation in managing physical hardware, we are ready to move up the stack to virtualization.

In the next lesson, we will explain the concept of virtualization (Type 1 vs Type 2 hypervisors) and set up a virtual machine for OS installation. You'll learn how to take a single powerful server and carve it into multiple, isolated virtual servers, a cornerstone of modern IT infrastructure.