Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 79

Crafting a Root Cause Analysis Report

Recommended resources

Hello! Welcome to the final lesson in your course on data center operations.

In our last lesson, you learned how to be proactive by implementing monitoring alerts to detect potential failure trends. Today, we'll address what happens when an incident occurs despite our best efforts. We'll focus on the critical process of learning from these events to build more resilient systems.

This lesson directly addresses the learning outcome: Write a structured Root Cause Analysis (RCA) report for a simulated production incident. This is a core competency for any role in operations, as it demonstrates accountability, analytical thinking, and a commitment to continuous improvement. Mastering this will not only make you a better engineer but also equip you with a key skill that hiring managers look for.

1. The "What" and "Why" of Incident Analysis

When a system fails, the first priority is always to restore service. But the work doesn't end there. The most valuable phase comes next: understanding what happened, why it happened, and how to prevent it from happening again. This process is formally known as a post-mortem, and the detailed investigation into the "why" is the Root Cause Analysis (RCA).

It's helpful to think of it this way:

  • RCA: The investigation to find the deepest underlying cause of a problem.
  • Post-Mortem: The entire process and report that documents the incident, including the RCA, timeline, impact, and future actions.

Before we dive into the structure of the report, let's establish the most important principle of this process: the blameless post-mortem.

How to run a Post-Mortem meeting, step-by-step

The video 'How to run a Post-Mortem meeting, step-by-step' by Bran van der Meer excellently explains the purpose of a post-mortem and the crucial concept of creating a 'blameless' culture. Understanding this mindset is non-negotiable in modern tech organizations.

Watch the first section, 'Blameless' (00:00 - 02:38). Pay close attention to why 'blame prevents learning' and how to shift the focus from 'who' to 'how and why'. This is a major cultural point you can discuss in job interviews.

The key idea is that individuals don't cause failures; flawed systems and processes do. A person may have taken an action that triggered the event, but a robust system should have prevented that action from causing an outage. The goal of an RCA is to find and fix those systemic flaws.

2. Anatomy of a Professional RCA Report

A good RCA report is structured, factual, and actionable. It tells a clear story that anyone in the organization can understand, from an engineer to a manager.

Let's look at a comprehensive article that breaks down the standard structure and provides excellent examples.

The Ultimate Post Mortem Template

The article 'The Ultimate Post Mortem Template' from UptimeRobot provides a fantastic breakdown of a professional post-mortem report. We will use its structure as our guide.

Please read the following sections: Blameless post mortems: This will reinforce the concept from the video with clear ground rules. Core structure of a post-mortem template: Focus on the 'Mandatory sections'. This is the blueprint we will use. Note how it captures the full story of the incident. FAQ's: Read the first FAQ that clarifies the difference between a Post-Mortem and an RCA.

Based on that reading, here are the mandatory sections of a high-quality RCA report:

  • Incident Summary: A high-level, plain-language overview.
  • Incident Timeline: A precise, timestamped log of key events.
  • Root Cause & Contributing Factors: The core analysis of why the incident occurred.
  • Impact: A quantification of the effect on users and the business.
  • Detection & Response Effectiveness: An evaluation of how your monitoring and team performed.
  • Corrective Actions: A list of specific, owned tasks to prevent recurrence.
  • Lessons Learned: Broader takeaways for the team and organization.

Here are a couple of examples of what these templates look like in a corporate environment.

Root Cause Analysis Report Template
This image shows a formal RCA report template, likely from an IT service management tool like ServiceNow. Notice the fields for collecting structured data like priority, impact duration, and root cause category.
Concise RCA - An Introduction and Checklist
This template from Wipro provides a concise, one-page structure for an RCA. It visually highlights the key steps: defining the problem, identifying the root cause, and defining countermeasures.

Finding the True Root Cause: The "5 Whys"

One of the most effective techniques for drilling down to the true root cause is called the "5 Whys." You start with the surface-level problem and repeatedly ask "Why?" until you arrive at a fundamental process or system flaw.

Let's review a resource that explains this practical technique.

Incident Postmortem Template

The 'Incident Postmortem Template' by Atlassian provides another great template and includes a clear explanation of the 'Five Whys' technique.

Read the section titled 'Root cause identification: The Five Whys'. Pay attention to the example provided. See how it moves from a technical failure ('database was locked') to a process failure ('we don't have a development process established for load testing').

3. Practice Your "Katana": Writing an RCA Report

Now, let's apply this theory. We'll use a simulated incident that builds on our previous lessons. Your task is to act as the lead engineer and write the RCA report.

First, to see what a completed report looks like, please study the excellent example from the UptimeRobot article.

The Ultimate Post Mortem Template

This section of 'The Ultimate Post Mortem Template' provides a step-by-step walkthrough of creating a post-mortem for a fictional API outage. It's a perfect model for our exercise.

Read the entire section 'Real example: a status page post mortem in action'. This will give you a concrete example of how to fill out each part of the report we're about to create.

Our Simulated Incident

Scenario: On July 26th, a critical web application server (web-prod-01) became unresponsive. The Prometheus alert you configured for PredictiveDiskFailure (based on smartctl data) had been firing for over an hour, but because it was only configured to send an email, the on-call engineer missed it. The issue was eventually reported by a customer at 14:30 UTC. The engineer, once paged, immediately rebooted the server at 14:45 UTC, restoring service by 14:50 UTC.

A later investigation revealed that a recent software update accidentally left verbose debug logging enabled for a new feature. This caused the application's log file to grow uncontrollably, filling the root disk (/dev/sda) to 100% and causing the operating system to freeze.

Your Task: Draft the RCA Report

Using the structure we've learned, draft the RCA report for this incident. I'll guide you through each section.

1. Incident Summary

  • Describe what happened, the duration of user-facing impact, and the high-level cause.
    • Example based on scenario: "Between 14:30 and 14:50 UTC, the main web application was unavailable. The outage was caused by a full disk on web-prod-01, triggered by excessive logging from a recent software update."

2. Incident Timeline (UTC)

  • Create a chronological list of key events.
    • 13:20: Prometheus alert PredictiveDiskFailure begins firing for web-prod-01.
    • 14:30: Customer support files a P1 ticket; application is unreachable. On-call engineer is paged.
    • 14:35: On-call engineer acknowledges page and begins investigation.
    • 14:45: Engineer identifies the unresponsive server and initiates a reboot as an immediate fix.
    • 14:50: Server web-prod-01 is back online; monitoring confirms service is restored.
    • 15:30: Investigation concludes, identifying the rogue logging process as the cause.

3. Root Cause & Contributing Factors

  • Use the "5 Whys" to find the root cause.
    1. Why did the application go down? The server OS froze.
    2. Why did the OS freeze? The root disk was 100% full.
    3. Why was the disk full? A log file grew to fill all available space.
    4. Why did the log file grow so large? A recent software update left verbose debugging enabled in production.
    5. Why wasn't the problem caught before it caused an outage? The automated alert was sent to a low-priority channel (email) and was not acted upon.
  • Primary Root Cause: A bug in the deployment process allowed debug-level logging to be enabled in production.
  • Contributing Factors:
    • The alerting channel for critical disk issues was not appropriate for an urgent response.
    • The system lacked automated log rotation or disk usage limits to act as a failsafe.

4. Impact

  • Describe the customer and business impact.
    • Duration: 20 minutes of total downtime for users.
    • User Impact: All users attempting to access the application during the outage received an error.
    • Business Impact: Potentially XX lost transactions and a high volume of customer support tickets.

5. Detection & Response Effectiveness

  • Evaluate your process.
    • Detection: The issue was detected automatically by Prometheus 70 minutes before user impact, but the alert was ineffective. The functional outage was detected manually by a customer.
    • Response: Once paged, the engineer responded quickly and restored service within 20 minutes.

6. Corrective Actions

  • This is the most important section. List specific, actionable tasks with owners.
    | Action Item | Owner | Deadline |
    | :--- | :--- | :--- |
    | Fix: Remove verbose logging from production code and deploy hotfix. | Dev Team Lead | EOD |
    | Alerting: Re-route PredictiveDiskFailure and other critical alerts to PagerDuty. | SRE Team | Aug 2 |
    | Containment: Implement log rotation for all application logs. | Ops Team | Aug 9 |
    | Prevention: Add a pre-deployment check to verify logging configuration. | DevOps Lead | Aug 16 |

7. Lessons Learned

  • Summarize the key takeaways.
    • "What": Alerting is useless if it doesn't reach the right person at the right time.
    • "What went well": The monitoring system did detect the underlying issue well in advance.
    • "Where to improve": Our deployment process needs a quality gate for production configurations like logging levels.

By completing this exercise, you've created a document that not only explains a failure but also provides a clear roadmap to making the system stronger.

Conclusion

Congratulations on completing the final lesson of the course! Today, you've learned one of the most important skills in data center and site reliability operations: how to transform a failure into an opportunity for improvement.

Key Takeaways:

  • The goal of a post-mortem is learning, not blame. Focus on fixing systems, not finding fault in people.
  • A structured RCA report typically includes a summary, timeline, root cause analysis, impact assessment, and corrective actions.
  • The "5 Whys" is a powerful yet simple technique to dig past symptoms and find the true, underlying root cause of an issue.
  • The most critical output of any RCA is a list of specific, owned, and time-bound corrective actions that will prevent the incident from happening again.

You have now journeyed through the entire lifecycle of data center infrastructure—from unboxing components and installing operating systems to deploying services, monitoring their health, and systematically learning from incidents. The practical skills and theoretical knowledge you've built throughout this course provide a solid foundation for achieving your career goals. Well done

End of Module,Exit lesson