Diagnosing API Latency Spikes: A Systematic Approach

1. The Interview Question: "Your API is Slow. What Do You Do?"

Imagine you are on-call. An alert fires: p99 latency for a critical API endpoint has jumped from 200ms to over 5 seconds. Users are reporting timeouts. This is a classic, open-ended interview scenario designed to test your entire approach to problem-solving.

To understand the interviewer's perspective, let's start with a video that frames this exact problem.

As the video highlights, the goal isn't to guess the correct cause immediately. The goal is to demonstrate a logical, methodical investigation that narrows down the possibilities.

2. The Framework: A Systematic Approach to Diagnosis

A structured approach prevents you from chasing red herrings. A proven methodology involves moving from a high-level service view down to specific resource or code-level issues.

Step 1: Check Service-Level Indicators (The RED Method)

Before diving into servers, start with what your users are experiencing. The RED Method provides a perfect framework for this initial assessment.

• Rate: The number of requests per second.
• Errors: The number of requests resulting in an error.
• Duration: The time it takes for requests to complete (latency distribution, e.g., p50, p95, p99).

By looking at these three metrics on your primary observability dashboard, you can form an initial hypothesis:

• Latency spike + Rate spike? Suggests a traffic surge, a denial-of-service attack, or a client-side retry storm.
• Latency spike + Error spike? Points to a failing downstream dependency or a partial service outage.
• Latency spike only? Suggests an internal bottleneck in your service, like resource contention or an inefficient code path.

Step 2: Correlate Metrics with Logs and Traces

Metrics tell you what is wrong, but logs and traces tell you why.

Your next action is to find a distributed trace for one of the slow requests. The trace waterfall will immediately show you which part of the request lifecycle is consuming the most time:

• Is it a long span within your application code?
• Is it a long span representing a database query?
• Is it a long span for an HTTP call to a downstream service?

This allows you to isolate the problem to a specific component.

3. Drilling Down: From Hypothesis to Root Cause

Once you've isolated the problematic component, you need to dig deeper using more specific tools and frameworks.

Let's put this into practice with a few scenarios.

Scenario A: The Trace Points to Your Application Server

The distributed trace shows that your service is the bottleneck. The latency is within your application code, not in external calls. Now you apply Brendan Gregg's USE Method to the server itself.

• Utilization: How busy is the resource (e.g., CPU is 90% utilized)?
• Saturation: Is there a queue of work that cannot be serviced (e.g., CPU run queue length is high)?
• Errors: Are there any resource-specific errors (e.g., out-of-memory errors)?

The following video demonstrates a hands-on diagnosis of performance issues on a Linux server, which is the environment most backend services run in.

Scenario B: The Trace Points to the Database

The trace shows a single database query is taking seconds.

• Hypothesis 1: Connection Pool Exhaustion. Your service's connection pool metric would show active connections are at or near the maximum limit. New requests are queuing up waiting for a connection.
• Hypothesis 2: Slow Query. You would extract the query from the trace, run EXPLAIN ANALYZE on it in PostgreSQL, and look for a full table scan or other inefficiencies. The fix might be adding an index, a topic we covered previously.
• Hypothesis 3: Lock Contention. The query might be blocked by another transaction holding a lock. You'd use database-specific views (like pg_stat_activity and pg_locks in PostgreSQL) to investigate waiting queries.

Scenario C: The Trace is Inconclusive, but Latency is High

Sometimes the application and database seem fine. The Last9 article gave a brilliant clue for this: look at the pattern of latency. If spikes follow a sequence like 1s, 3s, 7s, 15s, this strongly suggests TCP packet retransmissions due to network issues between services. This is a very impressive observation to bring up in an interview.

4. Structuring Your Interview Answer

Now, let's synthesize this into a structured answer.

Interviewer: "p99 latency for your /checkout endpoint just spiked from 300ms to 8 seconds. Walk me through how you'd investigate."

Your Answer:

1. Acknowledge and Scope: "My first step is to check our service's primary dashboard to understand the scope. I'm looking at the RED metrics: Rate, Errors, and Duration.

   • Did the request Rate spike? This could point to a traffic surge.
   • Did the Error rate also spike? This could mean a downstream service is failing.
   • If only the Duration (latency) is high, it suggests an internal bottleneck."

2. Isolate with Tracing: "Next, I'll grab a distributed trace for a slow /checkout request. The trace waterfall will immediately tell me where the time is being spent—in the API gateway, the checkout service, the payment service, or the database."

3. Drill Down with a Hypothesis: "Let's assume the trace shows the latency is within the checkout service itself. My hypothesis is a resource bottleneck. I'll apply the USE method to the server instance.

   • I'll SSH into the machine and start with top or htop to check for CPU utilization.
   • If CPU is high, I'll use pidstat to confirm it's my application process. Then, I might use perf record to sample the CPU stack and see which functions are hot.
   • If CPU seems fine, I'll check vmstat for memory pressure (swapping) or high iowait, which points to a disk I/O problem. I'd confirm that with iostat.
   • If the issue seems to be with system calls, strace can reveal if the application is stuck in a tight loop of reads or writes."

4. Consider Other Culprits: "If the server resources look fine, I'd investigate other common culprits. I'd check our GC logs for long 'stop-the-world' pauses. If the trace showed a long database query, I'd move to analyzing that query with EXPLAIN ANALYZE."

5. Communicate and Mitigate: "Throughout this process, I'd be communicating my findings in the team's incident channel. Once I have a likely root cause—say, a missing index—the immediate mitigation might be to roll back the code that introduced the query, while the long-term fix is to deploy a migration that adds the index."

Conclusion

You now have a robust, multi-layered framework for diagnosing latency spikes. This approach—moving from high-level service metrics (RED) to infrastructure metrics (USE) and then to specific code/system-level tools (perf, strace)—demonstrates the structured, calm, and methodical thinking that defines a senior backend engineer.

Key Takeaways:

• Start with service-level metrics (RED): Understand the user impact (Rate, Errors, Duration) before diving deep.
• Use distributed traces to isolate: Find the component (your service, a dependency, the database) responsible for the delay.
• Apply the USE method for resource bottlenecks: Systematically check for Utilization, Saturation, and Errors on CPU, memory, disk, and network.
• Master a few key command-line tools: Knowing when to use top, vmstat, iostat, strace, and perf is a powerful skill.
• Always have a list of common culprits: Keep DB connection pools, GC pauses, and network issues in your mental checklist.

Preview of the Next Lesson:

Diagnosing a technical problem is one type of interview challenge. Another, equally important one, is communicating your experience and ownership. In our next lesson, we will shift from a technical debugging scenario to a behavioral one. You will learn how to develop a structured response for a behavioral question about a past project failure using the STAR method, focusing on ownership, impact, and lessons learned.