Introduction
In our last lesson, we constructed a framework for making "build versus buy versus adopt" decisions, a critical task for setting the technical direction of a platform. Those decisions, made over the course of years by many teams, create the system you inherit and manage today. The cumulative result of those choices, both the expedient and the well-considered, is your organization's technical debt.
This lesson directly addresses the learning outcome: Develop a strategy for identifying, prioritizing, and managing technical debt in large-scale systems. For a Principal Engineer, managing technical debt is not about chasing perfect code; it's an exercise in portfolio management. You must move beyond simple static analysis and develop a sophisticated, data-driven strategy that aligns technical health with business velocity and risk. We will explore how to identify the debt that truly matters, create a framework for prioritizing it, and establish sustainable processes for managing it at scale.
The Inevitability of Debt in Evolving Systems
Large-scale systems are in a constant state of flux. This reality is captured by Lehman's laws of software evolution, which state that a system must continuously change or become less useful, and that as it evolves, its complexity increases unless work is done to reduce it. This dynamic is the genesis of technical debt.
Prioritizing Technical Debt as If Time & Money Matters • Adam Tornhill • GOTO 2022
To understand this dynamic, let's watch a segment from Adam Tornhill's presentation "Prioritizing Technical Debt as If Time & Money Matters." He provides a clear explanation of how Lehman's laws create a vicious cycle of increasing complexity and slowing delivery.
Watch the first part of the talk, from the introduction of Lehman's laws to his discussion of the various symptoms of technical debt—symptoms you've likely seen manifest as missed deadlines, unpredictable estimates, and high staff turnover.
As the video highlights, the root cause—the debt itself—is often invisible. The challenge at Google's scale is that this "debt" is not a monolith. It exists in many forms, some of which are not immediately obvious.
[PDF] Searching for Build Debt: Experiences Managing Technical Debt at ...
This research paper from Google provides a fascinating look into the specific, and sometimes unique, types of technical debt that accumulate in a massive, monolithic codebase. It moves beyond "messy code" to systemic issues.
As you read, focus on understanding the categories of debt. Don't worry about the implementation details of the tools. Read the Introduction to understand the context and Google's guiding principles for managing debt: automation and shaping developer behavior. Familiarize yourself with the concept of "Build Debt" and why it's a significant issue. In the Dependency Debt section, understand the pain points of under- and over-declared dependencies and underutilized dependencies. Read about "Zombie Targets," Google's term for dead code that incurs maintenance costs during large-scale refactorings. In the Visibility Debt section, understand how the lack of proper encapsulation created debt and how changing the default visibility was a strategy to pay it down. Finally, look at Dead Flags to see how unused configuration options contribute to complexity and debt.
Broadening the Definition: Debt in ML Systems
Your interest in AI tooling is particularly relevant here, as machine learning systems introduce novel and often more insidious forms of technical debt. Unlike traditional code, where dependencies are explicit, ML models create implicit dependencies on data and the external world, eroding abstraction boundaries in non-obvious ways.
Machine Learning, Technical Debt, and You - D. Sculley (Google)
D. Sculley, a director at Google Brain, gives a canonical overview of this topic in his talk, "Machine Learning, Technical Debt, and You."
Watch these two key segments: First, understand the core argument for why ML systems have a special ability to create debt beyond normal code issues. Next, absorb the descriptions of the primary sources of ML debt, including boundary erosion, feedback loops, data dependencies, and calibration cascades.
These examples from Google's build systems and ML platforms illustrate a critical point: a successful strategy must begin with identifying and categorizing the specific types of debt that exist in your ecosystem.
A Data-Driven Strategy for Prioritization
With a sprawling codebase, the most common failure mode is trying to fix everything, or getting paralyzed by the sheer volume of issues reported by static analysis tools. A Principal Engineer's primary contribution is to bring clarity and focus by answering the question: Which debt matters most?
The answer lies in treating technical debt like financial debt. It has a principal (the cost to fix it) and an interest rate (the ongoing cost of not fixing it, usually measured in wasted engineering time). Your goal is to pay down the highest-interest loans first.
Step 1: Identify the Debt Principal (Code Health)
Static analysis isn't useless, but its output needs to be refined. Instead of a meaningless dollar or time value, we can use it to create a more nuanced metric for the "health" of a piece of code.
Prioritizing Technical Debt as If Time & Money Matters • Adam Tornhill • GOTO 2022
Let's return to Adam Tornhill's talk. He first critiques the common approach of quantifying debt and then introduces a better way to measure the "principal."
First, watch the segment where he deconstructs the idea of quantifying technical debt in years, explaining why it's not actionable. Next, understand the concept of Code Health, which combines multiple complexity factors (like cohesion, brain methods, and nesting) into a single, categorized metric (e.g., green, yellow, red).
This "Code Health" score represents the principal of your debt. Red, unhealthy code is harder to understand, modify, and test.
Step 2: Measure the Interest Rate (Hotspots)
The most brilliant insight from behavioral code analysis is that the interest on technical debt is only paid when developers have to interact with the problematic code. We can measure this interaction frequency by analyzing version control history.
The combination of complexity (Code Health) and change frequency (commits) allows us to identify Hotspots.
Prioritizing Technical Debt as If Time & Money Matters • Adam Tornhill • GOTO 2022
This next segment from the same talk is the cornerstone of our prioritization strategy. It explains how to use version control data to find what truly matters.
Watch the section that introduces behavioral code analysis and hotspots. Pay close attention to the visualization that overlays change frequency (the "heat") on top of code complexity (the circle size). This is how you find the code that is both complicated and frequently worked on.
Step 3: Create the Prioritization Matrix
By combining code health and change frequency, we can build a simple but powerful prioritization matrix.
| High Change Frequency (Hot) | Low Change Frequency (Cold) | |
|---|---|---|
| Low Code Health (Red) | High Priority: Refactor Now These are your high-interest credit cards. They actively slow down development and are a primary source of bugs. |
Low Priority: Contain & Monitor This is legacy code that works. Don't refactor for the sake of it. Isolate it behind clean APIs. |
| High Code Health (Green) | Protect This is the healthy, active core of your system. Protect it with high test coverage and rigorous code reviews. |
Ignore This is healthy code that rarely changes. Leave it alone. Your time is better spent elsewhere. |
This matrix immediately cuts through the noise. As Tornhill points out, a vast majority of code falls into the long tail of the distribution—the "cold" parts.
This data-driven approach allows you to focus your limited resources where they will have the maximum impact on developer productivity and system stability.
Visualizing and Communicating the Strategy
As a Principal Engineer, you must communicate this strategy to leadership and other teams. A powerful way to do this is to map technical health to business capabilities.
This dashboard view instantly communicates where the biggest risks are. A "red" box in a critical area like "Core Services" is a much higher priority than a "red" box in a less critical internal tool.
Incorporating the Human Dimension
Technical debt is not just a property of code; it's also a function of the team that maintains it. A complex piece of code known inside and out by a team of experts is less of a liability than a simple piece of code that no one understands. This introduces the concept of knowledge distribution and the "bus factor."
Prioritizing Technical Debt as If Time & Money Matters • Adam Tornhill • GOTO 2022
The final part of this strategy involves analyzing the social or human side of your code.
Watch this concluding segment on the people side of technical debt. It demonstrates how to use version control data to visualize knowledge distribution, simulate the impact of a key developer leaving, and use that information to prioritize knowledge transfer and collaborative refactoring.
Your prioritization framework should therefore have a third dimension: Knowledge Distribution. A hotspot with low code health maintained by a single developer who is about to leave becomes your number one priority, as the risk of that component becoming unmaintainable is extremely high.
A Framework for Management
With a prioritized list, the final step is to create a sustainable process for managing the debt.
This process involves a portfolio of tactics:
- Inventory Your Debt: Use behavioral analysis and code health metrics to create a prioritized backlog.
- Adopt a Model for Action: Based on your prioritization matrix, decide what to do. For top-priority hotspots, you might use an "x-ray" technique to find the exact functions to refactor, as shown in the GOTO video.
- Prioritize High-Impact Debt: Focus on the red, hot, low-knowledge areas first.
- Invest in Tooling and Processes: This is where the Google principles shine.
- Automation: Build tools to automatically detect and, where possible, fix common debt patterns (like Google's tools for dependency management).
- Make it easy to do the right thing: Integrate health checks and guidance directly into the developer workflow (e.g., code review bots).
- Make it hard to do the wrong thing: Enforce architectural rules at the build or submit level (e.g., Google's shift to private-by-default visibility).
- Balance Debt with Business Goals: Allocate a consistent budget (e.g., 20% of capacity) for debt repayment. Frame this not as "cleaning up code," but as "investing in future feature velocity." Use your capability health map to justify this investment.
Conclusion
Developing a strategy for technical debt is a hallmark of a Principal Engineer's strategic thinking. It requires moving beyond code-level debates to a data-driven, system-level perspective.
Your key takeaways from this lesson are:
- Broaden Your Definition: Technical debt manifests in many forms, including build systems, configurations, and the implicit dependencies within ML systems.
- Prioritize with Data, Not Dogma: The most effective prioritization strategy combines code health (the principal) with change frequency (the interest rate) to identify hotspots where debt is actively harming productivity.
- Incorporate Human Factors: The distribution of knowledge is a critical risk multiplier. A hotspot known by only one person is a significant liability.
- Manage Systematically: Debt management is not a one-off project. It requires a continuous process of inventory, prioritization, and repayment, supported by automation and processes that guide developers toward healthier patterns.
In our next lesson, we will continue exploring the strategic trade-offs a Principal Engineer must navigate. We will focus on how to deconstruct competing requirements from multiple stakeholders into a clear set of engineering trade-offs, a skill that is essential when advocating for technical initiatives like debt repayment in the face of relentless pressure for new features.