Introduction
In our last lesson, we focused on transforming a high-level technical vision into a concrete, multi-year roadmap. That roadmap identifies the major initiatives—the "big rocks"—that will drive your organization's technical strategy. Now, as we pick up one of those rocks, a fundamental question arises: how should we actually implement it? Do we build a solution ourselves, buy a commercial product, or adopt an existing open-source project?
This lesson directly addresses the learning outcome: Create a framework for evaluating build versus buy versus adopt decisions for infrastructure components. This is a critical responsibility for a Principal Engineer. Your ability to guide teams through this decision-making process has profound, long-term consequences for cost, velocity, and competitive advantage. We will move beyond gut feelings and "Not Invented Here" syndrome to construct a systematic framework for making these pivotal choices.
The Spectrum of Choice: Build, Buy, Adopt, and Blend
The decision is rarely a simple binary choice between building from scratch and buying a product. It's a spectrum of options, each with a distinct profile of costs, benefits, and risks.
Digital Strategy Dilemma: Choosing Between Software Build vs. Buy In Digital Transformation
To begin, let's explore the classic "build vs. buy" dilemma. This video from Digital Transformation with Eric Kimberling provides a clear, high-level overview of the pros and cons of each path, and introduces the idea of hybrid models.
Please watch the following segments to ground our discussion: First, understand the core arguments for the build option, focusing on factors like tailoring for unique needs, competitive advantage, and flexibility. Next, review the primary benefits of the buy option, such as leveraging established technology, vendor R&D, and easier integration. Pay close attention to the hybrid options presented, like low-code platforms and customizing COTS (Commercial Off-The-Shelf) software. These represent a blend of building and buying. Finally, absorb the high-level guidance on when each option makes sense, which introduces key decision criteria we will formalize.
The video frames the decision primarily as "build vs. buy." In our context, especially for infrastructure, we must explicitly add a third major category:
- Build: Design and implement a bespoke solution in-house. You own the code, the architecture, and the full lifecycle.
- Buy: Purchase a license or subscription for a commercial product (e.g., Datadog for observability, Okta for identity). You are paying for a managed service or a supported product.
- Adopt: Utilize an open-source project (e.g., Kubernetes for orchestration, PostgreSQL for a database). While "free," it is not without cost, as you are responsible for its operation, maintenance, and integration.
These options aren't mutually exclusive. You might "buy" a platform like Salesforce and "build" custom applications on top of it, or "adopt" open-source Prometheus and "buy" a managed visualization layer like Grafana Cloud. The framework we develop must accommodate this nuance.
Strategic Positioning: Commodity vs. Differentiator
The single most important question in this framework is: Is this component a source of competitive advantage, or is it a commodity? Pouring valuable engineering resources into rebuilding a widely available commodity is a strategic error. The goal is to focus your creative energy on what makes your business unique.
A powerful tool for visualizing this is Wardley Mapping. A Wardley Map plots the components of a system on two axes: visibility to the user (the value chain) and evolution (from genesis to commodity).
Wardley Mapping Part 1: How and why to map your business | AWS Public Sector
Let's learn the fundamentals of Wardley Mapping from its creator, Simon Wardley. This technique is invaluable for developing strategic situational awareness.
Watch these key parts of his introduction: First, grasp the crucial distinction between a graph and a map from this explanation. In a map, space has meaning. Next, understand the three core elements of a map: anchor, position, and movement. This is the foundation for creating the map. Finally, see it in action with the insurance company example. Notice how mapping their server procurement process immediately revealed a flawed assumption (custom racks) that a six-month analysis missed.
Let's apply this. Imagine you are building a new e-commerce platform. A simplified Wardley Map might look like this:
- Anchor: The Customer
- Value Chain (from top to bottom):
- Product Web Page (very visible to user)
- Shopping Cart Service
- Payment Processing Service
- Product Database
- Compute Platform
- Power (invisible to user)
- Evolution (from left to right):
- Genesis: A novel, proprietary recommendation algorithm.
- Custom-Built: A highly tailored shopping cart experience.
- Product/Rental: The Product Database (e.g., using a managed service like AWS RDS).
- Commodity: Compute Platform (e.g., EC2 instances) and Power.
This map immediately provides strategic guidance:
- Build: The recommendation algorithm is your unique differentiator. This is where your best engineers should be focused.
- Buy/Adopt: Compute and the database are commodities or products. It makes no sense to build your own cloud or database from scratch. You should buy/adopt them.
- Debate: The shopping cart and payment processing are in the middle. Here, a debate is warranted. A standard "buy" solution (e.g., Stripe, Shopify) might be sufficient, or you might decide that a custom-built experience is a key part of your brand.
As a Principal Engineer, you lead this mapping exercise to force a discussion about what is truly core to your business versus what is context.
The Economic Lens: Total Cost of Ownership (TCO)
A common mistake is to compare the upfront cost of building versus the license fee of buying. This ignores the vast, submerged part of the cost iceberg. A rigorous decision requires analyzing the Total Cost of Ownership (TCO) over a multi-year horizon (typically 3-5 years).
The components of TCO differ for each option:
-
Build:
- Upfront: Engineer salaries for R&D, design, and initial implementation.
- Ongoing: Engineer salaries for maintenance, bug fixes, on-call support, and future enhancements. Security patching.
- Opportunity Cost: What other features or products are these engineers not building? This is often the largest and most overlooked cost.
-
Buy:
- Upfront: Licensing/subscription fees, implementation and integration consulting fees.
- Ongoing: Recurring subscription fees, training costs for staff, costs for support tiers.
- Hidden Costs: Costs associated with working around limitations of the product; potential switching costs due to vendor lock-in.
-
Adopt (Open Source):
- Upfront: Engineer salaries for integration, configuration, and deployment.
- Ongoing: Engineer salaries for patching, upgrades, and operational management. Cost of infrastructure to run it.
- Risk Costs: Potential for the project to be abandoned or for critical vulnerabilities to go unpatched. Cost of finding or training expert talent.
The Contextual Filter: Scale, Culture, and Constraints
The "right" decision is highly dependent on context. A solution that is brilliant at Google scale can be a catastrophic waste of resources at a startup. Your experience with large-scale systems is a double-edged sword: it gives you deep insight, but it can also create a "scale assumption mismatch."
Why Ex-Google Engineers Over-Architecture Everything
This article directly addresses a common challenge for engineers transitioning from massive-scale environments. It provides an essential filter for our framework.
Please read the following sections: Start with the introduction to understand the core problem of "scale assumption mismatch." Read the section "What Is Googleyness and How Does It Shape Technical Decisions?" to understand the cultural and technical context that drives the "build" instinct at Google. Review the specific examples in the section "What Specific Engineering Patterns from Google Don’t Translate to Smaller Companies?" Notice how many of these relate to premature "build" decisions. Read "When Do Google Patterns Help Versus Hinder at Smaller Companies?" and focus on the suggested thresholds and the need for data-driven justification. Finally, study the FAQ answer to the question "How do I evaluate if a Google pattern is appropriate for my context?" This provides a concise summary of the evaluation criteria.
This "context filter" is crucial. Inside Google, the decision to build foundational systems like the Google File System, Bigtable, or Blaze (their build system) was justified by unprecedented scale requirements that no off-the-shelf product could meet.
Build Systems and Build Philosophy - Software Engineering at Google
Let's look at a concrete, canonical example from Google's own history: the decision to build Blaze (now open-sourced as Bazel). This chapter from Software Engineering at Google provides a masterclass in justifying a "build" decision at scale.
Please read these sections to understand the rationale: First, read the problems that arise with simple scripts, which represents a DIY "build" approach that fails to scale. Next, study the section on "The dark side of task-based build systems." These systems (like Ant or Maven) represent the "buy/adopt" options of their time. Google found their limitations unacceptable for its scale. Finally, read the summary section "Time, Scale, Trade-Offs." This explicitly lays out a decision framework based on project scale, reinforcing that the choice is not absolute but contextual.
The key takeaway is that a "build" decision of this magnitude must be a response to well-understood limitations of existing solutions that are blocking your business at its specific scale.
Synthesizing the Decision Framework
As a Principal Engineer, your role is to lead a structured, evidence-based process. Here is a framework that integrates the concepts we've discussed:
1. Frame the Problem with a Wardley Map:
- Action: Involve product and engineering leads to map the value chain for the initiative.
- Output: A shared understanding of the component's position. Is it a commodity, a product, or a potential differentiator? This sets the strategic default. (Default to Buy/Adopt for commodities; consider Build for differentiators).
2. Identify and Shortlist Candidates:
- Action: Brainstorm the specific options. What does "Build" look like? What are the leading "Buy" (commercial) and "Adopt" (open-source) candidates?
- Output: A list of 2-4 viable options to analyze in depth.
3. Conduct a TCO Analysis:
- Action: For each shortlisted candidate, work with finance and engineering managers to estimate the 3-5 year TCO. Be rigorous in estimating hidden costs, especially engineering opportunity cost.
- Output: A quantitative comparison of the long-term financial impact of each option.
4. Evaluate Qualitative and Risk Factors:
- Action: Score each option against a checklist of non-financial factors.
- Output: A qualitative comparison table. Key factors include:
- Time to Market: How quickly can we deliver value with this option?
- Team Capabilities & Morale: Do we have the expertise? Will this choice help attract/retain talent?
- Architectural Fit: How well does it integrate with our existing stack?
- Flexibility: How easy is it to adapt, extend, or replace later?
- Risk Assessment:
- Build: Execution risk, maintenance burden.
- Buy: Vendor lock-in, vendor viability, price hikes.
- Adopt: OSS project health, community support, security posture.
5. Document and Decide:
- Action: Synthesize the Wardley Map, TCO analysis, and qualitative scorecard into a single document (e.g., an RFC or Architecture Decision Record).
- Output: A clear recommendation with explicit trade-offs. The document's purpose is not just to state the decision, but to articulate the reasoning behind it, ensuring alignment and providing a record for future review.
Conclusion
You now have a robust framework to move beyond simple "build vs. buy" debates and lead a strategic evaluation of infrastructure components. This process elevates the conversation from personal preference to a data-informed decision aligned with business objectives.
Your key takeaways are:
- Strategy First: Use tools like Wardley Mapping to determine if a component is a commodity or a differentiator. Don't waste resources building what you can buy or adopt.
- Analyze Holistically: Look beyond upfront costs. A Total Cost of Ownership (TCO) analysis reveals the true long-term financial impact of a decision.
- Context is King: The right answer depends on your organization's scale, resources, and strategic goals. What works at Google may be counterproductive elsewhere, and even at Google, not everything should be built.
- Lead a Structured Process: Your role as a Principal Engineer is to facilitate this framework, ensuring that decisions are transparent, evidence-based, and well-documented.
In our next lesson, we will address a direct consequence of past technology decisions: technical debt. We will explore how to develop a strategy for identifying, prioritizing, and managing technical debt in large-scale systems, a crucial skill for maintaining the long-term health and velocity of your platforms.