Estimating Log Throughput & Storage

The Art of Estimation

Before we start crunching numbers, it's important to set the right mindset. Back-of-the-envelope calculations are not about finding perfectly precise answers. They are about getting to the right order of magnitude to guide our high-level design. Is our system handling megabytes, gigabytes, or terabytes of data per day? This is the kind of question we aim to answer.

To get a feel for this process, let's start with a quick overview.

As the video explains, these estimates help us make crucial early decisions. For instance, knowing our write throughput helps us determine if a single server is sufficient or if we need a distributed cluster.

A Framework for Log Ingestion Capacity Planning

To structure our estimation, we'll follow a systematic, three-step process:

1. State Assumptions: We'll define the key variables: event volume, peak traffic patterns, and average log size.
2. Calculate Write Throughput: We'll determine the peak ingestion rate our API must handle (e.g., in MB/s).
3. Estimate Storage Needs: We'll calculate the total data volume over a given retention period, factoring in replication and compression.

Let's walk through this process for your log ingestion system.

Step 1: State Your Assumptions

This is the most critical part of the process. Since we don't have production metrics yet, we must make reasonable, clearly stated assumptions. Anyone reviewing our design should be able to understand and question these assumptions.

Assumption 1: Event Volume

First, we need to estimate the number of events per day. Let's base this on the number of users in your organization.

• Number of Employees: Let's assume your company has 10,000 employees.
• Events per Employee: Okta logs are generated for many activities: login, logout, password change, API access, etc. Let's estimate that an active employee generates, on average, 50 log events per day.
• Total Daily Events: 10,000 employees × 50 events/employee/day = 500,000 events/day.

Assumption 2: Peak Traffic

Traffic is never uniform. It spikes when everyone logs in around 9 AM and drops off overnight. We must design for the peak, not the average. A common rule of thumb is that peak traffic can be 2-5 times the average.

• Peak Multiplier: Let's assume traffic during peak hours is 3x the average.

Assumption 3: Average Log Size

The size of each log event is crucial for calculating both throughput and storage. Let's look at a sample Okta event log.

A single JSON log event, with its descriptive keys and nested objects, can vary in size. While some might be small, others can contain detailed metadata.

• A simple event might be ~200 bytes.
• A complex event with full user-agent strings, IP info, and multiple targets could be over 1 KB.

Let's be conservative and assume an average log size of 500 bytes (0.5 KB). This accounts for the JSON payload and some HTTP header overhead. The article "Exabyte-Scale Observability" (ID: LINK) also uses this as a conservative estimate in its calculations for a very large-scale system.

Step 2: Calculate Write Throughput

Now we can translate our assumptions into the demands placed on our ingestion API.

Average Write QPS (Queries Per Second)

First, let's find the average number of events arriving per second.

• Average QPS = Total Daily Events / Seconds in a Day $$\frac{500,000\text{ events}}{86,400\text{ s}}\approx 5.8\text{ events/sec (EPS)}$$

This seems very low, but it's just the average.

Peak Write QPS

Our system must handle peak load gracefully.

• Peak QPS = Average QPS × Peak Multiplier $$5.8\text{ EPS}\times 3\approx 17.4\text{ EPS}$$

Let's round this up to 20 EPS to give ourselves a small buffer. This is the number of POST requests our API endpoint should expect to handle per second during busy periods.

Ingestion Throughput (Bandwidth)

Finally, let's calculate the required network bandwidth for our ingestion service.

• Ingestion Throughput = Peak QPS × Average Log Size $$20\text{ events/sec}\times 0.5\text{ KB/event}=10\text{ KB/s}$$

This is a very modest number. However, these calculations are powerful because you can easily plug in different assumptions. If your company had 1 million employees, the peak QPS would be closer to ~1740 EPS and the required throughput would be ~870 KB/s.

Step 3: Estimate Storage Needs

Next, let's calculate how much disk space we'll need. This depends heavily on your company's data retention policies.

Assumption 4: Data Retention, Replication, and Compression

• Retention Policy: A common strategy is to keep logs in a fast, searchable "hot" storage for a short period and then move them to cheaper "cold" storage for long-term compliance. Let's assume:
  • 1 year in hot storage (e.g., Elasticsearch).
  • 7 years total retention (the remaining 6 years in cold storage like AWS S3).
• Replication: For durability and high availability, data is almost always replicated. A standard replication factor is 3x.
• Compression: Raw JSON is verbose. Using a storage system with good compression (or applying it ourselves with formats like Protobuf or Avro) can save massive amounts of space. The "Exabyte-Scale Observability" article notes that systems like ClickHouse can achieve 10-20x compression, while Elasticsearch is closer to 2x. Let's assume a conservative 3x compression ratio for our stored data.

Daily Data Volume (Uncompressed)

This is the amount of raw data we ingest per day.

• Daily Volume = Total Daily Events × Average Log Size $$500,000\text{ events/day}\times 0.5\text{ KB/event}=250,000\text{ KB/day}=250\text{ MB/day}$$

Total Storage Calculation

Now let's calculate the storage needed for 1 year of hot data.

• Total Hot Storage (1 year) = Daily Volume × 365 days/year × Replication Factor / Compression Ratio $$(250\text{ MB/day}\times 365)\times 3/3=91,250\text{ MB}\approx \text{91 GB}$$

This is the amount of disk space needed for your searchable logs for the first year. For the full 7-year retention, you would calculate the total storage similarly and plan for the bulk of it to reside in a cheaper object store.

Summary of Estimates

Here is a summary of our back-of-the-envelope calculation.

These numbers provide a concrete starting point. While they seem small for 10,000 employees, the framework is what's important. You can now confidently adjust the inputs based on better data from your real-world project and see how it impacts the entire system's scale.

To see how these same principles apply at massive scale, you can refer back to the "Exabyte-Scale Observability" article (ID: LINK), which calculates throughput in gigabytes per second and storage in petabytes per day. The fundamental math is the same.

Conclusion

In this lesson, we demystified capacity planning by breaking it down into a simple, repeatable process. We started with justifiable assumptions and used them to calculate concrete targets for write throughput and storage.

Key Takeaways:

• Start with assumptions: Clearly state your assumptions about volume, peak traffic, and data size. These are the foundation of your estimate.
• Design for the peak: Systems must be provisioned to handle peak load, not just the average.
• Throughput is a function of QPS and size: The required ingestion bandwidth is simply Peak QPS * Average Message Size.
• Storage is a function of volume, retention, and replication: The formula Daily Volume * Retention Days * Replication Factor is your guide, but remember to factor in the significant savings from compression.

You now have a quantitative understanding of the scale of your log ingestion system. This is no longer an abstract problem but one with tangible numbers that will drive our next set of engineering decisions.

Preview of the Next Lesson:
Our calculations show we need to handle a peak of ~20 requests per second. Our next challenge is to design an API endpoint that can handle this load reliably and correctly. In the next lesson, we will begin Module 2 by designing a robust API endpoint, focusing specifically on an idempotency strategy to correctly handle the "at-least-once" delivery guarantee from webhook providers like Okta.