Webhook System Requirements: Functional & Non-Functional

1. The Art of Asking the Right Questions

Before defining requirements, a good engineer collaborates with stakeholders to clarify the problem. The assumptions we make at the start will drastically shape the final design.

To get into this mindset, I'd like you to watch a segment of the following video. It outlines several key questions to ask when designing a webhook system. As you watch, think about how these questions apply to your goal of ingesting logs from Okta.

The video raises several excellent points. For our project, we can frame the answers like this:

• What types of events? Okta event logs, which are often related to security and user activity (e.g., user.session.start). These are critical for auditing and security monitoring, so reliability will be a high priority.
• What is the expected volume? This is something we'll need to estimate, but providers like Okta can generate a very high volume of events, especially in a large organization. We must design for scale.
• What delivery guarantees are required? We need to know if we should expect events "at least once," "at most once," or "exactly once." This dictates how we handle duplicates.
• Is event ordering important? For log analysis, chronological order is often crucial. We need to know if the provider guarantees order or if we must handle out-of-order events.
• What are the latency and security requirements? The provider will likely have a timeout for webhook calls, and we must handle authentication to ensure the requests are legitimate.

Answering these questions moves us from a vague problem statement to a set of concrete constraints we can design against.

2. Defining Functional Requirements

Functional requirements describe the specific behaviors and features the system must support. For our log ingestion pipeline, the core functionality is centered on receiving and processing the incoming data.

To explore this, let's watch another segment from the same video, which transitions from clarifying questions to defining concrete requirements.

The video discusses building a webhook provider system, which includes APIs for clients to register their webhooks. In our case, we are the webhook consumer. Okta is the provider. This means we won't be building a registration API for Okta to call. Instead, our functional requirements are focused on the receiving end:

1. Unified Log Ingestion: The system must expose a secure HTTPS endpoint to receive log events from external providers (initially Okta) via HTTP POST requests. This aligns with the "unified log collection" goal for any logging system.
2. Authentication: The system must authenticate every incoming request to verify it originates from a trusted source (Okta) and not a malicious actor.
3. Validation: The system must perform basic validation on the incoming payload to ensure it conforms to the expected structure.
4. Acknowledgement: The system must promptly send a success acknowledgement (e.g., an HTTP 2xx status) to the provider to confirm receipt. This is critical for preventing the provider from resending the same event unnecessarily.
5. Durable Storage: The system must durably persist the validated log event for further processing. "Durable" means the data survives machine failure.

This list defines what our system needs to do. Next, we'll define how well it needs to do it.

3. Defining Non-Functional Requirements (NFRs) & SLAs

Non-Functional Requirements (NFRs), or quality attributes, are just as important as functional ones. They define the system's operational characteristics and form the basis of our Service Level Agreements (SLAs). For a backend system like this, NFRs are paramount.

Let's look at a video that discusses NFRs in the context of a distributed logging system. It provides some concrete numbers which are useful for context.

Combining these general principles with the specific documentation from our provider, Okta, allows us to define precise NFRs.

Based on these resources, we can establish the following NFRs for our system:

• Availability (SLA): The ingestion endpoint must be highly available. A common target for such critical infrastructure is 99.9% to 99.99% uptime. If our endpoint is down, we lose logs, which could be a major security incident.
• Latency: The endpoint must respond to Okta within 3 seconds. The Okta documentation explicitly states this is their timeout. A slow response will be treated as a failure, causing retries and potential data duplication. This single constraint strongly suggests an asynchronous processing architecture, where we acknowledge the request immediately and process the data later.
• Reliability & Data Durability:
  • No Data Loss: The system must be designed to prevent log loss. Once we send a 2xx response, the event must be durably stored.
  • At-Least-Once Delivery: Okta guarantees "at-least-once" delivery. This means our system must be idempotent—processing the same event multiple times should have the same result as processing it once. We will design for this in a later lesson.
• Scalability: The system must scale horizontally to handle high-throughput bursts of logs without affecting performance. While Okta has a daily rate limit, our architecture should not be the bottleneck.
• Security:
  • The endpoint must be exposed over HTTPS (TLS).
  • It must validate a shared secret passed in the Authorization header of every request from Okta. This prevents unauthorized clients from pushing fake logs to our system.

4. Defining the API Contract and Data Format

Now let's synthesize this information into a formal API contract. This contract specifies exactly what our endpoint expects and how it will respond.

API Contract: Ingesting Okta Event Logs

This contract is our guide for implementation. The 3-second latency requirement is a strong driver for the design shown in this sequence diagram:

Data Format

The data we receive will be in a JSON format defined by Okta. Understanding this schema is crucial for parsing and storing the logs correctly.

The Okta documentation provides a sample payload. The key elements are:

• A root object containing metadata about the hook itself.
• A data.events field, which is an array of one or more log events. This means our service must be prepared to handle batches of events in a single POST request.
• Each event in the array has a rich structure, including:
  • eventId: A unique ID for the event, which will be critical for implementing idempotency.
  • eventType: e.g., user.session.start. Useful for filtering and routing.
  • published: The timestamp of the event.
  • actor, client, target: Rich objects describing who did what, from where, and to what.

For now, the key takeaway is that we are receiving structured JSON. Internally, for efficiency, you might consider serializing this data into a more compact binary format like Protocol Buffers (Protobuf) before sending it to a message queue, but the public contract is JSON.

Conclusion

In this lesson, we established the foundational requirements for our log ingestion system. This is the blueprint that will guide all our subsequent design decisions.

Key Takeaways:

• System requirements are paramount: We distinguished between functional requirements (what the system does) and non-functional requirements (its quality attributes).
• The provider dictates the contract: Our system must conform to the API contract defined by Okta, including the endpoint behavior, security model (Authorization header), and data format (JSON).
• NFRs drive architecture: A single non-functional requirement—the 3-second response timeout—strongly pushes us toward an asynchronous design using a message queue to decouple ingestion from processing.
• Reliability is key: Okta promises "at-least-once" delivery, which means our system must be designed for idempotency to handle duplicate events without causing data corruption.

With this clear set of requirements, we are now ready to start thinking about the scale of the system.

Preview of the Next Lesson: In the next lesson, we will move on to capacity planning. We will learn how to perform back-of-the-envelope calculations to estimate the required write throughput and storage needs based on expected log volume. This will help us choose the right technologies and provision adequate resources for our system.