Designing Batch and Streaming Ingestion Pipelines

Batch vs. Streaming: Two Paradigms of Data Processing

At the highest level, data processing can be categorized into two main paradigms: batch processing and stream processing.

• Batch Processing operates on large, bounded volumes of data. The data is collected over a period (e.g., an hour, a day), and then processed all at once in a single "batch." This approach is optimized for high throughput and cost-efficiency.
• Stream Processing operates on unbounded streams of data in real time, or near-real time. Each data event is processed as it arrives, typically within seconds or milliseconds. This approach is optimized for low latency.

These two paradigms give rise to different architectural patterns. The image below provides a high-level visual comparison.

While Lambda and Kappa architectures offer ways to combine these paradigms, our focus today is on designing the fundamental ingestion pipelines for each type of source.

Designing Batch Ingestion Pipelines

Batch ingestion is the workhorse for many traditional data warehousing and ML model training scenarios. The goal is to periodically extract data from source systems, transform it into a usable format, and load it into a central repository for analysis. This process is commonly known as ETL (Extract, Transform, Load).

A modern batch ingestion pipeline often looks something like this:

Let's break down the key stages of this architecture.

1. Extract

This is the process of getting data out of the source systems. For a source like a production database, there are several common strategies:

• Full Load: The entire dataset is copied in every run. This is simple but inefficient and doesn't scale well.
• Incremental Load: Only the data that has changed since the last run is copied. This is typically done by tracking an updated_at timestamp in the source records.
• Change Data Capture (CDC): This is a more advanced technique where the pipeline subscribes to the database's transaction log (e.g., the write-ahead log in PostgreSQL) to capture every single insert, update, and delete event in real time and apply it to the destination.

The following video provides a clear explanation of the ETL process and the trade-offs between different loading strategies.

2. Load (to a Data Lake) and Transform

Once extracted, the raw data is typically loaded into a cheap, scalable object store like Amazon S3 or Google Cloud Storage, which serves as the foundation of a data lake. This initial copy is often referred to as the "Bronze" or "raw" layer in what is known as the Medallion Architecture.

From there, a distributed processing framework like Apache Spark reads the raw data, applies transformations (cleaning, validation, enrichment, joining), and writes the results back to the data lake in structured layers:

• Silver Layer: Contains cleaned, validated, and conformed data. This is where you might join tables and resolve data quality issues.
• Gold Layer: Contains business-level aggregations and features ready for analytics or model training. For example, a table of user-level weekly activity summaries.

This layered approach provides a clear lineage from raw to analysis-ready data, improves reusability, and isolates different stages of processing.

Designing Streaming Ingestion Pipelines

Streaming ingestion is essential for use cases that require real-time insights or features, such as fraud detection, real-time recommendations, and monitoring. The architecture is fundamentally different from batch processing, prioritizing continuous flow and low latency.

Core Components of a Streaming Pipeline

A typical streaming pipeline consists of several logical layers.

For our design purposes, the most critical components are:

1. Message Queue / Streaming Platform: Systems like Apache Kafka or AWS Kinesis act as the central nervous system. They provide a durable, scalable, and high-throughput "inbox" for incoming events. Producers (e.g., web servers, mobile apps) send events to a specific "topic" in Kafka, and consumers can subscribe to these topics to process the events asynchronously. This decouples the data producers from the data consumers.

2. Stream Processor: A stream processing engine like Apache Flink or Spark Streaming consumes events from the message queue. It allows you to perform stateful computations on the data streams, such as transformations, filtering, aggregations over time windows (e.g., "count clicks per user over the last 5 minutes"), and joining different streams.

Ingestion Strategies: Push vs. Pull

How do events get into the pipeline in the first place? There are two main models.

• Push Model: The data source (e.g., an application server) actively sends events to an endpoint, typically the message queue. This is common for event-driven systems where you want to react to events as they happen.
• Pull Model: The ingestion pipeline periodically queries an external source (e.g., a third-party API) to check for new data.

The mock interview video below discusses this trade-off in the context of designing a pipeline for Netflix clickstream data.

For most real-time ML use cases, the push model is preferred because it minimizes data collection latency.

A Practical Example: Real-time Feature Generation

Let's walk through a sophisticated example of a streaming pipeline designed for generating ML features in real-time. This is a common pattern in large-scale recommendation and personalization systems.

As you read, consider the flow:

1. A user action (e.g., a click) generates an event from a client or backend service.
2. The event is published as a message to a specific Kafka topic (e.g., user_post_clicks).
3. A Flink job subscribes to this topic.
4. The Flink job processes the stream of events, perhaps calculating a rolling 5-minute click count for each post (.window(SlidingEventTimeWindows.of(Time.minutes(5), Time.minutes(1)))).
5. The result of this aggregation (the new feature value) is written to a low-latency online feature store (e.g., Redis, Cassandra), from where it can be read by the model serving system.

This architecture enables models to make predictions using features that are only seconds old, a powerful capability for capturing user intent in real-time.

Key Design Considerations

When designing any data ingestion pipeline, you must balance several competing requirements. These are the trade-offs you should explicitly discuss in a system design interview.

• Latency vs. Throughput: Streaming pipelines offer low latency but may have lower overall throughput than a batch system optimized for processing massive amounts of data at once.
• Scalability: Can your system handle growth in data volume? Both Spark (for batch) and Kafka/Flink (for streaming) are designed for horizontal scaling by adding more nodes to a cluster.
• Reliability & Fault Tolerance: What happens if a component fails? Streaming platforms like Kafka offer replication for durability, and processing engines like Flink use checkpointing to save their state, allowing them to recover from failures without losing data or recalculating everything from scratch.
• Cost: Real-time processing is generally more expensive than batch processing due to the need for always-on compute resources. Storing raw data in a data lake like S3 is very cost-effective.
• Data Correctness: How do you handle out-of-order events or failures? Flink uses a concept called "watermarking" to handle late-arriving data gracefully. For batch, idempotency in transformations is key to ensure that re-running a failed job doesn't produce incorrect results.

Conclusion

In this lesson, we have explored the design of data ingestion pipelines for both batch and streaming sources. You now understand the fundamental architectural patterns and the key technologies that underpin them.

Here are the main takeaways:

• Batch vs. Streaming: Batch processing is for high-throughput, scheduled processing of bounded data (e.g., daily model training data). Streaming is for low-latency, continuous processing of unbounded data (e.g., real-time features).
• Batch Pipeline Architecture (ETL): This involves Extracting data from sources (full vs. incremental load), Loading it into a raw data lake (like S3), and Transforming it into analysis-ready formats using tools like Spark, often following a Medallion Architecture (Bronze, Silver, Gold).
• Streaming Pipeline Architecture: This typically uses a message queue like Kafka for scalable ingestion and buffering, coupled with a stream processor like Flink to perform real-time transformations and aggregations over time windows.
• Design Trade-offs: A robust design always involves a conscious discussion of trade-offs between latency, throughput, cost, scalability, and reliability.

In our next lesson, we will build directly on this foundation. Now that we know how to get data into our systems, we will compare the major architectural patterns for storing and managing that data at scale: data lakes, data warehouses, and the modern lakehouse.