Reading Streaming Data with Spark Structured Streaming

1. The Structured Streaming Model

At its core, Spark Structured Streaming provides a simple but powerful abstraction: it treats a live data stream as a table that is continuously being appended. This allows you to use the familiar Spark DataFrame API—which you already know from batch processing—to perform transformations on streaming data.

This concept is key. Instead of learning a new, complex API for streaming, you can leverage your existing skills in writing queries and transformations. The Spark engine handles the complexity of running these queries incrementally and continuously on new data as it arrives.

This model can be visualized as running a batch-like query repeatedly on new data.

2. The Core API: spark.readStream

The entry point for all streaming queries is spark.readStream. This should feel very familiar, as it's the direct counterpart to spark.read which you use for batch jobs. The methods for defining the source format, options, and schema are nearly identical.

# Batch read
batch_df = spark.read.format("json").load("/path/to/data")




# Streaming read
streaming_df = spark.readStream.format("json").load("/path/to/data")

The key difference is that spark.readStream returns a streaming DataFrame. You can apply most of the same transformations to it (select, filter, withColumn, join, etc.), but the execution is triggered differently via writeStream, which we will cover in the next lesson.

3. Reading from Common Streaming Sources

As a data engineer, you will encounter data streams from various systems. Let's explore how to read from the three most common sources on Databricks.

A. File-Based Streams (and Auto Loader)

A very common pattern is to process files as they land in cloud storage (like S3 or ADLS Gen2). While you can use the basic spark.readStream.format("json").load(path) to monitor a directory, Databricks provides a much more powerful and optimized tool for this purpose: Auto Loader.

Auto Loader is specifically designed to incrementally and efficiently process new data files as they arrive in cloud storage. It's the recommended method for file ingestion on Databricks because it's more scalable and feature-rich than the basic file stream.

A typical pipeline using Auto Loader looks like this:

To see this in a more practical, end-to-end context, the following video demonstrates setting up a stream reader for JSON files arriving in a folder.

B. Delta Lake Streams

Another powerful pattern in the Lakehouse is to use a Delta table as a streaming source. This allows you to build reactive pipelines that trigger downstream actions whenever a table is updated. For example, you could have a pipeline that automatically refreshes an aggregated dashboard table whenever new data arrives in a silver-layer table.

A cleaner, more modern syntax for this is to use .table():

# Reads from the Delta table named 'my_delta_table'
stream_df = spark.readStream.table("my_delta_table")

Delta streaming also offers powerful options to control the starting point of the stream. For example, you can process all changes since a specific table version.

C. Message Bus Streams (e.g., Kafka)

For very low-latency, high-throughput scenarios, data is often streamed through a message bus like Apache Kafka or Azure Event Hubs. Structured Streaming has built-in connectors for these systems. While we won't do a deep dive, it's important to see that the pattern remains consistent.

4. The Critical Role of Checkpointing

You've seen checkpointLocation or checkpointPath mentioned in every example. This is arguably the most important concept for building reliable, fault-tolerant streaming pipelines.

A checkpoint is a location on durable storage (like S3) where a streaming query saves its progress. This "progress" includes:

• Source Tracking: Which files (for a file source) or offsets (for a Kafka source) have already been processed.
• Stateful Operations: Intermediate state for operations like streaming aggregations (which we'll cover later).

Without a checkpoint, if your stream stops for any reason (e.g., cluster shutdown, failure), it has no memory of where it left off. When restarted, it would either reprocess all data from the beginning or miss data. The checkpoint ensures exactly-once processing guarantees.

Every production streaming query must have a unique checkpoint location.

Conclusion

In this lesson, you have learned the foundational concepts of reading data with Spark Structured Streaming. You are now equipped to start building the ingestion part of any streaming data pipeline on Databricks.

Key Takeaways:

• Structured Streaming lets you use the familiar DataFrame API on streaming data by treating a stream as an unbounded table.
• spark.readStream is the entry point for creating a streaming DataFrame from a source.
• For file-based ingestion, Auto Loader (.format("cloudFiles")) is the recommended, high-performance method.
• You can stream directly from Delta tables, enabling reactive data pipelines within the lakehouse.
• A checkpoint location is mandatory for all production streams to ensure fault tolerance and prevent data loss or duplication.

Preview of the Next Lesson:
Now that you can read a stream, the next logical step is to write it to a destination, or a "sink". In our next lesson, we will focus on writeStream and explore the different output modes (append, complete, update) that determine how the results of your streaming query are written to the sink.