Auto Loader for New Files with `cloudFiles`

1. The Core Syntax: Introducing cloudFiles

At its heart, Auto Loader is a specialized source for Spark Structured Streaming. This is great news because it means the syntax builds on concepts you're already familiar with, like spark.readStream and spark.writeStream.

The key to invoking Auto Loader is to specify "cloudFiles" as the format in your streaming read.

df = spark.readStream.format("cloudFiles")

However, telling Spark to use Auto Loader isn't enough. Auto Loader itself needs to know the format of the underlying files it will be processing (e.g., CSV, JSON, Parquet). We provide this information using a specific option: cloudFiles.format.

Here is the fundamental structure of an Auto Loader streaming read:

# The basic pattern for an Auto Loader stream
streaming_df = spark.readStream \
  .format("cloudFiles") \
  .option("cloudFiles.format", "json") \
  .load("/path/to/your/data")

Let's break this down:

• .format("cloudFiles"): This tells Spark to use the Auto Loader source.
• .option("cloudFiles.format", "json"): This configures Auto Loader to expect JSON files. You would change "json" to "csv", "parquet", etc., based on your source data.
• .load("/path/to/your/data"): This points to the cloud storage directory that Auto Loader will monitor for new files.

2. Building Your First Auto Loader Stream: A Walkthrough

Now, let's see how these pieces fit together in a complete, working example. The following video provides a practical, step-by-step guide to setting up and running an Auto Loader pipeline. It will show you how to configure both the read stream and the write stream to a Delta table.

After watching the video, you've seen a complete end-to-end example. Let's solidify that knowledge by breaking down the most important configuration options you just saw.

3. Essential Auto Loader Configuration Options

The video demonstrated several key options. Understanding them is crucial for configuring Auto Loader correctly and is a common focus on the certification exam.

Here is a more detailed look at a complete, well-structured Auto Loader configuration, similar to what you saw in the video.

Based on the video and the reading, let's summarize the must-know options for this lesson's learning outcome:

• format("cloudFiles"): As we've discussed, this is non-negotiable. It's what activates Auto Loader.

• cloudFiles.format: (Required) Specifies the format of your source files (e.g., "csv", "json", "parquet", "avro"). Auto Loader needs this to parse the files correctly.

  .option("cloudFiles.format", "csv")
  

• load(input_data_path): (Required) The cloud storage path where your raw files are located. Auto Loader will monitor this directory.

• cloudFiles.schemaLocation: (Required for schema inference) This is a critical option. You provide a cloud storage path where Auto Loader will store and track the data's schema. This allows it to detect changes over time and prevents you from having to define the schema manually. We'll dive deeper into this in the next lesson, but for now, know that it's a required and essential configuration.

  .option("cloudFiles.schemaLocation", checkpoint_dir_path)
  

• checkpointLocation: (Required for streaming writes) When you write the data using .writeStream, you must provide a checkpoint location. Spark Structured Streaming uses this path to save the state of the stream—specifically, which files have been processed. This is how Auto Loader guarantees exactly-once processing, ensuring that files are neither missed nor duplicated, even if the stream stops and restarts.

  .writeStream \
  .option("checkpointLocation", checkpoint_dir_path)
  

  Note: While cloudFiles.schemaLocation and checkpointLocation can point to the same directory, it's a best practice to use distinct subdirectories (e.g., /path/to/_checkpoints/schema and /path/to/_checkpoints/stream_state) for clarity and better organization.

By combining these options, you provide Auto Loader with all the information it needs: where to find files, what format they're in, where to manage their schema, and where to track its own progress.

Conclusion

In this lesson, you made the leap from theory to practice. You learned how to configure a basic but powerful Auto Loader stream to incrementally read new files from a directory.

Key Takeaways

• Auto Loader is invoked using spark.readStream.format("cloudFiles").
• You must configure Auto Loader with the format of the source files using the cloudFiles.format option.
• Two paths are essential for a reliable stream:
  1. cloudFiles.schemaLocation: Tells Auto Loader where to store and manage the data schema.
  2. checkpointLocation: Tells the write stream where to store its progress to ensure exactly-once processing.
• The basic configuration involves chaining spark.readStream, .format(), .option(), and .load() to read, and .writeStream, .option(), and .toTable() or .start() to write.

Next Steps

You've now seen the cloudFiles.schemaLocation option and learned its general purpose. In our next lesson, we will explore this feature in detail. You will learn how to configure Auto Loader to automatically infer the schema of streaming data from your source files, one of Auto Loader's most powerful capabilities.