Schema Inference with Auto Loader

1. Enabling Schema Inference with schemaLocation

As we briefly touched on previously, enabling schema inference in Auto Loader is surprisingly simple. You only need to provide one option in your stream definition: cloudFiles.schemaLocation.

(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", "/path/to/schema/location")  # This enables schema inference
  .load("/path/to/source/data")
  ...
)

When you provide this path, you're telling Auto Loader two things:

1. "I don't want to provide a schema manually; please figure it out for me."
2. "Store the schema you discover at this location, so you can track it over time."

Auto Loader then performs a one-time process to infer the schema. To learn how it does this, please read the following section from the official Databricks documentation.

As you just read, for formats without an encoded schema like JSON or CSV, Auto Loader defaults to inferring all columns as string data types. This is a safe default to prevent type-mismatch errors that could fail your stream (e.g., a "5" being treated as a number when it should be a string).

However, you often need more specific data types for downstream analysis, like integers, dates, or floating-point numbers. Let's explore how to achieve that.

2. Refining and Guiding Schema Inference

While the default string type is safe, it's not always useful. Auto Loader provides two key options to help you arrive at a more accurate schema: cloudFiles.inferColumnTypes and cloudFiles.schemaHints.

The following video provides an excellent conceptual overview and demonstration of these options.

Let's break down these two powerful options.

A. Automatic Type Inference: cloudFiles.inferColumnTypes

By setting cloudFiles.inferColumnTypes to true, you instruct Auto Loader to sample the data and make an educated guess about the data type of each column, instead of defaulting to string.

• Syntax: .option("cloudFiles.inferColumnTypes", "true")
• Behavior: It will analyze values in your columns and infer types like integer, double, timestamp, etc. This is very convenient but relies on your sample data being clean and consistent.

B. Providing Guidance: cloudFiles.schemaHints

This is often the most robust and practical approach. Schema hints allow you to enforce specific data types for columns you know about, while letting Auto Loader infer the rest. This gives you precise control where it matters, without needing to define the entire schema. The syntax uses a comma-separated string that looks similar to a SQL DDL statement.

• Syntax: .option("cloudFiles.schemaHints", "columnA_name DATE, columnB_name LONG, nested.columnC_name STRING")
• Behavior: Auto Loader will apply your specified types to the hinted columns. This is particularly useful for:
  • Overriding an incorrect inference (e.g., forcing an ID column to be a long instead of an integer).
  • Parsing dates and timestamps correctly.
  • Ensuring consistency for columns that might have mixed types in the source data.

The following short video provides a very clear, practical example of using schemaHints.

For more detailed examples, especially for complex or nested data, the Databricks documentation is an excellent resource.

3. Watching Schema Inference in Action

Now, let's watch a complete demo that puts all these pieces together. You will see how an Auto Loader stream is configured for schema inference and what the output—the stored schema file—actually looks like.

Check Your Understanding

Imagine you are ingesting JSON files with the following structure:

{
  "event_id": 9876543210987654321,
  "event_time": "2023-10-27T10:00:00.000Z",
  "temperature": "25.5",
  "is_critical": "false"
}

Your goals are:

1. event_id must be a long to avoid overflow.
2. event_time must be a timestamp.
3. temperature should be a double.
4. is_critical should be a boolean.

Which configuration would best achieve this?

schema_hints = "event_id LONG, event_time TIMESTAMP, temperature DOUBLE, is_critical BOOLEAN"

(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .option("cloudFiles.schemaLocation", "/path/to/schema")
  .option("cloudFiles.schemaHints", schema_hints)
  .load("/path/to/data")
  ...
)

Conclusion

In this lesson, you've learned how to leverage one of Auto Loader's most significant advantages: automatic schema inference. By understanding and using the right options, you can build robust ingestion pipelines that adapt to your data without manual intervention.

Key Takeaways

• Schema inference is enabled by setting the cloudFiles.schemaLocation option. Auto Loader samples the initial data and stores the inferred schema at this location.
• By default, for JSON and CSV files, all columns are inferred as string to ensure reliability.
• You can enable more intelligent type inference by setting cloudFiles.inferColumnTypes to true.
• For the most control and robustness, you can use cloudFiles.schemaHints to enforce specific data types on certain columns, letting Auto Loader handle the rest.

Next Steps

You now know how Auto Loader determines the initial schema. But what happens if new files arrive with different columns? This is known as schema evolution. In our next lesson, you will learn how to manage schema evolution in an Auto Loader stream to accommodate these changes gracefully.