Loading Data into DataFrames with PySpark

1. The SparkSession and DataFrame

Before we can read any data, we need to understand two core concepts: the SparkSession and the DataFrame.

In a Databricks notebook, a SparkSession object, typically named spark, is automatically created for you. This object is your entry point to all of Spark's functionality. It's what you'll use to read data, execute queries, and configure your Spark application.

To understand what happens when you create this session, please read the short section below.

Once you have a spark session, the primary data structure you will work with is the DataFrame.

So, what exactly is a DataFrame? The Databricks documentation provides a succinct definition.

In essence, a DataFrame is like a table in a relational database or a pandas DataFrame, but with a crucial difference: it is distributed across multiple machines in a cluster. This distributed nature is what allows Spark to process massive datasets in parallel. Given your background in physics, you can think of it as a distributed matrix where columns can have different data types, optimized for large-scale computations.

2. Reading CSV Files

CSV (Comma-Separated Values) is one of the most common file formats you'll encounter. PySpark provides a powerful and flexible reader for CSV files.

The general syntax is spark.read.csv("path/to/file.csv"). However, you'll almost always need to add options to tell Spark how to correctly interpret the file.

This video provides an excellent walkthrough of reading CSV files with the most important options.

As you saw, the two most vital options are:

• header=True: Tells Spark to use the first row of the CSV file as the column names for the DataFrame.
• inferSchema=True: Instructs Spark to make an extra pass over the data to automatically determine the data type of each column (e.g., integer, string, date).

Hands-On: Reading a CSV File

Let's apply this with a hands-on exercise from the Databricks documentation. You will set up a path in a Unity Catalog Volume, download a public dataset into it, and then read it into a DataFrame.

Open a new Databricks notebook and follow these steps.

The code you ran in Step 3 should look like this:

df_csv = spark.read.csv(f"{path_volume}/{file_name}",
  header=True,
  inferSchema=True,
  sep=",")

display(df_csv)

This is the fundamental pattern for reading a well-formatted CSV file.

Advanced Reading: Multiple Files and Custom Schemas

In real-world scenarios, you often need to read many files at once or deal with data that requires a specific schema.

• Reading multiple files: You can read all CSV files in a folder by simply providing the folder path instead of a file path. spark.read.csv("path/to/folder/"). You can also use wildcards, like spark.read.csv("path/to/folder/*.csv").
• Defining a schema: While inferSchema=True is convenient, it can be slow and sometimes guess the wrong data types. For production jobs, it's a best practice to define the schema explicitly. This improves performance (Spark doesn't need to read the data twice) and ensures data quality.

This next segment of the video demonstrates these two important concepts.

3. Reading JSON Files

In our last lesson, we parsed JSON strings that were already inside a table. Now, we'll learn to read entire JSON files directly into a DataFrame.

The syntax is very similar to the CSV reader: spark.read.json("path/to/file.json"). JSON files are semi-structured and self-describing, so Spark can usually infer the schema without extra options. However, there's one crucial option you need to know about: multiLine.

This video will guide you through reading both single-line and multi-line JSON files.

Key points for reading JSON:

• By default, Spark assumes each line is a separate, self-contained JSON object (multiLine=False).
• If a single JSON object is formatted across multiple lines (pretty-printed), you must use multiLine=True.
• Similar to CSVs, you can pass a directory path or use wildcards to read many JSON files at once.
• You can also provide an explicit schema using StructType and StructField to enforce data types, which is still a best practice for production pipelines.

4. Reading Parquet Files

Finally, let's talk about Parquet. Apache Parquet is a columnar storage format, optimized for use with data processing frameworks like Spark. Unlike row-based formats like CSV, Parquet stores data by column, which leads to significant performance improvements for analytical queries that often only access a subset of columns.

Key advantages of Parquet:

• Performance: Columnar storage allows Spark to read only the columns needed for a query, drastically reducing I/O.
• Compression: Parquet has highly efficient compression algorithms, reducing storage costs.
• Schema-aware: Parquet files store the schema within the data itself. This means you never need to use inferSchema or define a schema manually when reading.

Reading a Parquet file is the simplest of all.

The syntax is simply spark.read.parquet("path/to/file.parquet"). Because the schema is embedded, Spark knows the column names and data types automatically. For these reasons, Parquet is the preferred file format for storing intermediate or final datasets in most Databricks workflows, especially for the Silver and Gold layers of the medallion architecture.

Conclusion

You have just learned the foundational skill of data ingestion in PySpark. Being able to correctly and efficiently read data from various file formats is the first step in building any reliable data pipeline.

Key Takeaways

• The SparkSession (available as the spark object) is your entry point to using Spark.
• DataFrames are distributed tables that form the core of working with PySpark.
• Use spark.read.csv() with header=True for files with headers and inferSchema=True for quick exploration.
• Use spark.read.json() and remember to set multiLine=True if your JSON objects span multiple lines.
• Parquet is the most efficient format for Spark. Use spark.read.parquet() for fast, schema-aware data loading.
• For robust production pipelines, always prefer to define your schema explicitly using StructType and StructField when reading CSV or JSON files.

Next Steps

Now that we know how to load data into a DataFrame, the next logical step is to learn how to manipulate it. In our next lesson, we will explore how to select, rename, and drop columns in a DataFrame using the PySpark API, starting you on the path to performing complex data transformations programmatically.