Creating Pandas Series and DataFrames

1. Why Pandas? From NumPy Arrays to DataFrames

You might be wondering: if we have NumPy, why do we need another library? NumPy arrays are fantastic for fast numerical computation, but they have two main limitations for data analysis:

1. Homogeneous Data: All elements in a NumPy array must be of the same data type. Real-world datasets often have columns of different types (e.g., a text column for names, a numeric column for age, a boolean for status).
2. Implicit Indexing: You access data using integer indices (array[0], array[5, 2]). It's up to you to remember that column 0 is 'age' and column 1 is 'income'. This becomes difficult with many columns.

Pandas is built on top of NumPy to solve exactly these problems. It introduces data structures with explicit labels for rows and columns, and it handles columns of different data types gracefully.

The following video provides an excellent introduction to why Pandas is necessary for practical data analysis.

2. The Core Data Structures: Series and DataFrame

Pandas has two main data structures you'll use constantly:

• Series: A one-dimensional labeled array, similar to a single column in a spreadsheet. It wraps a NumPy array and adds an index, which is a set of labels for each element.
• DataFrame: A two-dimensional labeled data structure with columns of potentially different types. You can think of it as a spreadsheet, a SQL table, or, most accurately, a dictionary of Series objects that all share the same index.

The next video segment clearly defines these two structures and their properties.

3. Creating a Pandas Series

A Series is the building block of a DataFrame. You create one using the pd.Series() constructor. The two most important arguments are data and index.

Here are the most common ways to create a Series:

From a Python List

If you only provide a list, Pandas automatically creates an integer index from 0.

import pandas as pd
import numpy as np

# A Series of student grades
grades = pd.Series([85, 92, 78, 95])
print(grades)

Output:

0    85
1    92
2    78
3    95
dtype: int64

You can also provide a custom index.

grades = pd.Series([85, 92, 78, 95], index=['Alice', 'Bob', 'Charlie', 'David'])
print(grades)

Output:

Alice      85
Bob        92
Charlie    78
David      95
dtype: int64

From a Python Dictionary

When you create a Series from a dictionary, Pandas uses the dictionary keys as the index and the values as the data. This is often very convenient.

student_ages = {'Alice': 21, 'Bob': 22, 'Charlie': 21}
age_series = pd.Series(student_ages)
print(age_series)

Output:

Alice      21
Bob        22
Charlie    21
dtype: int64

import pandas as pd

prices_dict = {
    "Data Science from Scratch": 25.50,
    "Python for Data Analysis": 30.00,
    "Deep Learning": 35.75
}

book_prices = pd.Series(prices_dict)
print(book_prices)

4. Creating a Pandas DataFrame

The DataFrame is the primary object you'll work with. It's a versatile structure that can be created from many different kinds of data. The following video gives a great practical overview of the most common methods.

Let's review the main in-memory methods shown in the video.

From a Dictionary of Lists (Column-Oriented)

This is arguably the most common way to create a DataFrame from scratch. Each key becomes a column name, and the list associated with it becomes the data for that column. Since you have a CS background, you can think of this as a hash map where keys are column labels and values are the column vectors.

data = {
    'Student': ['Alice', 'Bob', 'Charlie', 'David'],
    'Grade': [85, 92, 78, 95],
    'Major': ['CS', 'Math', 'CS', 'Physics']
}

df = pd.DataFrame(data)
print(df)

Output:

   Student  Grade    Major
0    Alice     85       CS
1      Bob     92     Math
2  Charlie     78       CS
3    David     95  Physics

From a List of Dictionaries (Row-Oriented)

This is another popular method, especially when your data is generated record by record (e.g., from a JSON file or an API). Each dictionary in the list represents a single row.

data = [
    {'Student': 'Alice', 'Grade': 85, 'Major': 'CS'},
    {'Student': 'Bob', 'Grade': 92, 'Major': 'Math'},
    {'Student': 'Charlie', 'Grade': 78, 'Major': 'CS'},
    # Notice this record is missing the 'Major'
    {'Student': 'Eve', 'Grade': 88} 
]

df = pd.DataFrame(data)
print(df)

Output:

   Student  Grade Major
0    Alice     85    CS
1      Bob     92  Math
2  Charlie     78    CS
3      Eve     88   NaN

Notice how Pandas automatically handles the missing 'Major' for Eve, filling it with NaN (Not a Number), its standard marker for missing data.

This image provides a great visual summary of the different ways to create a DataFrame from Python objects.

From External Files

In practice, you will most often create DataFrames by reading data from files. Pandas has highly optimized readers for various formats. The most common is pd.read_csv().

Let's assume you have a file named weather.csv with the following content:

day,temperature,windspeed,event
1/1/2017,32,6,Rain
1/2/2017,35,7,Sunny
1/3/2017,28,2,Snow

You can load it into a DataFrame with a single command:

# This assumes 'weather.csv' is in the same directory
# You would replace 'weather.csv' with the actual path to your file.
df_from_csv = pd.read_csv('weather.csv')
print(df_from_csv)

Output:

        day  temperature  windspeed  event
0  1/1/2017           32          6   Rain
1  1/2/2017           35          7  Sunny
2  1/3/2017           28          2   Snow

For more details on all the possible ways to create a DataFrame, you can skim the official documentation.

Conclusion

In this lesson, we made the transition from NumPy's array-centric world to Pandas' data-centric paradigm. You now have a solid foundation for the rest of this module and your future work in data science.

Key Takeaways:

• Pandas extends NumPy by adding explicit labels (indexes and columns) and the ability to handle columns with different data types.
• The Series is a 1D labeled array, representing a single column of data.
• The DataFrame is a 2D labeled table, representing an entire dataset. It can be seen as a collection of Series.
• You can create these objects from various in-memory structures like dictionaries and lists, with the choice of method often depending on whether your data is organized by columns or by rows.
• The most common real-world method for creating a DataFrame is by reading from a file, using functions like pd.read_csv().

Preview of the Next Lesson:
Now that you know how to create Series and DataFrame objects, the next logical step is to learn how to interact with them. In the next lesson, we will cover how to inspect the properties of a DataFrame (like its shape, data types, and first few rows) and how to select specific rows, columns, and individual cells using various indexing techniques.