Pandas Data Selection with `loc`, `iloc`, and Boolean Indexing

1. The Building Blocks: Series and DataFrames

Before we can select data, we need to understand what we're selecting from. Pandas has two primary data structures:

• Series: A one-dimensional array-like object that can hold any data type. It's essentially a column of data with an associated index, which provides labels for each element.
• DataFrame: A two-dimensional, size-mutable, tabular data structure with labeled axes (rows and columns). You can think of a DataFrame as a dictionary of Series objects, where each Series is a column. This structure is analogous to a spreadsheet or an SQL table.

The following video provides an excellent introduction to these concepts and shows how you can construct a DataFrame from scratch, which should feel familiar given your programming experience with dictionaries.

As you saw, a DataFrame is an intuitive object for holding tabular data, and selecting one or more columns is a straightforward first step.

2. Selection by Position and Label: iloc vs. loc

Pandas offers two primary methods for more advanced data selection, and understanding their distinction is crucial:

• .iloc (Integer Location): Selects data based on its integer position (from 0 to length-1 of the axis). It behaves just like standard Python list slicing.
• .loc (Location): Selects data based on its label. This uses the actual index values and column names.

This is a critical distinction: iloc cares about the position (is it the 1st, 2nd, 50th row?), while loc cares about the label (is the index label 'USA', '2023-01-01', or 49?).

The video you started watching provides a great walkthrough of both.

To reinforce these concepts and have a handy reference, the image below provides a great summary.

For a more detailed textual explanation that you can refer back to, the following article is an excellent resource.

3. Filtering with Boolean Indexing

The most powerful and common method for selecting data is boolean indexing. This is conceptually similar to using a WHERE clause in SQL. The process involves creating a "mask"—a Series of True/False values—and then using that mask to filter the DataFrame.

The steps are:

1. Write a conditional expression that evaluates to True or False for each row (e.g., df['age'] > 30). This produces the boolean mask.
2. Pass this mask to .loc to select only the rows where the mask is True (e.g., df.loc[df['age'] > 30]).

You can combine multiple conditions using the following bitwise operators:

• & for AND
• | for OR
• ~ for NOT

Important: Each condition must be wrapped in parentheses due to Python's operator precedence rules.

This next video demonstrates filtering from basic principles to more complex, real-world examples.

Your Turn: Practice Selection

Let's put this all together. Here is a small DataFrame. Your task is to write the code to answer each of the questions below.

import pandas as pd

data = {'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eva'],
        'year': [2020, 2021, 2020, 2022, 2021],
        'role': ['Engineer', 'Analyst', 'Engineer', 'Manager', 'Analyst'],
        'salary': [80000, 65000, 82000, 105000, 70000]}
df = pd.DataFrame(data, index=['ID1', 'ID2', 'ID3', 'ID4', 'ID5'])

print("Original DataFrame:")
print(df)
print("-" * 30)




# 1. Select the 'name' and 'salary' for all rows.
name_and_salary = df.loc[:, ['name', 'salary']]
print("\n1. Name and Salary:")
print(name_and_salary)
print("-" * 30)




# 2. Select the first two rows of the DataFrame using their integer position.
first_two_rows = df.iloc[0:2]
print("\n2. First two rows (iloc):")
print(first_two_rows)
print("-" * 30)




# 3. Select the row with the index label 'ID4'.
row_id4 = df.loc['ID4']
print("\n3. Row with index 'ID4' (loc):")
print(row_id4)
print("-" * 30)




# 4. Select all Engineers who were hired in 2020.
engineers_2020 = df.loc[(df['role'] == 'Engineer') & (df['year'] == 2020)]
print("\n4. Engineers from 2020:")
print(engineers_2020)
print("-" * 30)




# 5. Select all Analysts OR anyone with a salary greater than $100,000.
analysts_or_high_salary = df.loc[(df['role'] == 'Analyst') | (df['salary'] > 100000)]
print("\n5. Analysts or high earners:")
print(analysts_or_high_salary)
print("-" * 30)

Try to predict the output of each step before running the code. Experiment by changing the conditions to solidify your understanding.

Conclusion

In this lesson, you've learned the fundamental techniques for selecting and filtering data in Pandas, a skill you will use in every single data analysis project.

Key Takeaways:

• A Series is a 1D labeled array (a column), and a DataFrame is a 2D labeled structure (a table).
• Use .iloc to select data by its integer position, which is useful for tasks like grabbing the first or last N rows.
• Use .loc to select data by its index and column labels, which is more explicit and often more readable.
• Boolean indexing is the most powerful method for filtering rows based on data content, using conditional expressions and operators & (AND), | (OR), and ~ (NOT).

These selection methods are the foundation upon which more complex data manipulations are built. In our next lesson, we will learn how to read data from common file formats like CSVs, which will allow us to start working with larger, real-world datasets. We will also cover how to handle missing data and perform initial data inspection—essential first steps in any analysis pipeline.