Selecting Data in Pandas

What You'll Learn in This Lesson

This lesson focuses on the learning outcome: Read, filter, and select data from pandas DataFrames. Pandas provides two main data structures, the Series and the DataFrame, which allow you to work with labeled, tabular data—think of a spreadsheet or a database table, but with the full power of Python.

This is a fundamental skill for nearly all AI and machine learning tasks. In software testing, you might use pandas to analyze:

• Test execution results from a CSV file.
• Bug report data exported from a tracking system.
• API performance metrics logged over time.

We will cover:

• The basic structures: DataFrame (a table) and Series (a column).
• How to select specific columns of data.
• How to select specific rows and columns using .loc (label-based) and .iloc (position-based).
• How to filter your data based on single or multiple conditions to find exactly what you need.

Let's begin.

1. The Core of pandas: DataFrames and Series

At its heart, pandas is built on top of NumPy. A pandas DataFrame is a two-dimensional table with labeled rows and columns. Each column in a DataFrame is a pandas Series—a one-dimensional, labeled array.

You can think of a DataFrame as a dictionary of Series objects, where the keys are the column names and the values are the columns themselves. Given your programming experience, this mental model is a great starting point.

To get a solid grasp of these concepts and see how they're created, let's watch a section of a tutorial from Corey Schafer.

As the video shows, creating a DataFrame is straightforward. Let's create one that represents a small set of test results.

import pandas as pd




# Data for our test cases
test_data = {
    'test_id': ['TC-001', 'TC-002', 'TC-003', 'TC-004'],
    'component': ['Login', 'Login', 'Dashboard', 'Logout'],
    'status': ['Pass', 'Fail', 'Pass', 'Pass'],
    'exec_time_sec': [5.2, 7.8, 12.1, 3.5]
}




# Create the DataFrame
df = pd.DataFrame(test_data)




# Display the DataFrame
print(df)

Output:

  test_id  component status  exec_time_sec
0  TC-001      Login   Pass            5.2
1  TC-002      Login   Fail            7.8
2  TC-003  Dashboard   Pass           12.1
3  TC-004     Logout   Pass            3.5

This is our DataFrame. It's a structured, labeled table that's easy to read and, as we'll see, easy to query.

2. Selecting Subsets of Data

Often, you're not interested in the entire dataset at once. You might want to examine only the test statuses or look at the results of a specific test run. Pandas offers several ways to do this.

Selecting Columns

This is the most common selection you'll perform.

• To select a single column, use square brackets [] with the column name. This returns a Series.
• To select multiple columns, use double square brackets [[]] with a list of column names. This returns a DataFrame.

# Select the 'status' column (returns a Series)
statuses = df['status']
print("--- Status Column (Series) ---")
print(statuses)
print(f"\nType: {type(statuses)}")




# Select the 'test_id' and 'status' columns (returns a DataFrame)
test_outcomes = df[['test_id', 'status']]
print("\n--- Test ID and Status Columns (DataFrame) ---")
print(test_outcomes)
print(f"\nType: {type(test_outcomes)}")

Selecting Rows and Columns with .loc and .iloc

For more advanced selection, pandas provides two powerful indexers:

• .loc: Selects data by label. You use the row index labels and column names.
• .iloc: Selects data by integer position. You use the integer indices for both rows and columns (starting from 0).

The syntax for both is df.loc[rows, columns].

Let's continue with Corey Schafer's video to see how loc and iloc work in practice.

Let's apply this to our test data DataFrame:

# --- Using .loc (labels) ---




# Get the row with index label 1
test_case_row_1 = df.loc[1]
print("--- Row with index 1 ---")
print(test_case_row_1)




# Get rows 0 and 2, and only the 'component' and 'status' columns
specific_data = df.loc[[0, 2], ['component', 'status']]
print("\n--- Specific rows and columns with .loc ---")
print(specific_data)





# --- Using .iloc (integer positions) ---




# Get the first row (at position 0)
first_row = df.iloc[0]
print("\n--- First row with .iloc ---")
print(first_row)




# Get the first two rows (positions 0, 1) and the last two columns (positions 2, 3)
# Slicing works here: :2 means rows 0, 1. 2: means columns from position 2 to the end.
subset = df.iloc[:2, 2:]
print("\n--- Slice of rows and columns with .iloc ---")
print(subset)

3. Filtering Data with Conditions

Selecting specific rows is useful, but the real power comes from filtering data based on its content. For example, "show me all the failed tests" or "show me all tests that took longer than 10 seconds."

This is done using a technique called boolean masking. Here's how it works:

1. You write a condition (e.g., df['status'] == 'Fail').
2. Pandas evaluates this condition for every row, producing a Series of True and False values.
3. You pass this boolean Series back into the DataFrame inside square brackets [].
4. The DataFrame returns only the rows where the boolean Series had a True value.

The following video is entirely dedicated to filtering and is an excellent guide to mastering this crucial skill.

Let's put these techniques into practice with our test results DataFrame.

Single Condition

Find all tests that failed.

failed_tests = df[df['status'] == 'Fail']
print("--- Failed Tests ---")
print(failed_tests)

Multiple Conditions

Find all tests for the 'Login' component that passed. Remember to wrap each condition in parentheses (). Use & for AND and | for OR.

passed_login_tests = df[(df['component'] == 'Login') & (df['status'] == 'Pass')]
print("\n--- Passed Login Tests ---")
print(passed_login_tests)

Filtering with .isin()

Find all tests related to the UI ('Login' or 'Dashboard').

ui_components = ['Login', 'Dashboard']
ui_tests = df[df['component'].isin(ui_components)]
print("\n--- UI Component Tests ---")
print(ui_tests)

This is much cleaner than writing (df['component'] == 'Login') | (df['component'] == 'Dashboard').

To solidify your understanding, please take a few minutes to read through the official pandas tutorial on this topic. It provides concise examples and reinforces what you've just seen in the videos.

Conclusion

Excellent work today! You've learned the fundamental skills for exploring and querying data with pandas. Being able to slice, dice, and filter data is the first step in any data analysis or machine learning project. These techniques will be your bread and butter as we move forward.

Key Takeaways

• DataFrame and Series: The two primary data structures in pandas. A DataFrame is a 2D table, and each of its columns is a Series.
• Column Selection: Use single brackets df['col'] for a Series and double brackets df[['col1', 'col2']] for a DataFrame.
• Label vs. Position Selection: Use .loc for label-based selection (e.g., df.loc[0, 'status']) and .iloc for integer position-based selection (e.g., df.iloc[0, 2]).
• Boolean Masking: The core of filtering. Create a Series of True/False values from a condition and use it to select rows from a DataFrame.
• Combining Filters: Use & (AND) and | (OR) with parentheses to create complex filters. Methods like .isin() and .str.contains() provide powerful shortcuts.

What's Next?

We now know how to work with data once it's inside a pandas DataFrame. But how do we get it there in the first place? In the next lesson, we will cover how to read and write data using common file formats like CSV and JSON. This will bridge the gap between your raw test data and the powerful analytical capabilities of pandas.