Filtering Rows and Creating Columns in Pandas DataFrames

Hello! Last lesson, you used NumPy to adjust and summarize whole arrays without looping over individual values. This time, you’ll work with pandas DataFrames: tables whose columns have names such as input_tokens and latency_ms. You’ll derive useful columns from existing ones, then filter the table to the rows relevant to a question. These are everyday steps in inspecting AI application data. Allow about 40 minutes, including time to run the code.

Start with a labeled table

Imagine a small log of AI inference requests. Each row is one request; each column records a property of it. Activate your project’s .venv, then install pandas and update the pinned dependencies in requirements.txt:

python -m pip install pandas
python -m pip freeze > requirements.txt

Create run_pandas_demo.py in your project’s outer directory and start with:

import pandas as pd


requests = pd.DataFrame(
    [
        [101, "small", "ok", 120, 30, 800],
        [102, "large", "ok", 500, 100, 2100],
        [103, "small", "error", 260, 0, 500],
        [104, "small", "ok", 400, 130, 1800],
        [105, "large", "ok", 80, 25, None],
        [106, "experimental", "ok", 200, 60, 900],
    ],
    columns=[
        "request_id",
        "model",
        "status",
        "input_tokens",
        "output_tokens",
        "latency_ms",
    ],
)

print(requests)
print(requests.shape)  # (6, 6)

Run it with python run_pandas_demo.py. As with a two-dimensional NumPy array, .shape gives the row and column counts. Unlike a plain array, the column names let you express what a calculation means: requests["input_tokens"] selects one named column. That result is a one-dimensional pandas Series; requests[["input_tokens", "output_tokens"]] selects two columns and returns a DataFrame.

The final row has model "experimental", while request 105 has no recorded latency. Both details will matter when you decide which rows to keep.

Derive columns without a row-by-row loop

Suppose you want total token usage and latency in seconds. The pandas documentation’s How to create new columns derived from existing columns shows the same pattern with air-quality measurements. Read the London conversion example and the subsequent Paris-to-Antwerp ratio; the later renaming examples are not needed here.

Add these lines below the DataFrame construction in your script:

requests["total_tokens"] = (
    requests["input_tokens"] + requests["output_tokens"]
)
requests["latency_seconds"] = requests["latency_ms"] / 1000

print(requests[["request_id", "total_tokens", "latency_seconds"]])

The first assignment adds corresponding input and output counts: request 104 has 400 + 130, or 530 total tokens. The second divides every recorded millisecond value by 1000: its 1800 ms becomes 1.8 seconds. You write each calculation once, not once per row.

For request 105, the missing latency remains missing in latency_seconds; dividing it does not invent a measurement. That is a useful distinction when you later select requests whose latency can actually be inspected.

Turn a condition into a row filter

A comparison on a pandas column does not immediately return rows. It first produces a Series of True and False values—one decision for each row. Watch Corey Schafer’s Python Pandas Tutorial (Part 4): Filtering - Using Conditionals to Filter Rows and Columns for that mechanism and for .loc, which selects rows and columns together.

Try a single condition in your script, after creating the derived columns:

high_volume = requests["total_tokens"] >= 500

print(high_volume)
print(requests.loc[high_volume, ["request_id", "total_tokens"]])

high_volume is a boolean Series. Passing it before the comma in .loc[rows, columns] keeps only its True rows; the list after the comma limits the displayed columns. You should see requests 102 and 104, with 600 and 530 total tokens. Their leftmost DataFrame index labels stay as they were; request_id is a separate column.

You may also encounter requests[high_volume], which filters rows while keeping all columns. Use .loc when you want to state both the row condition and the columns of interest.

Combine conditions to answer a specific question

Now ask: Which successful requests used the small or experimental model, consumed at least 200 tokens, and have a recorded latency? This requires more than one comparison. Pandas uses & for elementwise and and | for elementwise or; put parentheses around each condition. Python’s ordinary and and or do not combine boolean Series this way.

The pandas subset tutorial adds two particularly useful conditions: .isin() tests membership in a list, and .notna() identifies recorded rather than missing values.

Replace the simple filter’s purpose with the full question by adding this code below it:

usable = (
    requests["model"].isin(["small", "experimental"])
    & (requests["status"] == "ok")
    & (requests["total_tokens"] >= 200)
    & requests["latency_ms"].notna()
)

selected = requests.loc[
    usable,
    ["request_id", "model", "total_tokens", "latency_seconds"],
]
print(selected)

Each part answers one requirement. .isin(...) accepts either of the two model names. The status and token comparisons narrow the set further. .notna() rules out a request without a latency, so the resulting latency_seconds values are usable. The result contains requests 104 and 106: their total token counts are 530 and 260, and their latencies are 1.8 and 0.9 seconds.

Notice why request 103 does not appear: it meets the model and token conditions and has a latency, but its status is "error". A filter is most trustworthy when you can explain both an included row and an excluded one. If your question instead allowed either of two separate conditions, you would join their parenthesized boolean Series with |.

Here you derived columns on the original DataFrame before filtering it. That keeps the code straightforward and avoids assigning into an ambiguously selected slice of a table. In a larger project, this same sequence can live inside a reusable function.

Takeaways

A pandas column is a Series, so arithmetic and comparisons operate across its rows without an explicit loop. Assign an arithmetic result to dataframe["new_column"] to derive a column; pass a boolean condition to .loc[condition, columns] to select the rows and fields you need. For compound filters, use parenthesized conditions with & or |, and make an explicit choice about missing values.

Next, you’ll use pandas to aggregate rows into summaries and merge related tables. The filtered, derived request table you built here is a useful starting point for both.