Adding and Modifying DataFrame Columns

1. Adding New Columns to Your DataFrame

One of the most common tasks in data analysis is adding new columns. This can be done by combining existing columns, splitting a single column into multiple parts, or applying conditional logic.

1.1 Creating Columns from Existing Data

The most straightforward way to add a new column is to assign it the result of an operation on one or more existing columns.

Let's start with a simple example. The following video demonstrates how to combine first and last name columns into a new full_name column.

As the video shows, the syntax is simple: df['new_column_name'] = ...operation...

Now, let's apply this to a business context from your AdTech background. A key performance indicator (KPI) for online advertising is the Click-Through Rate (CTR), which measures how often people who see your ad end up clicking it. It's calculated as:

$\text{CTR}=\left(\frac{\text{Clicks}}{\text{Impressions}}\right)\times 100$

Let's see how to add CTR and Conversion Rate as new columns to a campaign performance DataFrame.

This direct assignment is powerful and efficient. You can use it for all standard arithmetic operations (+, -, *, /) between columns.

1.2 Splitting One Column into Many

Sometimes, a single column contains multiple pieces of information you want to analyze separately. A common example is splitting a full name into first and last names, or a date-time stamp into date and time components.

Pandas' string methods (.str) make this easy. You can use .str.split() to break up a string and the expand=True argument to automatically create new columns from the results.

1.3 Creating Columns with Conditional Logic

Often, the value in a new column depends on a condition. For instance, you might want to categorize users as 'High Value' or 'Low Value' based on their spending, or label leads as 'Hot' or 'Cold' based on their activity.

The best way to do this in pandas is with .loc and a boolean condition. The syntax is:
df.loc[condition, 'new_column_name'] = 'value_if_true'

The following resource shows how to create a 'position label' column by bucketing the 'Average Position' of a search query.

2. Modifying Existing Columns

Besides adding new columns, you'll frequently need to modify existing ones. This includes renaming them for clarity or transforming their values for consistency.

2.1 Renaming Columns

Clear and consistent column names are essential for readable code and clear reports. The df.rename() method is your tool for this.

The following video explains how to use df.rename() to change specific column names, and also how to modify all column names at once (e.g., converting all to lowercase).

2.2 Transforming Column Values

Modifying the actual data within a column is a core part of data cleaning and preparation. Pandas offers several methods for this, each suited for different tasks.

Vectorized String Methods (.str)

For string columns, always try to use vectorized string methods first. They are fast and efficient. We already saw .str.split(), but others like .str.lower(), .str.upper(), .str.replace(), and .str.strip() are extremely useful for standardizing text data.

This clip shows how to apply .str.lower() to an entire email column at once.

The apply, map, and replace Methods

For more complex transformations that can't be handled by simple arithmetic or built-in string methods, pandas provides apply, map, and replace. Understanding the difference is key to using them correctly.

This next video segment is crucial. It gives a detailed breakdown of these four methods (apply, applymap, map, and replace).

Here's a quick summary to help you keep them straight:

status_map = {'Active': True, 'Inactive': False, 'Pending': False}
df['is_active'] = df['Status'].map(status_map)

df['is_active'] = df['Status'].apply(lambda x: x == 'Active')

3. A Crucial Warning: The SettingWithCopyWarning

When you start modifying DataFrames, you will inevitably encounter the SettingWithCopyWarning. This warning appears when you try to modify a "copy" of a DataFrame slice instead of the original DataFrame itself. It can be confusing, and ignoring it often means your changes aren't being saved correctly.

The most common cause is "chained indexing" during an assignment, like df[df['column'] > 5]['other_column'] = 'new_value'.

The video you just watched has an excellent section explaining this problem and demonstrating the correct solution: always use .loc for assignments that involve both row and column selection.

Conclusion

You've now learned the essential techniques for enriching and transforming your data in pandas. By adding and modifying columns, you can turn raw data into a powerful analytical dataset ready to produce business insights.

Key Takeaways:

• Add columns using direct assignment (df['new'] = df['a'] + df['b']), .str.split(expand=True), or conditional logic with df.loc[...].
• Modify columns by renaming them with df.rename() or transforming their values.
• For value transformation, prefer vectorized methods (.str, arithmetic) for speed. Use .map() for substitutions and .apply() for complex, custom functions.
• Avoid the SettingWithCopyWarning by using .loc for any assignment that filters rows and selects columns simultaneously.

Preview of the Next Lesson:
We've cleaned our data, and now we've enriched it at the row level. The next logical step is to start summarizing it. In our next lesson, we will dive into one of the most powerful features of pandas: performing data aggregation and summarization using groupby operations. This will allow us to calculate metrics across different segments and categories, a core activity for any Business Analyst.