Descriptive Statistics with Pandas

1. The Building Blocks of Data Summary

Descriptive statistics provide a simple summary of our data. Before we dive into the "how," let's get a formal introduction to these concepts.

2. Measures of Central Tendency: Finding the "Middle"

When you have a set of numbers, what is the most representative value? The answer isn't always straightforward. We'll look at two primary measures: the mean and the median.

The Mean (Average)

The mean is the most common measure of center. It's simply the sum of all values divided by the count of values. In our previous lesson, we called this the expected value $\mu$ for a theoretical distribution. When we calculate it from data, it's called the sample mean.

The Median (The Middle Value)

The median is the value that sits right in the middle of your data when it's sorted. 50% of the data is below the median, and 50% is above it.

Mean vs. Median: The Problem of Outliers

So why do we need both? The mean is very sensitive to extreme values, or outliers. The median is not. This makes the median a more robust statistic. Let's watch a video that explains this critical difference with a brilliant example.

Calculating Mean and Median in Pandas

Now that you understand the concepts, let's see how easy it is to calculate these in Python using Pandas. We'll load a dataset and apply the built-in functions.

To summarize, you can calculate the mean and median for a column (a Pandas Series) or for an entire DataFrame:

# Assuming 'df' is your DataFrame and 'column_name' is the column you're interested in
mean_value = df['column_name'].mean()
median_value = df['column_name'].median()

# You can also apply it to all numeric columns at once
all_means = df.mean()
all_medians = df.median()

3. Measures of Dispersion: Gauging the Spread

Knowing the center of the data isn't enough. We also need to know how spread out the data points are. Are they all clustered tightly around the mean, or are they all over the place? This is measured by variance and standard deviation.

• Variance: This is the average of the squared differences from the mean. A larger variance means the data is more spread out. The formula is ${\sigma }^2=\frac{\sum (x_i-\mu {)}^2}{N}$.
• Standard Deviation: This is simply the square root of the variance ($\sigma$). We often prefer it because it's in the same units as the original data, making it easier to interpret. For example, if you're measuring heights in centimeters, the standard deviation will also be in centimeters, while the variance would be in "centimeters squared".

Let's see how to calculate and interpret these measures.

The Pandas functions are just as straightforward:

# Calculate variance and standard deviation for a column
variance_value = df['column_name'].var()
std_dev_value = df['column_name'].std()

4. Putting It All Together: The .describe() Method

Calculating each of these statistics one by one is useful, but for a quick overview, Pandas provides a wonderfully convenient method: .describe(). This single command gives you a summary of the key descriptive statistics for all numerical columns in your DataFrame.

The .describe() method is one of the first things you'll run when you get a new dataset. It provides a wealth of information at a glance.

count    100.0
mean      35.0
std       10.0
min       20.0
25%       28.0
50%       34.0
75%       42.0
max       70.0

Conclusion

Excellent work! You've taken a major step from understanding theoretical probability to performing practical data analysis. You are now equipped to take any numerical dataset and generate a meaningful statistical summary.

Key Takeaways:

• Descriptive statistics summarize the main features of a dataset.
• Measures of Central Tendency like mean (average) and median (middle value) describe the "center" of the data. The median is more robust to outliers.
• Measures of Dispersion like variance and standard deviation describe the "spread" of the data. Standard deviation is often preferred for interpretation as it's in the original units.
• Pandas makes calculation easy with .mean(), .median(), .var(), and .std().
• The .describe() method is a powerful tool for getting a quick, comprehensive statistical summary of your numerical data.

Preview of the Next Lesson:

Reading numbers from a table is useful, but a picture is often worth a thousand words. In our next lesson, you will learn how to create and interpret histograms, box plots, and scatter plots. These visualizations will allow you to "see" the distribution of your data, making concepts like central tendency, dispersion, and outliers immediately obvious.