Initial Data Inspection with Pandas

1. The DataFrame's "ID Card": .info()

Before you dive into any analysis, you need to understand the basic structure of your data. What are the columns? What kind of data does each column hold? How much data is missing? The .info() method provides a concise summary that answers all these questions at once.

It gives you a high-level overview, including:

• The number of entries (rows) and the index range.
• A list of all columns.
• The count of non-null values for each column (a great first check for missing data, tying back to our last lesson).
• The data type (Dtype) of each column (e.g., int64, float64, object, datetime64).
• The memory usage of the DataFrame.

This guide from Deepnote provides an excellent summary of these initial inspection commands.

To see .info() in a real-world context, the following video demonstrates its use as one of the first steps in an exploratory data analysis workflow.

2. Statistical Summary for Numerical Data: .describe()

Once you understand the structure, you'll want to understand the content. For numerical columns, the .describe() method is invaluable. It instantly computes several key descriptive statistics, giving you a sense of the data's distribution and scale.

The output of .describe() includes:

• count: The number of non-null values.
• mean: The average value.
• std: The standard deviation, which measures the data's spread or dispersion.
• min and max: The minimum and maximum values. These are useful for spotting potential outliers or data entry errors.
• 25%, 50%, 75%: The quartiles. The 50th percentile is the median, which is often a more robust measure of central tendency than the mean when outliers are present.

Let's see this in action.

An interesting feature of .describe() is that it behaves differently when called on categorical (or object) data. Instead of mean and standard deviation, it will provide the count, the number of unique values, the top (most frequent) value, and the freq (frequency of the top value).

The official Pandas documentation provides excellent, runnable examples for these summary statistics.

3. Frequency Counts for Categorical Data: .value_counts()

While .describe() is perfect for numbers, it's not very useful for categorical columns like 'Country', 'Product Category', or 'Gender'. For these, you need to know which categories exist and how often each one appears. This is exactly what the .value_counts() method does.

Unlike .info() and .describe(), which are DataFrame methods, .value_counts() is a Series method. This means you call it on a single column at a time: df['your_column'].value_counts().

This visual breaks down the concept perfectly.

This image shows how value_counts() takes a column of categorical data and transforms it into a summary table, counting the occurrences of each unique value.

The syntax is straightforward.

This image breaks down the syntax for calling the .value_counts() method on a Pandas Series.

A very useful parameter for this method is normalize=True, which returns the relative frequencies (percentages) instead of raw counts. This is great for understanding the proportion of each category in your data.

This video provides a great walkthrough of value_counts() and its common uses.

Let's try this with some code.

import pandas as pd

# Let's use the Titanic dataset, a classic for data analysis
df = pd.read_csv('https://raw.githubusercontent.com/pandas-dev/pandas/main/doc/data/titanic.csv')

# 1. High-level overview with .info()
print("--- DataFrame Info ---")
df.info()
print("\n" + "="*40 + "\n")

# 2. Statistical summary with .describe()
print("--- Numerical Description ---")
print(df.describe())
print("\n" + "="*40 + "\n")

# You can also describe object columns
print("--- Categorical Description (for 'Sex' and 'Embarked') ---")
print(df[['Sex', 'Embarked']].describe())
print("\n" + "="*40 + "\n")


# 3. Frequency counts with .value_counts()
print("--- Value Counts for 'Pclass' (Passenger Class) ---")
print(df['Pclass'].value_counts())
print("\n--- Normalized Value Counts for 'Pclass' ---")
print(df['Pclass'].value_counts(normalize=True))

Running this code, you can immediately gather key insights:

1. From .info(), you'll see that the Age and Cabin columns have many missing values.
2. From .describe(), you can see the average age was about 29.7, but the passenger ages ranged from infants (0.42 years) to 80 years old. The ticket Fare also has a huge range, suggesting some very expensive tickets.
3. From .value_counts(), you'll quickly see that the majority of passengers were in the 3rd class.

This "triage" process—running .info(), .describe(), and .value_counts()—takes only a few seconds but provides a wealth of information to guide your next steps in any data analysis project.

Conclusion

You've now learned the three essential commands for the first look at any dataset in Pandas. This initial inspection is a critical, non-negotiable step in the exploratory data analysis process. It helps you form hypotheses, identify data quality issues, and plan your subsequent analysis and visualization steps.

Key Takeaways:

• df.info(): Your first command on any new DataFrame. It provides a structural summary, showing column names, data types, and non-null counts.
• df.describe(): Your tool for a quick statistical summary of all numerical columns. It reveals central tendency, spread, and range, helping to spot potential outliers.
• df['column'].value_counts(): Your go-to method for understanding the distribution of categorical columns. Use normalize=True to see proportions instead of raw counts.

In our next lesson, we'll address another common real-world scenario: data that is split across multiple files or tables. You will learn how to merge, join, and concatenate multiple DataFrames, a skill that draws parallels to SQL JOINs and is fundamental for preparing a complete dataset for analysis.