Welcome back! In our last lesson, you learned a systematic process for auditing and cleaning a new dataset. This is the essential first step of any analysis, ensuring the data you work with is accurate and reliable. Now that you have a clean dataset, you're ready to move on to the next, more creative phase: exploratory data analysis (EDA).
The central purpose of EDA is to develop an understanding of your data. It's an iterative, question-driven process where you use visualizations and transformations to uncover patterns, spot anomalies, and build intuition. For a researcher like yourself, this is the computational equivalent of observing a new phenomenon under a microscope—you're looking for interesting features that can be turned into testable hypotheses. This lesson will guide you through this process, focusing on how to use distribution and relationship plots to transform unexpected patterns into concrete scientific questions.
The Mindset of Exploratory Data Analysis
Before we dive into specific plot types, it's crucial to understand the philosophy of EDA. It's not a linear procedure with a fixed endpoint, but a cycle of inquiry.
10 Exploratory data analysis - R for Data Science (2e)
This reading introduces the iterative cycle of EDA and frames it as a question-driven process. It will help you adopt the right "state of mind" for exploration.
Please read the Introduction and Questions sections. Focus on the idea that EDA is a cycle of generating questions, searching for answers by visualizing and transforming data, and then using those discoveries to refine your questions. Notice the two fundamental types of questions posed: one about variation and one about covariation. These two questions will form the structure of our lesson.
As the text highlights, EDA is about asking a large quantity of questions to find a few high-quality ones. You start with broad questions and progressively narrow your focus as you uncover interesting aspects of the data.
Visualizing Variation: Understanding Single Variables
The first fundamental question we can ask is: "What type of variation occurs within my variables?" Variation is the tendency of a variable's values to change from one measurement to the next. Visualizing this variation helps us understand the distribution of our data.
Numerical Variables: Histograms and Density Plots
For numerical data, like gene expression levels or patient age, histograms and density plots are the primary tools. They show which values are common, which are rare, and how the values are spread out.
Let's look at the gene expression data from the "Modern Statistics for Modern Biology" resource. A histogram can quickly show us the distribution of expression values for a single sample.
3 Data visualization – Modern Statistics for Modern Biology
This chapter provides an excellent overview of different ways to visualize distributions, using a real single-cell gene expression dataset.
First, read the section on histograms to see how geom_histogram is used. Then, skim the following subsections on Boxplots, Dot plots, Density plots, and Violin plots. As you read, think about the pros and cons of each visualization for answering the following questions about a variable's distribution: Where is the center of the data? How spread out are the values? Is the distribution symmetric, or is it skewed? Are there multiple peaks (bimodality/multimodality)? Are there outliers or extreme values?
When you see a pattern in a distribution plot, your research instincts should kick in to generate questions. For example:
- A Bimodal Distribution: You plot the expression of a transcription factor and see two distinct peaks.
- Testable Question: Do these two peaks correspond to cells from two different lineages (e.g., epiblast vs. primitive endoderm in the
Hiiragi2013dataset) that you can verify with known markers?
- Testable Question: Do these two peaks correspond to cells from two different lineages (e.g., epiblast vs. primitive endoderm in the
- A Long Tail (Skewness): You plot the distribution of Minimum Inhibitory Concentrations (MICs) for an antibiotic. Most values are low, but there's a long tail of high-MIC isolates.
- Testable Question: Do the isolates in the high-MIC tail share a specific genetic variant (e.g., a mutation in a drug efflux pump) that is absent in the low-MIC isolates?
- Outliers: A plot of patient ages in a clinical study shows a few values below zero.
- Question: This is almost certainly a data entry error that should have been caught during the cleaning phase (Lesson 1.1), but EDA provides a second chance to catch such issues. A real biological outlier might be a patient with an extremely high biomarker level, which could prompt the question: Does this patient have a rare genetic condition affecting the biomarker's pathway?
Visualizing Covariation: Uncovering Relationships
While understanding individual variables is important, the most powerful hypotheses often come from exploring how variables change together. This is the second fundamental question: "What type of covariation occurs between my variables?"
Covariation is the tendency for the values of two or more variables to vary in a related way. Visualizing these relationships is the core of hypothesis generation in EDA.
A Categorical and a Numerical Variable
How does a numerical measurement change across different categories? This is a common question in biology (e.g., comparing a continuous outcome between treatment and control groups). Boxplots and violin plots are excellent for this.
10 Exploratory data analysis - R for Data Science (2e)
This section provides a clear guide to visualizing relationships between different types of variables using ggplot2.
Please read the subsection on categorical and numerical variables. The example explores how diamond price varies with cut quality. Pay close attention to the surprising initial finding and how the author encourages you to dig deeper. This is a key part of the EDA mindset.
Imagine you're exploring your C. neoformans chemotyping data. You could plot the distribution of a specific chemical shift value (a numerical variable) across the different structural motifs (a categorical variable).
- Pattern: You create boxplots and see that the median chemical shift for Motif A is consistently higher than for all other motifs.
- Testable Question: Is this specific chemical shift a unique and reliable biomarker for identifying Motif A? Can a simple threshold on this value be used to classify unknown samples with high accuracy?
Two Numerical Variables
To explore the relationship between two numerical variables, the scatter plot is your primary tool. With the large datasets common in bioinformatics, however, simple scatter plots can suffer from overplotting, where dense clouds of points obscure the true relationship.
10 Exploratory data analysis - R for Data Science (2e)
Continuing in the same chapter, we'll now focus on relationships between two continuous variables.
Read the subsection on two numerical variables. Focus on the techniques used to deal with overplotting in large datasets, such as using the alpha aesthetic for transparency or binning points with geom_bin2d() and geom_hex().
As a molecular biologist, you frequently work with gene expression. You might plot the expression of one gene against another for hundreds of single cells.
- Pattern: You see a strong positive correlation between the expression of Gene X and Gene Y. When X is high, Y is high, and vice-versa.
- Testable Question: Are Gene X and Gene Y co-regulated? Do they share a common transcription factor binding site in their promoter regions? Are they known members of the same biological pathway?
Two Categorical Variables
To see how two categorical variables are related, you need to count the number of observations for each combination. A heatmap of counts is an effective visualization.
- Pattern: You plot a heatmap of microbial species (categorical) vs. antibiotic resistance phenotype (categorical: Resistant/Susceptible). You observe a bright spot indicating that for Species Z, a large proportion of isolates are resistant.
- Testable Question: Does Species Z possess an intrinsic resistance mechanism (like a specific beta-lactamase) that is less common in other species?
Case Study: From Patterns to Questions
The best way to solidify this process is to see it in action. In this talk, Hadley Wickham provides a masterclass in using ggplot2 to explore a dataset of mortality in Mexico, turning visual patterns into insightful, testable questions.
Engineering Data Analysis (with R and ggplot2)
This segment of the talk is a perfect demonstration of the EDA workflow. He starts with a broad pattern and systematically drills down to find interesting anomalies, which he then formulates as hypotheses.
Please watch two key sections: First, watch the section on finding unusual hourly death patterns. Notice how he doesn't just look at thousands of plots. He first establishes an overall pattern, calculates a "distance" from that pattern for each disease, visualizes that distance against sample size to find true outliers, and only then investigates the most interesting cases. Next, watch the segment on investigating temperature and wind speed. He plots death frequency against weather variables. The plot of deaths vs. wind speed reveals a striking negative relationship, leading directly to a testable hypothesis about air pollution.
This case study is the embodiment of this lesson's goal. Wickham doesn't just display data; he interrogates it. He uses visualization to find where his expectations are violated and uses those violations to ask new, more specific questions. This is the skill you are developing.
Conclusion
In this lesson, we have moved from the structured task of data cleaning to the creative, iterative process of exploratory data analysis. You've learned that EDA is a scientific inquiry in its own right, guided by curiosity and skepticism.
Here are the key takeaways:
- EDA is a question-driven cycle: You generate questions, seek answers with plots, and refine your questions based on what you find.
- Start with variation and covariation: The two most fundamental questions you can ask are "How do my variables vary?" and "How do my variables covary?"
- Use the right plot for the job:
- One numerical variable: Histograms, density plots.
- Categorical vs. Numerical: Boxplots, violin plots.
- Two numerical variables: Scatter plots (with attention to overplotting).
- Two categorical variables: Heatmaps or stacked bar charts.
- Turn patterns into questions: The ultimate goal of EDA is not just to create plots, but to interpret the patterns you see and formulate them as testable biological hypotheses.
The visualizations you create in EDA often reveal that your data is not in the ideal format for the plot you want to make. For instance, you might have data in a "wide" format when a "long" format is needed for ggplot2. In our next lesson, we will tackle this head-on by learning how to use powerful tools from the tidyverse—readr, dplyr, and tidyr—to import and reshape your data, giving you full control over its structure.