In our last lesson, you explored the iterative, question-driven process of exploratory data analysis (EDA). A key insight from that process is that our ability to ask and answer questions is often limited by the shape of our data. To create the insightful plots we discussed—comparing distributions with boxplots or showing relationships with scatterplots—your data must be structured correctly. This is where data wrangling, or "munging," becomes essential.
This lesson bridges the gap between raw data and tidy, analysis-ready data. You will learn the foundational techniques for importing and reshaping biological tables using a suite of powerful R tools: readr, dplyr, and tidyr. These packages are part of the tidyverse, an ecosystem of tools designed for data science that share a common design philosophy. Given your experience with pandas in Python, you'll find the goals familiar—selecting columns, filtering rows, and reshaping tables—but the tidyverse approach offers a distinct and highly expressive "grammar" for data manipulation.
The Foundation: "Tidy" Data
Before we can effectively manipulate data, we need to understand the ideal format we're aiming for. In the tidyverse world, this format is called "tidy data." It's a simple yet powerful convention that makes data wrangling, visualization, and modeling much more straightforward.
The book ModernDive provides an excellent introduction to this concept. It explains why a consistent data structure is so crucial, especially for tools like ggplot2 which are designed to work with tidy data.
Chapter 4 Data Importing and “Tidy” Data
Please read the introduction to the concept of "tidy" data in Section 4.2 of ModernDive. This section will define the core principles of tidy data and illustrate the difference between "wide" and "long" data formats using clear examples.
Focus on the section motivating example that shows why the drinks_smaller data frame is difficult to plot. Then, pay close attention to the formal definition. The three rules presented are the bedrock of the tidyverse philosophy: Each variable forms a column. Each observation forms a row. Each type of observational unit forms a table. The stock price example that follows provides a simple, non-biological case to solidify your understanding.
Think about a typical high-throughput screening result. You might have a 96-well plate where columns 1-12 represent different compounds and rows A-H represent different concentrations. A "wide" representation of this might have 12 columns, one for each compound, with measurements in the cells. A "tidy" representation would have columns for Well, Compound, Concentration, and Measurement. This tidy format is much easier to plot, for instance, by mapping Compound to color to see all dose-response curves on one graph.
From Wide to Tidy: The Power of pivot_longer()
The most common reshaping task is converting data from a wide format, which is often how data is recorded, to a tidy (long) format. The workhorse function for this is pivot_longer() from the tidyr package.
The following video provides an excellent walkthrough of pivot_longer() using a biological dataset of body temperature measurements taken at different times. It clearly breaks down the function's syntax and purpose.
Tidying Data in R with pivot_longer()
This video from the Equitable Equations channel, "Tidying Data in R with pivot_longer()", is a focused tutorial on the main topic of this lesson. It explains not just how to use the function, but why you need it.
Watch the entire video. Key moments to focus on are: The motivation: The instructor shows a "wide" dataset and explains why it's not tidy. The basic syntax: This part breaks down the essential arguments: cols, names_to, and values_to. Dropping NAs: The values_drop_na = TRUE argument is a very practical tool for cleaning up the results of a pivot. Splitting columns: This advanced usage shows how to split a single messy column name into multiple new variables during the pivot, a very common and powerful technique.
To reinforce this, let's look at the same process in a text-based resource. The ModernDive chapter you read earlier also provides a clear, step-by-step example using pivot_longer() on the drinks data.
Chapter 4 Data Importing and “Tidy” Data
Now, let's see pivot_longer() in action within the ModernDive text. You'll apply the concept to two different datasets.
First, review Subsection 4.2.2. This walks through converting the drinks_smaller data frame you saw earlier. Pay close attention to the explanation of the names_to, values_to, and cols arguments. Next, work through the case study in Section 4.3. This is another excellent example of a common "messy" data format, where years are used as column headers. This is directly analogous to having different time points or experimental conditions as columns in a biological dataset. Notice the use of names_transform to ensure the new year column is treated as a number, not text.
The Grammar of Data Manipulation: dplyr
Reshaping is only one part of the puzzle. We also need to select specific columns, filter for rows that meet certain criteria, and create new variables. The dplyr package provides a set of functions, or "verbs," that make these tasks intuitive. Combined with the pipe operator (%>%), you can build complex, yet readable, data manipulation workflows.
The pipe %>% takes the output of the function on its left and passes it as the first argument to the function on its right. This allows you to chain commands together without creating intermediate variables, similar to how you might chain methods in pandas (e.g., df.filter().select()...).
The Data Carpentry lesson "Manipulating, analyzing and exporting data with tidyverse" provides a comprehensive, hands-on tour of these tools using a portal ecology dataset that's a great proxy for many biological survey studies.
Manipulating, analyzing and exporting data with tidyverse
This resource is a practical, step-by-step tutorial. We will walk through the core dplyr verbs and see how they combine with tidyr.
Please read through the following sections, and I encourage you to run the code in your R environment as you go. Start with the introduction to dplyr and tidyr. It introduces the tidyverse and the read_csv function, which you should use for importing data. Work through Pipes. This section introduces the core verbs select() and filter() and, most importantly, the %>% pipe operator. Understanding the pipe is crucial for writing effective tidyverse code. Continue with Mutate to learn how to create new columns. This is equivalent to pandas's .assign() or direct column assignment. Finally, read the section on the split-apply-combine paradigm, which introduces group_by() and summarize(). Your experience with pandas.groupby().agg() will make this concept very familiar. As you read, consider your research on Pseudomonas aeruginosa. You could filter a genomic dataset for specific strains, select columns related to antibiotic resistance genes, mutate to calculate the length of a gene, and then group_by gene family and summarize to find the average length per family.
Putting It All Together
Let's conclude with a video that shows a complete workflow, from importing data to manipulation and finally to plotting, all within a single, elegant tidyverse pipeline. This will demonstrate the efficiency and clarity you gain by combining these tools.
How to Import, Manipulate & Visualize Data Using the tidyverse in R | readr, dplyr & ggplot2 Package
This "Statistics Globe" video demonstrates a full data analysis workflow using the tidyverse on a dataset about tourism. While the data is not biological, the principles and the code structure are universally applicable.
The key part of this video is its demonstration of chaining. The presenter initially shows each step (rename, select, mutate, filter) creating a new intermediate object. Then, they show how to combine all those steps into one fluid pipeline. Watch from the data manipulation section, where the presenter cleans the data step-by-step. Then, pay close attention to the final demonstration starting at the "one single pipe" section. This is the goal: to be able to reason about and construct such a chain to move from raw data to your desired result.
Conclusion
In this lesson, you've acquired the foundational skills for data wrangling in R using the tidyverse. You now understand the "what" (the tidy data format) and the "how" (the tools to achieve it).
Here are the key takeaways:
- Tidy Data: The ideal data structure has one observation per row, one variable per column, and one value per cell. This format is the key that unlocks the power of the
tidyverse. readr::read_csv(): Your go-to function for importing tabular data into R.tidyr::pivot_longer(): The essential tool for transforming "wide" data, where column names are actually values of a variable, into the tidy "long" format.dplyrVerbs: A consistent set of functions (filter,select,mutate,group_by,summarize) that form the grammar of data manipulation.- The Pipe
%>%: The operator that glues everything together, allowing you to build readable and powerful data processing pipelines without cluttering your workspace with intermediate variables.
You are now equipped to take a raw, messy data table and whip it into shape for analysis. In our next lesson, we will put this new skill to immediate use. You'll take a dataset, reshape it as needed, and use ggplot2 to recreate a complex exploratory visualization, solidifying the connection between data shape and data visualization.