Welcome to your first lesson. Your goal is to build a strong portfolio in computational biology, and every robust analysis begins not with complex modeling, but with a thorough understanding and cleaning of the data. Just as you wouldn't start a wet-lab experiment without calibrating your instruments and checking your reagents, you cannot start a computational analysis without ensuring your data is clean, consistent, and valid. This is often called data "janitoring" or "wrangling," and it is one of the most critical skills in the field.
This lesson will equip you with a systematic approach to this process. Guided by the learning outcome, you will learn how to audit an unfamiliar biological table for five common types of issues: incorrect data types, missing values, duplicate entries, invalid data ranges, and inconsistent identifiers. By the end, you will have a practical checklist to apply to any new dataset you encounter.
The Importance of a Data Audit
Before we dive into the "how," let's briefly touch on the "why." Every dataset, especially those compiled from experimental or clinical sources, contains errors. These can range from simple typos to systematic issues arising from data collection methods. Failing to identify and address these problems can lead to incorrect or non-reproducible results, undermining your entire analysis. Your prior research on pathogens like Mycobacterium abscessus and Cryptococcus neoformans likely involved meticulously recorded data; the principles of rigor and verification you applied in the lab are equally, if not more, important here.
A Systematic Approach to Auditing
We will approach data auditing as a step-by-step process. For our examples, we will be using R with the tidyverse suite of packages, which are standard tools for data exploration and manipulation.
1. The First Look: Getting an Overview
The first step with any new dataset is to get a high-level summary. How many rows and columns are there? What are the data types of each column? How many values are missing?
A powerful tool for this initial reconnaissance is the skim() function from the skimr package. It provides a comprehensive yet readable summary of a dataframe.
8 Cleaning data and core functions
Please read the introduction and the "Review" section of this chapter from The Epidemiologist R Handbook.
Focus on the example output of the skimr::skim() function. You'll find it under the "Review" heading, starting after the text You can use. Observe how the output is organized by variable type (character, numeric, etc.) and provides key metrics for each column: n_missing: The count of missing values. complete_rate: The proportion of non-missing values. n_unique: The number of distinct values in a column. For numeric data, it provides a mini-histogram (hist) and summary statistics (mean, standard deviation, percentiles), which are crucial for spotting strange values.
This single command gives you an immediate sense of your dataset's overall health and points you toward specific columns that need a closer look.
2. Auditing Identifiers
In biological datasets, identifiers are the glue that holds everything together. Errors here can completely scramble your data. We'll look at two common types of identifier issues.
Column Name Inconsistencies
Column names can be problematic if they contain spaces, special characters, or are inconsistent in their naming convention (e.g., date_onset vs. infection date). This makes them difficult to work with in code. The janitor package offers a simple solution.
8 Cleaning data and core functions
Now, let's look at how to handle messy column names. Continue in the same resource.
Read the section on "Column names," starting from the introduction and continuing through the "Manual name cleaning" subsection. Pay attention to the janitor::clean_names() function for automatic standardization and the dplyr::rename() function for targeted changes. This section, ending just before "Rename by column position," provides a clear workflow for tidying your column identifiers.
Sample and Feature Identifier Mismatches
A more pernicious issue arises when identifiers for the same samples or features (like genes) don't match across different files. For example, the sample IDs in your metadata file must perfectly correspond to the sample IDs in your gene expression matrix.
The following resource provides a practical example of checking and resolving this.
RNA-Seq: Data Readiness for Machine Learning Applications: Data Readiness: Data Format and Integrity
This Carpentries lesson demonstrates auditing a real-world Inflammatory Bowel Disease (IBD) dataset, which consists of a sample information file and a count matrix.
First, read the subsection "Check 1: Unique identifier matching between counts matrix and sample information". It begins just after the first "Challenge 1" box. Read from the text All of the checklist down to the first code block that uses dplyr::glimpse(). This shows how to programmatically verify that your sample identifiers line up. Then, skip down to the "IDB dataset: Counts Matrix Clean Up" section and read from the heading "Check 2: Unique variable names" until the ERROR message appears after the if(!identical(...)) code block. You can find this starting around the identity check. This demonstrates what happens when the identifiers don't match and how to fix them by cleaning the column names in the count matrix to match the sample info.
3. Checking Data Types
R needs to know if a column contains numbers, text, or categorical data to perform the correct operations. A column of ages stored as text ("character") instead of numbers ("numeric") can't be used for calculations. Similarly, a categorical variable like sex or disease status is best represented as a "factor".
This video provides an excellent, beginner-friendly introduction to checking and converting data types in R.
Clean your data with R. R programming for beginners.
The presenter uses a starwars dataset, but the principles are universal. This segment will show you how to inspect and change variable types.
Watch the segment from variable types. The key functions to note are: glimpse() and class() to check a variable's type. unique() to see all the distinct values within a character variable. as.factor() to convert a character variable into a categorical factor. factor() with the levels argument to control the specific order of categories, which is often important for plots and models.
4. Finding and Understanding Missing Values
Missing data, often represented as NA (Not Available), is a fact of life in biological research. Your approach to handling it depends on why it's missing. Is a value missing because a measurement failed, or is it missing because it's not applicable (e.g., the hair color of a droid, as the video amusingly notes)?
The following segment of the video demonstrates how to find missing values and implement different strategies for handling them.
Clean your data with R. R programming for beginners.
This part of the video explains why a blanket "delete all missing data" approach is risky and shows more nuanced techniques.
Please watch from missing data. Focus on these concepts: Using filter(!complete.cases(.)) to isolate and inspect rows that contain missing values. Using drop_na() to remove rows where a specific column has a missing value. Using mutate() with replace_na() to replace missing values with something meaningful, like "none" or "not_applicable".
5. Detecting Duplicates
Duplicate rows can arise from data entry errors or from merging datasets. They can bias your results by giving undue weight to the duplicated observations.
Clean your data with R. R programming for beginners.
The final video segment shows how to identify and remove duplicate rows. The presenter first creates a small sample dataset with duplicates to make the concept clear.
Watch from duplicates. The main takeaway is the distinct() function from tidyverse, which is the simplest and most common way to remove rows that are 100% duplicates of another.
6. Validating Data Ranges
The final check in our audit is for invalid ranges. This means checking for values that are biologically or logically impossible. For example:
- A human
agecolumn with negative values. - A
wt_kg(weight in kg) column with a value of 500. - A
pHvalue of 15. - A percentage greater than 100.
There isn't a single function for this; it requires your domain knowledge. The process is straightforward:
- Ensure the column is the correct numeric type (Step 3).
- Use the
summary()function or the output fromskim()(Step 1) to view the minimum, maximum, and quartile values. - Examine these summary statistics for anything that seems implausible. A
minage of -1 or amaxtemperature of 200°C for a clinical sample are clear red flags that require investigation.
This step is your "sanity check," combining your scientific expertise with your data skills.
Conclusion
In this lesson, you've learned the fundamentals of data auditing, a cornerstone of reproducible computational research. We've established a systematic, six-step process for inspecting a new dataset:
- Get an overview with
skimr::skim(). - Audit identifiers by cleaning column names (
janitor::clean_names(),dplyr::rename()) and checking for consistency across files. - Check data types with
glimpse()andclass(), and convert them with functions likeas.factor()oras.numeric(). - Handle missing values thoughtfully by inspecting them (
!complete.cases()) and then choosing to either drop (drop_na()) or replace (replace_na()). - Detect duplicates and remove them with
dplyr::distinct(). - Validate data ranges using
summary()and your own domain knowledge to spot impossible values.
Mastering this process will save you countless hours and prevent significant errors down the road. It ensures the foundation of your analysis is solid.
In the next lesson, we will build directly on this foundation. Once you have a clean dataset, the next step is to explore it visually. You will learn to use distribution and relationship plots to uncover patterns, spot anomalies, and transform your observations into testable biological questions.