Handling Missing Values (NA)

1. The "Viral" Nature of NA

In R, NA (Not Available) is not just a placeholder; it's a special logical value with specific properties. One of its most important characteristics is its "viral" nature: any arithmetic or logical operation that includes an NA will almost always result in NA.

For example, what is 5 + NA? We don't know what the missing value is, so the result is also unknown. R's answer is NA. This is a deliberate and crucial feature of the language. It acts as a safety mechanism, preventing you from accidentally getting a numerical result from an incomplete calculation. If R simply ignored NAs by default, you could easily compute a misleading average or sum without ever realizing your data was incomplete.

Let's see how this plays out with a common statistical function.

As the article demonstrates, R forces you to be explicit about your intentions. This is a core principle of good data analysis: you, the analyst, must decide how missing data should be handled.

2. Finding and Counting Missing Values with is.na()

Before you can handle missing values, you must first find them. R's primary tool for this is the is.na() function. It takes a vector, data frame, or other R object and returns an object of the same size, but with TRUE wherever an NA was present and FALSE otherwise.

This TRUE/FALSE output is incredibly useful because R treats TRUE as 1 and FALSE as 0 in mathematical operations. This allows for a simple and powerful way to count missing values.

• sum(is.na(x)) will count the total number of NAs in a vector x.
• colSums(is.na(df)) is a very common idiom that will give you the count of NAs in each column of a data frame df.

To see this in action, the following short video provides a good visual summary of applying is.na() and then sum(is.na()) to a dataset.

3. Calculating with Missing Data: The na.rm Argument

As we saw earlier, functions like mean(), median(), sum(), sd(), and min() will return NA if their input contains any missing values. The solution is the na.rm = TRUE argument, where rm stands for "remove". By adding this argument, you are telling the function to strip out the NA values before performing the calculation.

This is the most common and direct way to handle missing values for a specific calculation without modifying your underlying data.

This approach is powerful because it's localized. You deal with the NAs for one specific summary statistic, but the NAs remain in your original data frame, which is important. You may want to handle them differently for other analyses (for instance, by removing the entire row, which we'll touch on next).

4. A Note on Excluding Rows

Sometimes, ignoring an NA for a single calculation isn't enough. For some analyses, you might decide that any observation (row) with a missing value in a critical variable is unusable. In these cases, you might want to remove the entire row.

Base R offers two common functions for this: na.omit() and complete.cases(). Both can be used to filter a data frame, keeping only the rows that have no missing values. However, this method should be used with extreme caution.

The danger highlighted in the reading is significant, especially in clinical research where a participant might have complete data on primary outcomes but be missing a secondary demographic variable. Using na.omit() carelessly could discard valuable data.

The tidyverse offers a more controlled function for this, tidyr::drop_na(), which we will cover in a future lesson. For now, the key takeaway is to prefer the na.rm argument for calculations and to be very deliberate if you choose to remove entire rows of data.

Let's Practice

Let's apply these concepts in your clinical-trial-practice RStudio Project. You can create a new script, perhaps named 05-missing-values.R.

1. Load packages and data. We will use the penguins dataset again, as it contains several missing values.

   library(tidyverse)
   
   penguins_url <- "https://raw.githubusercontent.com/allisonhorst/palmerpenguins/main/inst/extdata/penguins.csv"
   penguins_data <- read_csv(penguins_url)
   

2. Task 1: Inspect and Count NAs.

   • First, get a quick overview with summary(). This function, as we've seen, conveniently lists the number of NAs per column.

     summary(penguins_data)
     

   • Now, let's practice using our new tools to get the same information. Get a count of NAs for each column using colSums() and is.na().

     colSums(is.na(penguins_data))
     

     Your results should match the summary() output. This is a great way to start any data exploration.

3. Task 2: Practice with na.rm.

   • Try to calculate the average body mass of the penguins. What happens?

     mean(penguins_data$body_mass_g)
     

     R returns NA because the body_mass_g column contains missing values.
   • Now, perform the calculation correctly using na.rm = TRUE.

     mean(penguins_data$body_mass_g, na.rm = TRUE)
     

     You should now get a valid numerical answer.

4. Task 3: Filter to Inspect NA Rows.

   • Sometimes it's useful to look at the specific rows that contain missing data. You can combine the filter() function (which you'll learn more about soon) with is.na() to do this. Let's find all the penguins where the sex information is missing.

     penguins_data %>% 
       filter(is.na(sex))
     

   • This shows you all 11 rows where sex is NA. Inspecting these rows might give you clues about why the data is missing. For example, are they all from a particular island or year? This is a key step in thoughtful data analysis.

Conclusion

You have now learned the essential techniques for one of the most common tasks in data cleaning: handling missing values. This foundational skill prevents errors and forces a more deliberate and conscious approach to data analysis.

Key Takeaways:

• NA is Viral: R's default behavior is to propagate NAs in calculations to prevent you from getting a silently incorrect answer.
• Detect with is.na(): The is.na() function is your primary tool for locating missing values. Combining it with sum() or colSums() allows you to quickly quantify the extent of missing data in your dataset.
• Calculate with na.rm = TRUE: For summary statistics (mean, sd, median, etc.), use the na.rm = TRUE argument to perform the calculation on the available data without altering your original dataset.
• Exclude Rows with Caution: Functions like na.omit() remove entire rows containing NAs. While sometimes necessary, this can lead to significant data loss and should be used with careful consideration.

Preview of the Next Lesson:
So far in this module, we've learned how to bring data into R from various file types and how to perform initial inspections, including dealing with factors and missing values. The final step in this part of the data workflow is saving your work. In our next lesson, we will learn how to export data frames to CSV files using readr::write_csv(), completing the import-clean-export cycle.