Common Data Quality Issues

The Cost of "Dirty" Data

Before diving into specifics, it's important to frame the problem for a management audience. Poor data quality isn't just a technical inconvenience; it has a real and significant financial impact on a business.

To set the stage, let's start with a short, effective analogy.

As the video highlights, even the best team and tools can't produce good results from bad "ingredients." The Google video "Understanding Data Cleaning" (resource ID: LINK) estimates this cost to be trillions of dollars for the US economy alone. For your workshop, framing data quality as a matter of risk management and financial prudence will resonate strongly with managers.

The Core Dimensions of Data Quality

Data quality is not a single concept but a collection of attributes. For your workshop, it's useful to break it down into a few key dimensions. This visual provides a clear framework.

Let's briefly define these dimensions, which you can then explain to your audience:

• Completeness: Is all the necessary data present? Are there missing values?
• Accuracy: Does the data correctly reflect the real world? Is a customer's address correct?
• Consistency: Is the data uniform across different systems and tables? Is "United States" recorded as "USA" in one file and "U.S." in another?
• Uniqueness: Is each record or entity represented only once? Are there duplicate customer entries?
• Timeliness: Is the data up-to-date and available when needed? Are you making decisions based on last year's sales figures?
• Validity: Does the data conform to the rules and constraints of its field? (e.g., a date in a date column, an email address in the correct format).

The IBM video you just saw also provides clear explanations for four of these core concepts.

A Gallery of Common Data Quality Issues

Now let's create a "rogue's gallery" of the specific problems that arise when data fails to meet these quality dimensions. Understanding these specific issues will help managers know what to look for in their own departments.

The following article provides an excellent, manager-friendly overview of these problems.

Based on that reading and other resources, here is a summary of the key issues to present in your workshop:

1. Duplicate Data: The same record appears multiple times.

   • Example: A customer is entered twice with slightly different spellings of their name, or a sales transaction is accidentally recorded twice.
   • Impact: Skewed sales reports, wasted marketing spend (contacting the same person twice), and inventory errors.

2. Incomplete Data (Missing Values or Nulls): Fields that should contain information are left blank.

   • Example: A customer record is missing a phone number or email address, or a product record is missing its price.
   • Impact: Inability to contact customers, flawed analysis (you can't calculate total sales if prices are missing), and poor user experience.

3. Inconsistent Data: The same data point is represented in different ways across your dataset.

   • Example: Storing dates as "12/31/2023" in one file and "Dec 31, 2023" in another. Using state abbreviations like "CA" in some records and the full name "California" in others.
   • Impact: Difficulty in merging data from different sources, incorrect grouping and filtering, and misleading reports.

4. Inaccurate or Invalid Data: The data is present but factually wrong or doesn't conform to the expected format.

   • Example: A customer's age is listed as "150", a product price is entered as a negative number, or an email address is just "johnsmith" with no "@" symbol. This is often caused by human error during data entry.
   • Impact: Failed deliveries from wrong addresses, incorrect financial calculations, and system errors when data types don't match (e.g., text in a number field).

From Theory to Practice: Spotting Issues in Power Query

For your workshop, it's essential to move from discussing these issues abstractly to showing managers how to find them. This is where you'll introduce the tools in Excel and Power BI. Power Query has excellent built-in Data Profiling tools for this exact purpose.

Please read the following documentation from Microsoft to understand these features.

Here's how you can connect the data quality issues we've discussed to what managers will see in Power Query's Data Profiling panes:

• Column Quality: This view gives a quick, color-coded bar for each column.

  • The dark grey (Empty) portion directly shows you Incomplete Data. A column with a large grey bar has many missing values.
  • The red (Error) portion shows Invalid Data. This happens when the data doesn't match the column's type (e.g., text in a number column).

• Column Distribution: This view shows a histogram of the most frequent values in a column.

  • It's a powerful tool for spotting Inconsistent Data. If you see "TX", "Tx.", and "Texas" as three separate bars in the distribution chart for a State column, you've found an inconsistency.
  • It also helps identify potential Duplicates and understand the uniqueness of your data. The distinct and unique value counts are crucial here. If a column that should be unique (like OrderID) has fewer unique values than distinct values, you have duplicates.

By teaching managers to enable and interpret these simple visuals, you empower them to quickly diagnose the health of their data without writing a single line of code.

Conclusion

You are now ready to teach a comprehensive lesson on identifying data quality problems. This is a critical prerequisite before any meaningful analysis or AI work can begin.

Key Takeaways for Your Workshop:

• Poor data quality costs businesses real money and leads to bad decisions ("Garbage In, Garbage Out").
• The most common data problems are duplicates, incomplete data, inconsistent data, and inaccurate/invalid data.
• These issues can be easily spotted without code using the Data Profiling tools in Power Query, specifically the Column Quality and Column Distribution features.

Preview of the Next Lesson:
We have now discussed the types of data and the quality of data. In our next lesson, we will address the final foundational data concepts: the importance of data volume and variety for AI applications. We'll explore why AI models often need large and diverse datasets to be effective.