Hello! Welcome to the first lesson of our course.
Over the next few weeks, we'll build the foundational knowledge you need to tackle a significant project at the intersection of Machine Learning and Software Quality Assurance. As we discussed, the first module is designed to give you a clear overview of three potential project areas: flaky test detection, technical debt prioritization, and automated test case generation. This will help you make an informed decision about which project to build.
Today, we'll start with the first of these topics. Our goal is to define what flaky tests are and understand their serious impact on the reliability of CI/CD pipelines. You'll learn why a test that sometimes passes and sometimes fails is more than just an annoyance—it's a major threat to a development team's velocity and confidence.
Let's begin.
What is a Flaky Test?
At its core, a flaky test is one that exhibits non-deterministic behavior: it can pass or fail for the same version of the code without any changes. Imagine running a test suite, seeing a test fail, and then simply re-running it to see it pass. That's a flaky test.
This unpredictability violates a core principle of automated testing: determinism. A reliable test should fail only when there is a genuine bug and pass when the functionality is correct.
To get a quick and engaging overview of this concept, let's watch a short video.
Why Your Tests Keep LYING To You?
The following video from the Modern Software Engineering channel provides a concise and clear definition of flaky tests.
Please watch the segment from 00:38 to 01:05. Pay attention to the different names used for flaky tests and the core definition provided.
Now that you have a general idea, let's add more detail. Flakiness isn't a single phenomenon; it can manifest in several ways.
What are Flaky Tests? How to Fix Flaky Tests?
The article "What are Flaky Tests?" from Semaphore provides a more detailed breakdown of the different forms flakiness can take.
Read the first two sections of the article: "What are flaky tests?" and "Why flaky tests are a problem". Focus on understanding the three types of flakiness mentioned: random, environmental, and branch flakiness.
The Impact on CI/CD Pipeline Reliability
Your experience with QA and continuous integration tools has likely shown you the importance of a "green" build. A green build signals that the code is stable and ready for the next stage, whether that's merging, further testing, or deployment. Flaky tests directly sabotage this process.
When a test fails in a CI/CD pipeline, it typically stops the entire process. The team must then investigate. This is where the real damage begins.
To understand this impact in more detail, let's read a short article that frames the problem in the context of CI/CD.
Detecting Flaky Tests in CI/CD Using Machine Learning
This article from Dev.to clearly outlines how flaky tests damage development velocity and why traditional, reactive approaches often fall short.
Read the sections titled "The Problem" and "Why It Damages CI/CD Velocity". Notice the connection between a single flaky test and its broader impact on developer productivity and decision-making.
The core issue is the erosion of trust. If developers can't trust the test results, the entire suite becomes noise.
- A false positive (a flaky test failing) causes developers to waste time investigating a non-existent bug.
- Worse, it creates a "boy who cried wolf" scenario. When a true positive occurs (a real bug is found), it might be dismissed as "just another flaky test," allowing the bug to slip through to production.
This directly affects team morale and productivity. Let's return to the first video to hear a developer's perspective on this.
Why Your Tests Keep LYING To You?
This segment of the video focuses on the human and team-level consequences of dealing with flaky tests.
Watch the segment from 01:22 to 02:16. The speaker discusses how flaky tests erode trust not just in specific tests, but in the entire test suite and even the quality process itself.
A Glimpse into the Causes of Flakiness
To eventually build a system that can predict flaky tests, we need to understand their root causes. Given your background in development and testing with tools like Selenium, you've likely encountered some of these issues before. The causes are often subtle and related to the interaction between the test code and the application environment.
Common culprits include:
- Asynchronous Operations: The test script moves faster than the application. For example, a test clicks a button and immediately checks for a result, but a background API call hasn't finished loading the data yet.
- Concurrency Issues: Two or more tests run in parallel and interfere with each other by using a shared resource, like a database record or a file. This can lead to race conditions or deadlocks.
- Environment Dependencies: The test relies on something specific to an environment that isn't guaranteed to be present or consistent everywhere (e.g., specific OS settings, network speed, or even the current date/time).
- Unstable Test Locators: In UI testing, if the way you identify an element (e.g., via a long XPath or a dynamically generated CSS class) is brittle, small UI changes can break the test.
Let's look at a concrete example of an asynchronous issue, one of the most common sources of flakiness in web application testing.
3 Reasons for Playwright Flaky Tests (and How To Fix It)
This video by Artem Bondar uses a practical example in Playwright to demonstrate how timing issues between a test script and the application's asynchronous behavior cause flakiness. The principles are universal and apply to Selenium or any other UI automation framework.
Watch the segment from 00:19 to 04:03. Don't worry about the specific Playwright syntax. Focus on understanding the problem: why does the test fail intermittently? Pay attention to the explanation around the 03:05 mark, where the presenter describes the race condition between the test filling an input and the application's API call overwriting it.
This example perfectly illustrates how a test can fail not because the application logic is wrong, but because the test isn't correctly synchronized with the app's behavior. Identifying patterns like these is the first step toward building an ML model for detection.
Key Takeaways
This lesson provided the essential definition and context for our first project theme. Here are the main points:
- A flaky test is one that passes and fails intermittently for the same code, making it non-deterministic and unreliable.
- The primary impact of flaky tests on a CI/CD pipeline is the erosion of trust, leading to wasted engineering time, slowed development velocity, and the risk of real bugs being ignored.
- Common causes of flakiness are often subtle and stem from issues like asynchronous timing, resource contention (concurrency), and environmental differences.
Understanding what a flaky test is and why it's a critical problem is the foundation for everything that follows. In our next lesson, we will explore our second potential project area: technical debt. We will define what it is, discuss how it accumulates in a codebase, and start thinking about how we might use ML to help manage it.