Mapping Software Engineering Problems to ML Tasks

A Quick Tour of ML Problem Types

At a high level, most supervised machine learning tasks fall into one of three families:

1. Classification: Assigning a predefined category or label to an input. Is this email spam or not spam? Is this test flaky or not flaky?
2. Regression/Ranking: Predicting a continuous numerical value or an ordered list. What will the price of this stock be tomorrow? Which technical debt items are most important to fix?
3. Generation: Creating new, structured content that mimics data it was trained on. Generate an image of an astronaut on a horse. Generate a unit test for this Python function.

Let's now map each of your project ideas to one of these paradigms.

1. Flaky Test Detection as a Classification Problem

Imagine you have a test that sometimes passes and sometimes fails in your CI/CD pipeline. The core question you want to answer is simple: "Is this test flaky?" This is a classic binary classification problem.

Your goal is to build a model that takes a test case as input and outputs one of two labels: flaky or not_flaky.

To make a prediction, the model needs data, or features, about the test case. These could include:

• The source code of the test itself.
• Historical execution data (e.g., pass/fail history, run duration).
• Code metrics of the application code being tested.
• Information from the version control system (e.g., how often the test file is changed).

The model learns patterns from this data to distinguish between flaky and reliable tests.

To see how this is done in practice, let's look at a research paper that built a framework called FlaKat for exactly this purpose.

As the paper describes, the process involves taking unstructured data (raw Java code) and systematically converting it into a structured format (vector embeddings) that classifiers like Random Forest or SVMs can process. This pipeline of parsing -> vectorizing -> classifying is fundamental to applying ML to code.

How Do You Measure Success?

For classification problems, you might think of using accuracy. However, flaky tests are often rare. If only 1% of your tests are flaky, a model that always predicts not_flaky would be 99% accurate but completely useless! This is known as the class imbalance problem.

Therefore, we use more informative metrics. The following video gives a great overview of the most important ones.

To put it in context:

• Precision: Of all the tests your model labeled as flaky, how many actually were? High precision means you don't waste developers' time investigating non-flaky tests (low false positives).
• Recall: Of all the tests that truly are flaky, how many did your model find? High recall means your model is good at catching flaky tests and they don't slip through into the codebase (low false negatives).
• F1-Score: The harmonic mean of precision and recall. It provides a single number that balances the two concerns, making it a very common metric for imbalanced problems like flake detection.

2. Technical Debt Prioritization as a Ranking Problem

Now, let's consider technical debt. Static analysis tools can easily find thousands of "debt" items in a large codebase. The problem isn't just finding them; it's deciding which ones to fix first.

A simple "yes/no" classification isn't enough. You need to produce an ordered list, from most critical to least critical. This is a ranking problem.

The goal is to build a model that takes a list of technical debt items and sorts them based on their "urgency" or "impact."

So, what determines this urgency? A simple metric like code complexity isn't enough.

Tornhill's key insight is that the most critical technical debt lies at the intersection of:

1. Code Complexity: The code is hard to understand.
2. Development Hotspot: Developers are frequently changing this code.
3. Knowledge Loss: The developers who know the code best are leaving.

An ML model can learn to weigh these factors (and many others) to produce a priority score for each debt item, which can then be used to rank them.

It's Not Just Ranking

While ranking is the most nuanced approach, the problem can also be simplified into a classification task. A PhD thesis on this exact topic explored several ways to frame it.

As the thesis shows, you can frame the problem in three ways:

1. Ranking (Ordered List): The most detailed output, producing a specific payment order (Item 42 -> Item 36 -> ...). This is often tackled with "learning-to-rank" models.
2. Multi-class Classification (Categories): A simpler approach where you assign each debt item to a priority bucket, such as High, Medium, or Low.
3. Binary Classification (Boolean): The simplest approach, where the model makes a Pay or Don't Pay decision for each item.

Your project could implement any of these, from a simple binary classifier to a more sophisticated ranking model.

3. Test Case Generation as a Generation Problem

Finally, we have test case generation. Unlike the previous two problems, the goal here isn't to predict a label or a score. It's to create a new artifact: a piece of executable source code. This is a generation problem.

As we covered in the last lesson, this task is often handled by Large Language Models (LLMs), a specific type of generative model.

The formulation is straightforward:

• Input: Context about the code you want to test. This could be the function's source code, its signature and docstring, or a natural language description of its behavior (e.g., from a user story).
• Output: A complete, syntactically correct test function that verifies the behavior of the input code.

For example:

Input:

def calculate_shipping_cost(weight, distance, is_express):
  """Calculates shipping cost.
  
  Args:
    weight: Weight of the package in kg.
    distance: Distance in km.
    is_express: Boolean for express shipping.
  
  Returns:
    The total cost.
  """



  # ... implementation ...

Output (generated by the model):

import pytest

def test_calculate_shipping_cost_basic():
  assert calculate_shipping_cost(5, 100, False) == 15.0 # example value

def test_calculate_shipping_cost_express():
  assert calculate_shipping_cost(5, 100, True) == 25.0 # example value

def test_with_zero_weight():
  with pytest.raises(ValueError):
    calculate_shipping_cost(0, 100, False)

How Do You Measure Success?

Evaluating generative models is more complex. We can't use simple metrics like F1-score. Instead, we have to assess the quality of the generated output:

• Execution: Does the generated test code run without syntax errors?
• Validity: Does the test pass when the application code is correct and fail when it's incorrect? (This can be tested with mutation testing).
• Coverage: How many lines of the application code does the generated test execute?

We will dive much deeper into prompt engineering and evaluating LLMs for code in Module 6. For now, the key is to recognize this as a fundamentally different type of ML problem.

Conclusion

You've now seen how each of your three project ideas maps onto a standard machine learning paradigm. This formalization is the bridge from a high-level goal to a concrete technical plan.

Key Takeaways:

Preview of the Next Lesson:
With this framework in mind, you're ready for the final lesson of this module. We will directly compare the three project directions across several practical criteria: data requirements, implementation complexity, and potential impact on a software team. This will equip you to make a final, informed decision on which project you want to build.