Problem Definition, Data, and Metrics

The Machine Learning Project Lifecycle

A successful machine learning project is a structured, iterative process. It doesn't start with modeling; it starts with a question. The following flowchart provides a high-level map of the journey we're about to embark on. Today, we are firmly in the "Problem Definition" stage.

1. Defining the Machine Learning Problem

The first step is to frame a real-world goal in machine learning terms. This means translating a business need or a point of curiosity into a concrete task for a model.

There are generally two ways to approach this:

1. Problem-First: You start with a question you want to answer (e.g., "Can I predict which customers are likely to churn?") and then search for data to help you answer it.
2. Data-First: You find an interesting dataset and then explore what questions you can answer with it (e.g., "I have a dataset on housing sales; can I predict a house's price?").

For portfolio projects, the problem-first approach often leads to more compelling and unique work. The following video offers a great demonstration of this process.

From Ideal Outcome to Model Goal

Once you have a general idea, you need to refine it. The goal is to connect a high-level ideal outcome to a specific model goal.

For example, if you were building a project to help a bank, the:

• Ideal Outcome: Reduce financial losses from defaulted loans.
• Model's Goal: Predict whether a given loan applicant will default.

This simple but explicit statement defines the scope of your project and what you will be building.

2. Selecting an Appropriate Dataset

With a problem defined, you need data. For your portfolio, you'll likely use a public dataset. The quality and nature of the dataset will significantly impact your project.

Where to Find Datasets

There are many excellent repositories for public datasets. The video below gives a tour of the most popular ones. Since you're highly proficient with Git and GitHub, pay special attention to the segment on finding datasets within GitHub repositories.

What Makes a Good Dataset for a Portfolio Project?

Not all datasets are created equal. For an end-to-end project, you want something that allows you to demonstrate a range of skills. An ideal dataset isn't just a simple, clean CSV file.

3. Choosing a Model Type and Evaluation Metric

With a problem and a dataset, the final step is to define how you will build a solution and how you will measure its success.

Classification vs. Regression

Your problem statement naturally leads to a model type.

• Regression: Predicts a continuous numerical value (e.g., predicting a house price, a salary, or temperature).
• Classification: Predicts a discrete category (e.g., spam or not_spam, will_churn or will_not_churn, fraud or not_fraud).

Sometimes the choice is subtle. If your problem has fixed numerical thresholds that trigger different actions, a classification model might be better even if the underlying value is a number.

Success Metrics vs. Evaluation Metrics

It's important to distinguish between two types of metrics:

1. Success Metrics (Business Metrics): Did the project achieve its high-level goal? (e.g., "Reduced customer churn by 5%", "Increased average user session time by 2 minutes"). These are what the business cares about.
2. Evaluation Metrics (Model Metrics): How well does the model perform its specific task? (e.g., Accuracy, Precision, Recall, RMSE). These are what you use to train and compare models.

Your choice of evaluation metric should be a proxy for your success metric.

Selecting the Right Evaluation Metric

Choosing the right evaluation metric is critical. Using the wrong one can lead you to optimize for a model that is useless in practice. This is especially true for imbalanced datasets, where one class is much rarer than another (e.g., fraud detection, disease diagnosis).

The Dataquest article provides a perfect illustration of this.

This thought process—connecting the model's errors to real-world consequences—is a hallmark of a skilled data scientist.

Your Turn: Plan Your Project

Now it's time to apply these concepts and define your own end-to-end project. Choose one of the following domains or propose your own idea.

Project Options:

1. Customer Churn Prediction (Classification): Use a dataset like the Telco Customer Churn dataset on Kaggle.
2. Credit Card Fraud Detection (Classification): Use a dataset like the Credit Card Fraud Detection dataset on Kaggle. This is a classic example of an extremely imbalanced problem.
3. House Price Prediction (Regression): Use a dataset like the Ames Housing dataset. This dataset is rich with features, offering great opportunities for feature engineering.

Task (to be completed in a new README.md file in your project folder):

For your chosen project, write a brief project plan that answers the following questions:

1. Problem Definition:

   • What is the ideal outcome? (e.g., "Help a telecom company retain more customers.")
   • What is the specific model's goal? (e.g., "Predict whether a customer will churn within the next month based on their account information and usage patterns.")

2. Model and Target:

   • What is the target variable you will be predicting?
   • Is this a regression or classification problem?

3. Evaluation Metric:

   • What will be your primary evaluation metric?
   • Justify your choice. For example, in churn prediction, is it more costly to mistakenly offer a retention bonus to a happy customer (a false positive) or to fail to identify a customer who is about to leave (a false negative)? Your choice of Precision, Recall, F1-score, or AUC should reflect this. For regression, will you use RMSE or MAE, and why?

Completing this plan will give you a clear roadmap for the rest of the project.

Conclusion

You've now learned the structured process for beginning any data science project. This foundational work is what separates successful, impactful projects from aimless explorations.

Key Takeaways:

• Problem framing is key: Start with a clear "ideal outcome" and a specific "model goal."
• Choose datasets with purpose: Look for data that has enough complexity and nuance to build a compelling portfolio project.
• Connect metrics to business value: Select an evaluation metric (like Precision, Recall, or RMSE) that serves as a good proxy for the real-world success of your project, especially considering factors like class imbalance.

In our next lesson, you will take the project plan you've just created, acquire the data, and dive into the first hands-on step of the workflow: building a complete feature engineering and preprocessing pipeline.