Hyperparameter Tuning with Validation Sets

1. Parameters vs. Hyperparameters

First, it's crucial to understand the distinction between two types of "parameters" in machine learning:

• Model Parameters: These are values that the model learns from the data during training. The goal of training is to find the optimal values for these parameters. Examples include the weights ($\beta$) in your linear and logistic regression models, or the weights and biases in your neural network. You never set these by hand.
• Hyperparameters: These are configuration settings for the learning algorithm itself, which you, the practitioner, must set before training begins. They are not learned from the data but rather guide the learning process.

Examples of hyperparameters in the models you've built include:

• The learning_rate ($\eta$) and n_iterations in your gradient descent optimizer.
• The regularization strength (lambda or alpha) and type (L1 or L2) in your regularized regression models.
• The soft-margin constant C in your SVM.
• The number of trees (n_trees), max_depth, and min_samples_split in your Random Forest.
• The number of hidden layers and neurons in your MLP.

The process of finding the best combination of these settings is called hyperparameter tuning.

To get a quick overview of the main concepts, watch the first part of this video.

2. The Core of Tuning: The Validation Set

As the video mentioned, we cannot use the test set to tune hyperparameters. If you tune your model's settings based on its performance on the test set, you are effectively "leaking" information from the test set into your model. Your model will become optimized for that specific test set, and its performance score will be an overly optimistic estimate of how it would perform on new, unseen data.

This is where the validation set comes in. You split your original training data into two new parts:

1. A smaller training set for learning the model parameters (e.g., weights).
2. A validation set for evaluating the model with a given set of hyperparameters.

The process looks like this:

1. Pick a set of hyperparameters.
2. Train the model on the smaller training set.
3. Evaluate the model's performance on the validation set.
4. Repeat with a different set of hyperparameters.
5. After trying all combinations, select the hyperparameters that gave the best score on the validation set.
6. Finally, train your model one last time on the entire original training set using these best hyperparameters, and then get your final, unbiased performance score from the test set.

3. Manual Hyperparameter Tuning: A From-Scratch Approach

Let's implement this logic. In the spirit of your learning style, we will first build a manual loop to tune a single hyperparameter.

The following resource walks through an example of tuning the regularization parameter alpha for a linear model. It plots the performance on both the training and validation sets, which is an excellent way to diagnose overfitting and underfitting.

Your task now is to apply this logic to one of your own models. Let's choose your Regularized Linear Regression model and tune its lambda (or alpha) hyperparameter.

Your Implementation Task:

1. Take your full training dataset (X_train, y_train).
2. Use your train_test_split function from Module 1 to split it again, creating a (X_train_small, y_train_small) and a (X_val, y_val). A 80/20 split is a good starting point.
3. Create a list of lambda values to test (e.g., [0.001, 0.01, 0.1, 1, 10, 100]).
4. Write a for loop that iterates through your list of lambdas.
5. Inside the loop:
   • Instantiate your Linear Regression model with the current lambda.
   • Train it on (X_train_small, y_train_small).
   • Calculate the R-squared score (or MSE) on y_val and store it.
6. After the loop, identify the lambda that produced the best validation score. This is your optimal hyperparameter from this search!

This manual process gives you a deep, practical understanding of how tuning works.

4. Robust Tuning with K-Fold Cross-Validation

A single validation set works, but the results can be sensitive to how you split the data. A more robust and widely-used method is k-fold cross-validation, which you implemented in Module 1.

Instead of one validation set, we create k of them. For each hyperparameter combination, we train and evaluate the model k times, and the performance metric is the average score across all k folds.

This process, known as Grid Search with Cross-Validation, is the gold standard for hyperparameter tuning. Let's see how it works before automating it.

GridSearchCV is a powerful tool that automates this entire process. It takes an estimator (your model), a parameter grid (param_grid), and a cross-validation (cv) strategy.

Here is how you would set it up in practice, integrating the Pipeline concept from our last lesson. This is an extremely common and powerful pattern in professional machine learning.

The param_grid is a dictionary where keys are the names of the hyperparameters you want to tune (using the step_name__parameter_name syntax for pipelines) and values are lists of the values you want to try. GridSearchCV will then exhaustively test every single combination.

5. Your Capstone Tuning Task

It's time to apply these concepts to your project. Your goal is to tune at least one key hyperparameter for three of the models you built. This will involve adapting the logic from your manual tuning exercise to work with your cross-validation code.

Instructions:

1. Choose Your Models and Hyperparameters: Select at least three models from your collection (e.g., SVM, Random Forest, Logistic Regression) and pick one key hyperparameter for each to tune.

   • Logistic Regression: lambda (regularization strength)
   • SVM: C (soft-margin constant)
   • Random Forest: max_depth
   • Gradient Boosting: n_estimators (number of trees)
   • MLP: learning_rate

2. Implement the Tuning Logic: For each chosen model and hyperparameter:

   • Define a list of values to test for the hyperparameter.
   • Use your k-fold cross-validation implementation from Module 1.
   • Create a loop that iterates through your list of hyperparameter values.
   • Inside this loop, run your k-fold cross-validation logic. The performance for that hyperparameter value is the average score across all k folds.
   • Keep track of the scores for each hyperparameter value.
   • After the loop, identify the hyperparameter value that yielded the best average cross-validation score.

3. Update and Re-run Your Evaluation:

   • Take the best hyperparameter values you found for each model.
   • Update your "algorithm gauntlet" script from the previous lesson to use these new, optimized hyperparameters instead of the defaults.
   • Re-run the evaluation on your chosen benchmark datasets.

4. Analyze the Results: Did the tuned models perform better on the final test set than their default versions? Briefly summarize your findings.

This exercise is the culmination of your work, combining data splitting, model training, evaluation, and now, optimization.

Conclusion

You have now mastered one of the most critical skills in applied machine learning: hyperparameter tuning. You understand not just how to do it, but why it's done, and the importance of using a validation set or cross-validation to do it correctly.

Key Takeaways:

• Hyperparameters are settings that control the learning process and must be set before training.
• Tuning should always be done using a validation set or cross-validation on the training data, never the test set.
• A manual loop is a great way to understand the mechanics of tuning.
• Grid Search with Cross-Validation is a robust and systematic method for finding optimal hyperparameter combinations.
• Tools like scikit-learn's GridSearchCV automate this process efficiently, especially when combined with Pipeline.

Preview of the Next Lesson:
This was the final implementation-heavy lesson. You now have trained, evaluated, and tuned a whole suite of supervised learning models. The final step in this capstone is to bring everything together. In our last lesson, you will generate a comparative report that analyzes the performance, training time, and characteristics of each algorithm, concluding your comprehensive study.