Model Evaluation & Threshold Tuning for Business Costs

1. Why the Default Threshold is a Business Risk

When we use a classifier's .predict() method, it typically uses a default probability threshold of 0.5. Any prediction with a probability of default > 0.5 is classified as a default (1), and anything <= 0.5 is classified as non-default (0).

This 50/50 split is often arbitrary and rarely optimal in a real-world business setting. It implicitly assumes that the cost of a False Positive is the same as the cost of a False Negative. In our loan default scenario, this assumption is dangerously wrong.

To understand why, let's analyze the costs associated with each type of error.

As the article makes clear:

• False Negative (FN): We predict 'non-default', but the customer does default. Cost: High. The bank loses the entire principal of the loan, plus administrative costs.
• False Positive (FP): We predict 'default', but the customer would have paid it back. Cost: Low. The bank loses out on the potential interest income from that single loan.

Losing the entire loan amount is far worse than losing potential interest. Therefore, our evaluation and decision-making process must reflect this cost imbalance. By adjusting the classification threshold, we can tune the model's behavior to be more cautious, thereby minimizing the number of costly False Negatives.

2. Strategy 1: Simulation to Maximize Business Value

One powerful way to find an optimal threshold is to simulate the business impact of various threshold choices. We can create a "strategy table" that explores how changing our loan acceptance criteria affects key business indicators.

This involves introducing a few business-centric metrics:

• Acceptance Rate: The percentage of loan applications that our model approves. A lower threshold means we are stricter, approving fewer people, leading to a lower acceptance rate.
• Bad Rate: The percentage of approved loans that end up defaulting. This is our primary risk indicator.
• Estimated Value: A simulated profit/loss calculation based on the number of accepted loans, the bad rate, and the average loan amount. This metric directly estimates the financial outcome of our strategy.

The following resource provides a superb walkthrough of this entire simulation process, from defining the metrics to creating and interpreting the final results.

The Core Logic

The analysis you just read follows these steps:

1. Use the trained XGBoost model to get the predicted probabilities of default for the test set.
2. Define a range of potential Acceptance Rates to test (e.g., from 95% down to 5%).
3. For each Acceptance Rate, calculate the corresponding probability threshold. For example, an acceptance rate of 80% means we approve the 80% of applicants with the lowest predicted probability of default. The threshold is the highest probability within this approved group.
4. Apply this threshold to classify applicants as 'accepted' or 'rejected'.
5. For the 'accepted' group, calculate the Bad Rate (the actual default rate within this group) and the Estimated Value.
6. Plot Estimated Value vs. Acceptance Rate to find the peak. This peak represents the strategy that maximizes our simulated financial return, and it gives us the optimal threshold to use.

The chart of "Estimated Value by Acceptance Rate" in the article is the key deliverable of this analysis. It provides a clear, data-driven recommendation for the business on which threshold to adopt to maximize profitability.

3. Strategy 2: Automated Cost-Sensitive Tuning

While simulation is intuitive, a more direct and automated approach is to define a custom scoring function that represents our business costs and then use a tool to find the threshold that optimizes it. scikit-learn provides an excellent tool for this: TunedThresholdClassifierCV.

First, we need to create a function that calculates a "score" based on our cost matrix. We want to maximize this score. Let's assign a "gain" to each outcome:

• True Negative (TN): Approved a good loan. Gain = + interest (for simplicity, we can ignore this and set it to 0, focusing only on costs).
• True Positive (TP): Correctly rejected a bad loan. Gain = 0.
• False Positive (FP): Rejected a good loan. Gain = -1 (representing the lost opportunity cost).
• False Negative (FN): Approved a bad loan. Gain = -5 (representing a much larger loss).

The scikit-learn documentation provides a perfect template for this.

Now that you understand how to define the cost function, let's see how to apply it efficiently. Since we have already trained our XGBoost model in the previous lesson, we don't need to retrain it repeatedly during the threshold search. We can use the cv='prefit' option.

Implementation

Here's how you would implement this with our trained XGBoost model.

import numpy as np
from sklearn.metrics import make_scorer, confusion_matrix
from sklearn.model_selection import TunedThresholdClassifierCV




# Assume 'xgb_model' is your trained XGBoost classifier from the previous lesson
# Assume 'X_val', 'y_val' is a validation set held out for tuning the threshold




# 1. Define the custom scoring function based on our cost matrix
def credit_gain_scorer(y_true, y_pred, **kwargs):
    cm = confusion_matrix(y_true, y_pred)



    # Gain Matrix: [[TN, FP], [FN, TP]]
    # We define gains as: TN=0, TP=0, FP=-1, FN=-5
    gain_matrix = np.array([[0, -1], [-5, 0]])
    return np.sum(cm * gain_matrix)




# Make it a scikit-learn scorer
business_scorer = make_scorer(credit_gain_scorer, greater_is_better=True)




# 2. Use TunedThresholdClassifierCV on the pre-trained model
# We use a validation set (X_val, y_val) to find the threshold to avoid overfitting.
tuned_xgb = TunedThresholdClassifierCV(
    estimator=xgb_model,          # Our already trained XGBoost model
    scoring=business_scorer,      # The business metric to maximize
    cv="prefit",                  # IMPORTANT: Tells it not to re-train the estimator
    refit=False,                  # We don't need to refit the final model
    store_cv_results=True
)




# Fit it on the validation data to find the best threshold
tuned_xgb.fit(X_val, y_val)




# 3. Analyze the results
print(f"Optimal threshold based on business cost: {tuned_xgb.best_threshold_:.4f}")




# Compare the score with the default 0.5 threshold vs. the tuned one
default_preds = (xgb_model.predict_proba(X_val)[:, 1] > 0.5).astype(int)
tuned_preds = tuned_xgb.predict(X_val)

print(f"Business score with default 0.5 threshold: {credit_gain_scorer(y_val, default_preds)}")
print(f"Business score with optimal threshold: {credit_gain_scorer(y_val, tuned_preds)}")

You would find that the optimal threshold is likely much lower than 0.5 (e.g., 0.2-0.3). This means the model becomes much stricter, flagging more applicants as potential defaults. While this increases False Positives, it significantly cuts down on the very costly False Negatives, leading to a better overall business score (a smaller loss).

Conclusion

In this lesson, we elevated our model evaluation from a purely statistical exercise to a business-centric optimization process. You learned how to align your model's decision-making with the financial realities of the business problem.

Key Takeaways:

• The Default Threshold is Arbitrary: The standard 0.5 probability threshold is rarely optimal for business problems with asymmetric costs.
• Threshold Tuning is a Business Lever: Adjusting the classification threshold is a powerful way to control the trade-off between risk and opportunity (e.g., False Negatives vs. False Positives).
• Connect to Business KPIs: Techniques like simulation (strategy tables) and custom cost functions allow you to translate model performance directly into business metrics like profit, loss, or risk exposure.
• Automate with the Right Tools: scikit-learn's TunedThresholdClassifierCV provides an efficient, robust way to find an optimal threshold based on a custom, cost-sensitive metric.

Next Steps:

We have now selected our final model (XGBoost) and optimized its decision threshold to maximize business value. However, it remains a "black box." For a bank to trust and deploy this model, especially in a regulated environment, they need to understand why it makes certain decisions.

In our next and final lesson, Generate and interpret model explanations (e.g., using SHAP) to communicate key risk drivers to business stakeholders, we will learn how to open this black box and extract actionable insights, making our model not just predictive, but also explainable.