Model Evaluation: Predicted vs. Actual

The Core of Model Evaluation

At its heart, evaluating a regression model is simple: you compare the value the model predicted with the actual value that was observed. The difference between these two is called the residual or error.

$\text{Error}=\text{Actual Value}-\text{Predicted Value}$

A model with smaller errors is, naturally, a better model.

Visual Evaluation: The Scatter Plot

One of the most intuitive ways to assess performance is to create a scatter plot of the actual values against the predicted values.

In a good model, you want to see the points clustered tightly around this imaginary diagonal line. If the points are widely scattered, it indicates the model's predictions are often far from the actual values, meaning it has a high error rate.

Key Performance Metrics for Regression

While a visual plot is a great start, we need objective, numerical metrics to quantify a model's performance and compare it to others. Simply summing the errors isn't useful, as positive errors (under-predictions) and negative errors (over-predictions) would cancel each other out.

Instead, we use metrics that handle this issue:

• Mean Absolute Error (MAE): This is the average of the absolute differences between the actual and predicted values. It's easy to interpret as it's in the same units as your target variable.
  • Interpretation: "On average, our model's predictions are off by [MAE value] units."
• Root Mean Square Error (RMSE): This is the square root of the average of the squared errors. By squaring the errors, this metric penalizes larger errors more heavily than smaller ones. Like MAE, it's also in the original units of the target variable.
  • Interpretation: RMSE is a standard way to measure the error of a model. A lower RMSE is better. It's particularly useful when large errors are especially undesirable.

For both MAE and RMSE, the lower the value, the better the model's performance.

Calculating Performance Metrics in Alteryx

There are two primary ways to evaluate model performance in Alteryx: a "manual" approach that reinforces the concepts, and a dedicated tool that automates the process.

1. The Manual Approach (for Deeper Understanding)

To truly understand what these metrics represent, it's helpful to build the calculation yourself using standard Alteryx tools. This process leverages the skills you already have from your Core Certification.

The following article from The Data School walks through exactly how to do this.

As the article explains, the key steps are:

1. Score the Data: Use the Score tool on your validation/test dataset to get the Score (predicted) column.
2. Calculate Error: Use a Formula tool to create a new column for the squared error: POW([Actual_Value] - [Score], 2). Squaring ensures all error values are positive.
3. Aggregate: Use a Summarize tool to find the Sum of the squared error column and the Count of records.
4. Final Calculation: Use another Formula tool to calculate the final metric. For RMSE, the formula would be SQRT([Sum_Squared_Error] / [Count]).

This hands-on method cements the statistical concepts behind model evaluation.

2. The Automated Approach: The Model Comparison Tool

For the Alteryx Advanced exam and most real-world scenarios, you'll use a specialized tool that does all this work for you: the Model Comparison tool.

This powerful tool, found in the Predictive tool palette, is designed specifically to evaluate and compare the performance of one or more models on a test dataset.

Using the Model Comparison Tool:

1. Inputs:

   • M Anchor: Connect the model object(s) from the 'O' output of your predictive tool(s) (e.g., Linear Regression). You can even Union multiple models together to compare them at once.
   • D Anchor: Connect your test or validation dataset (e.g., from the 'V' output of a Create Samples tool). This is crucial for an unbiased evaluation.

2. Outputs:

   • R Anchor: A full report with summary tables and plots.
   • E Anchor: A concise data table containing the key error measures (RMSE, MAE, etc.) for each model.
   • P Anchor: A table of the actual values alongside the predicted values from each model.

For the Advanced Certification, being proficient with the Model Comparison tool is essential. It is the standard, most efficient way to perform model evaluation in Alteryx.

Conclusion

You have now completed the entire linear regression workflow in Alteryx! You can prepare data, train a model, score new data, and—as of today—rigorously evaluate your model's performance.

Key Takeaways:

• Model evaluation for regression involves comparing predicted values to actual values.
• Visual evaluation using an Actual vs. Predicted scatter plot provides a quick, intuitive assessment of model fit.
• RMSE and MAE are the two most common metrics for quantifying regression error. Lower values are better.
• The Model Comparison tool is the standard Alteryx tool for automated performance evaluation. It takes a model object and a test dataset and outputs a comprehensive report and error metrics.

Preview of the Next Lesson:

We've now thoroughly covered regression, which is used for predicting continuous numerical values (like sales, temperature, or price). In the next module, we will shift gears to a different type of predictive modeling: classification. You'll learn how to build models that predict a category or class (e.g., "Yes/No", "Spam/Not Spam", or "Customer Type A/B/C"). We will start by exploring the fundamental concepts of classification.