Optimizing AI Performance with Test Data & Metrics

1. Understanding AI Model Performance Metrics

After AI Builder trains a model, it automatically sets aside a portion of your data as a "test set." It then uses this unseen data to evaluate the model's performance, presenting the results through several key metrics.

Performance Grades

The most immediate feedback you get is a performance grade: A, B, C, or D. This grade provides a quick assessment of your model's quality.

As you read, you'll see that a 92% accuracy score could be an 'A' grade for a problem with a 50/50 split but only a 'B' grade for a problem with an 80/20 split. This is because the model's improvement over a random guess is much greater in the first case. This nuance is key to a professional evaluation.

A 'D' grade is a significant warning sign. It indicates either:

• Underfitting: The model performs worse than a random guess. It hasn't learned the patterns in your data.
• Overfitting: The model performs perfectly or near-perfectly (e.g., >99%). This is a classic machine learning problem. It often means the model has "memorized" the training data, including its noise and idiosyncrasies, and will likely fail when it encounters new, real-world data. A common cause is including a data column that directly gives away the answer (e.g., predicting if a loan is 'Paid' when your data includes an 'IsPaid' column).

Core Performance Scores

Beyond the letter grade, AI Builder provides specific scores.

• Accuracy Score: Used for models that classify or identify things (e.g., prediction, object detection, category classification). It's the percentage of correct predictions made on the test dataset. $$\text{Accuracy}=\frac{\text{Number of Correct Predictions}}{\text{Total Number of Predictions}}$$
• R-squared: Used for models that predict a number (e.g., estimating the price of a house). This statistical measure indicates the "goodness of fit." A value of 85% means that 85% of the variation in the output values is explained by the model's inputs. A higher R-squared value generally indicates a better fit.
• Confidence Score: This is not a model-wide metric but a score assigned to each individual prediction the model makes at runtime. For example, when a form processing model extracts an invoice number, it will also return a confidence score (e.g., 98%) indicating how certain it is about that specific extraction. Low confidence scores on published models are a strong signal that you need to go back and retrain with more examples.

2. How to Evaluate a Model in Practice

Let's move from theory to practice. When you open a trained model in AI Builder, you're presented with a details page that is your primary tool for evaluation. We will use a Form Processing model as an example because its evaluation screen is particularly detailed, but the principles apply across all model types.

Once you're in the evaluation panel, you can drill down from the overall score to see how the model performs on individual fields, tables, or collections.

This detailed view is where your analysis begins. An overall score of 90% might look good, but it could be hiding a critical field that is only being extracted correctly 50% of the time.

This screen is your starting point for debugging. By identifying the specific fields, tables, or collections with low scores, you can focus your improvement efforts where they are needed most.

3. A Developer's Guide to Improving Model Performance

Once you've evaluated your model and identified its weaknesses, the next step is to improve it. This is an iterative process: you'll apply a change, retrain the model, and evaluate it again. Drawing from your background, you can think of this as a "debug, fix, recompile, and test" cycle.

Here are the most effective strategies you can use.

1. Provide More (and More Varied) Data

This is the single most effective way to improve most models. The model needs to see enough examples to learn the patterns reliably.

• Quantity: Aim for well over the minimum requirement. If the minimum is 15 images per object, try for 50+. If it's 50 rows of data, aim for 1,000+.
• Variety: This is just as important as quantity. Your training data must reflect the real-world data your model will encounter. For object detection, this means different angles, lighting, and backgrounds. For form processing, this means different layouts (which should be separated into collections).

2. Verify Your Labels and Tags

This is the "Garbage In, Garbage Out" principle. If your training data is incorrectly labeled, the model will learn the wrong things.

• Go back through your low-performing items.
• In a form processing model, ensure the correct text is tagged for each field.
• In an object detection model, check that bounding boxes are tight and correctly labeled.

3. Balance Your Data Distribution

A model trained on imbalanced data will be biased towards the majority case.

• For a prediction model, if you're predicting Yes/No and 98% of your training data is Yes, the model will struggle to ever predict No. You may need to find more No examples or use techniques to balance the dataset.
• This is less of a concern for form processing or object detection, but you should still ensure all your labels have a sufficient number of examples.

4. Review Top Influencers and Refine Columns (Prediction Models)

For prediction models, the model details page shows you which data columns had the most influence on the outcome.

• Use this for a sanity check. If you are predicting customer churn and the top influencer is CustomerID, something is wrong. This suggests data leakage or an unwanted correlation.
• Add or remove columns. If the model is underperforming, perhaps you are missing key data. Could you add columns with more information (e.g., adding customer tenure to a churn prediction model)? Conversely, removing irrelevant or noisy columns can sometimes help the model focus on the important signals.

Conclusion

In this lesson, you've moved beyond simply building AI models to professionally assessing and refining them. This iterative cycle of evaluating performance and strategically improving the training data is fundamental to creating AI solutions that are robust and trustworthy.

Key Takeaways:

• AI Builder provides performance grades (A-D) and scores (Accuracy, R-squared) to evaluate models.
• Performance is relative; a good score depends on your data's baseline distribution. A 'D' grade with very high accuracy indicates overfitting.
• Evaluation involves drilling down into detailed reports to find specific areas of weakness (e.g., low-performing fields).
• Improvement is an iterative process focused on enhancing the training data:
  • Adding more and varied examples.
  • Correcting tags and labels.
  • Balancing data distribution.
  • Refining the features/columns used for training.

What's Next?

Now that you know how to build a custom model and ensure it performs well, you are ready to put it to work. In our next lesson, you will learn how to integrate a trained AI Builder model into a Power App to process user input or images, turning your intelligent model into a functional business application.