Optimizing Chatbot Performance

1. High-Level Health Check: The Built-in Analytics Dashboard

Copilot Studio provides a set of out-of-the-box analytics dashboards that offer a quick overview of your copilot's performance. Think of this as the executive summary: it's great for understanding key performance indicators (KPIs) and sharing high-level trends with stakeholders.

Let's start with a brief video that introduces the concept of the analytics dashboard.

The built-in dashboards have evolved to become quite powerful, offering detailed insights without any custom setup.

As you can see from the dashboard, you can quickly assess:

• Summary: Core metrics like total sessions, engagement rate, and resolution rate.
• User Satisfaction: CSAT scores if you have implemented post-chat surveys.
• Topic Analytics: Which topics are triggered most, their completion rates, and which have high abandonment or escalation rates. This is crucial for pinpointing problematic or confusing conversation flows.
• Billing: Monitor your billed sessions against your capacity.

While excellent for an overview, these dashboards have limitations for a developer. You can't perform complex ad-hoc queries, correlate data with external systems, or define custom metrics. For that, we need a more powerful tool.

2. Deep-Dive Analysis with Azure Application Insights

For professional-grade monitoring, you can integrate your copilot with Azure Application Insights. This is the same powerful APM service you might use to monitor web apps, APIs, and other services in Azure. It provides a rich, queryable telemetry store for deep analysis.

This integration elevates your monitoring capabilities from simply viewing pre-built reports to actively interrogating the data to answer specific, complex questions.

As you've read, the process is straightforward:

1. Create an Application Insights resource in your Azure subscription.
2. Copy its Connection String.
3. In Copilot Studio, navigate to Settings > General, paste the string into the Application Insights section, and enable the desired logging options.

Once connected, your copilot will start streaming detailed telemetry events to your Application Insights instance.

3. Querying Telemetry with Kusto Query Language (KQL)

The real power of Application Insights comes from the Kusto Query Language (KQL). If you're familiar with SQL, you'll pick up KQL quickly. The main syntactical difference is its pipe-based structure, where you chain together operators to filter and shape the data, similar to a PowerShell or Linux pipeline.

All copilot telemetry is primarily stored in the customEvents table. The most important details of each event—like the topic name, channel, and user messages—are stored in a JSON-like field called customDimensions.

Let's dive into the practical application of KQL with some essential queries. We'll use two excellent resources that provide a range of examples.

Essential Best Practice: Filtering Test Data

Before you run any analysis, it's critical to separate real user data from your own testing activities. Fortunately, Copilot Studio automatically tags test canvas interactions. You should almost always include this filter in your queries.

// Filter out conversations from the test canvas
customEvents
| where tostring(customDimensions.designMode) == "False"

Core Analytical Queries

Now let's explore some queries that help you answer key business questions. We will use queries from both the Microsoft Learn article and the blog post.

Let's synthesize what you've read into a few patterns.

1. Which topics are most popular?
This helps you focus your optimization efforts on high-traffic areas.

// Summarize the count of conversations per topic
customEvents
| where tostring(customDimensions.designMode) == "False"
| where name == "BotMessageSend" // An event that fires when the bot sends a message
| extend topicName = tostring(customDimensions.topicName)
| where isnotempty(topicName)
| summarize count() by topicName
| order by count_ desc
| render barchart

2. What are the most common errors?
This is invaluable for debugging. The following query intelligently parses the error code from the message text.

3. Which topics are the slowest?
Performance is key to user experience. The next query is a powerful example of data transformation within KQL. It calculates the time difference between consecutive events in a session to measure the duration of each step. Given your C# and data processing background, you'll appreciate the logic here.

Conclusion

Congratulations on completing the module on Microsoft Copilot Studio! In this lesson, you've bridged the gap between deploying a copilot and ensuring its long-term success through continuous monitoring and improvement.

Key Takeaways:

• Two Tiers of Monitoring: Use the built-in analytics for high-level KPIs and stakeholder reporting, and leverage Azure Application Insights for deep, developer-centric analysis.
• KQL is Your Power Tool: Application Insights combined with KQL provides a virtually limitless capability to query, analyze, and visualize your copilot's performance and user interactions.
• Analysis Drives Improvement: The goal of monitoring is to find actionable insights. By identifying popular topics, common errors, and performance bottlenecks, you can prioritize your development efforts to have the greatest impact on user experience.

What's Next?

We are now moving to a new and exciting area of the Power Platform: AI Builder. While Copilot Studio excels at conversational AI, AI Builder provides a suite of pre-built and custom-trainable AI models for different business tasks.

In our next lesson, you will learn how to select the appropriate AI Builder model type for a given business problem, exploring models for form processing, object detection, prediction, and more. This will further expand your toolkit for infusing intelligence into your Power Platform solutions.