Insights into failed scenarios
Analyze failed scenarios in an automated evaluation and see the recurring failure patterns behind them, to prioritize what to fix without reviewing every failed scenario one by one.
View top failure patterns
After an automated evaluation with failed scenarios finishes running, Assistant Designer analyzes the failed scenarios and clusters them into failure patterns. The Top issues detected section on the evaluation results page lists these patterns so you can see where your assistant is breaking down across the whole data set.
[Omitted image "severity-insights-01.png"] Alt text: Evaluation results page showing the metric scores and the Top issues detected section.
By default, up to four failure patterns are shown, ranked by severity and then by frequency, from highest to lowest. You can:
- Sort the list by Severity or Frequency.
- Select Show issues to expand the list and view every failure pattern that was detected.
- Select View scenarios next to a failure pattern to see the specific scenarios where it occurred.
View scenarios for a failure pattern
Select View scenarios from either the Top issues detected list or a metric's issue chart to open the Scenarios list, filtered to the scenarios where that failure pattern occurred. If you select more than one failure pattern, the list shows the scenarios where any of the selected patterns occurred.
[Omitted image "severity-insights-03.png"] Alt text: Scenario list filtered by issue types detected.
The list includes:
- Scenario: The scenario name, which links to the scenario's transcript.
- Issues detected: The failure patterns found in that scenario, grouped by metric category.
By default, scenarios are sorted by severity and then alphabetically. Select a column header to sort by Scenario or Issues detected instead.
View a scenario transcript
Select a scenario name from the Scenario list to open its transcript.
[Omitted image "severity-insights-04.png"] Alt text: Scenario transcript with the Why was this flagged panel.
The transcript page shows:
- The full conversation transcript. Select Download to save the transcript as a CSV file.
- Why was this flagged?: Every issue detected in the scenario, along with an explanation, sorted by severity from highest to lowest.
Note: An issue tied to a per-conversation metric isn't associated with a specific turn in the transcript. An issue tied to a per-turn metric is labeled at the turn where it occurred, when possible.
Select Back to list of scenarios to return to the filtered scenario list.
Parent Topic:Testing assistant conversations
Related topics
--- title: Insights into failed scenarios description: Analyze failed scenarios in an automated evaluation and see the recurring failure patterns behind them, to prioritize what to fix without reviewing every failed scenario one by one. locale: en-US canonical_url: https://www.servicenow.com/docs/r/australia/conversational-interfaces/virtual-agent/severity-insights.html release: australia product: Virtual Agent classification: virtual-agent topic_type: concept last_updated: "2026-10-05" reading_time_minutes: 2 breadcrumb: [Testing assistant conversations, Test and improve, Build conversations, Virtual Agent, Conversational Interfaces] --- # Insights into failed scenarios Analyze failed scenarios in an automated evaluation and see the recurring failure patterns behind them, to prioritize what to fix without reviewing every failed scenario one by one. ## View top failure patterns After an automated evaluation with failed scenarios finishes running, Assistant Designer analyzes the failed scenarios and clusters them into failure patterns. The **Top issues detected** section on the evaluation results page lists these patterns so you can see where your assistant is breaking down across the whole data set. \[Omitted image "severity-insights-01.png"\] Alt text: Evaluation results page showing the metric scores and the Top issues detected section. By default, up to four failure patterns are shown, ranked by severity and then by frequency, from highest to lowest. You can: - Sort the list by **Severity** or **Frequency**. - Select **Show issues** to expand the list and view every failure pattern that was detected. - Select **View scenarios** next to a failure pattern to see the specific scenarios where it occurred. ## View scenarios for a failure pattern Select **View scenarios** from either the **Top issues detected** list or a metric's issue chart to open the **Scenarios** list, filtered to the scenarios where that failure pattern occurred. If you select more than one failure pattern, the list shows the scenarios where any of the selected patterns occurred. \[Omitted image "severity-insights-03.png"\] Alt text: Scenario list filtered by issue types detected. The list includes: - **Scenario**: The scenario name, which links to the scenario's transcript. - **Issues detected**: The failure patterns found in that scenario, grouped by metric category. By default, scenarios are sorted by severity and then alphabetically. Select a column header to sort by **Scenario** or **Issues detected** instead. ## View a scenario transcript Select a scenario name from the **Scenario** list to open its transcript. \[Omitted image "severity-insights-04.png"\] Alt text: Scenario transcript with the Why was this flagged panel. The transcript page shows: - The full conversation transcript. Select **Download** to save the transcript as a CSV file. - **Why was this flagged?**: Every issue detected in the scenario, along with an explanation, sorted by severity from highest to lowest. **Note:** An issue tied to a per-conversation metric isn't associated with a specific turn in the transcript. An issue tied to a per-turn metric is labeled at the turn where it occurred, when possible. Select **Back to list of scenarios** to return to the filtered scenario list. **Parent Topic:**[Testing assistant conversations](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/conversational-interfaces/virtual-agent/evaluations-ad.md) **Related topics** [Testing assistant conversations](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/conversational-interfaces/virtual-agent/evaluations-ad.md) [Customizing severity levels](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/conversational-interfaces/virtual-agent/customize-severity-levels.md) [Evaluation metrics for automated evaluation](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/conversational-interfaces/virtual-agent/evaluation-metrics-auto-eval-ad.md)
| title | Insights into failed scenarios |
| description | Analyze failed scenarios in an automated evaluation and see the recurring failure patterns behind them, to prioritize what to fix without reviewing every failed scenario one by one. |
| locale | en-US |
| canonical_url | https://www.servicenow.com/docs/r/australia/conversational-interfaces/virtual-agent/severity-insights.html |
| release | australia |
| product | Virtual Agent |
| classification | virtual-agent |
| topic_type | concept |
| last_updated | 2026-10-05 |
| reading_time_minutes | 2 |
| breadcrumb | Testing assistant conversationsTest and improveBuild conversationsVirtual AgentConversational Interfaces |