Evaluation flow for batch evaluations

reference brazil Updated Sep 10, 2026 4 min read ai-eval-flow-batch.md
Raw file servicenow.com

Batch evaluation enables Eval admins to evaluate up to 100 completed virtual agent conversations at once, based on a saved query.

Flow name: Execute Batch Evaluation.

The flow creates evaluation records and invokes Now Assist skills for each eligible conversation, mirroring the single-conversation evaluation logic, but at scale. It enforces HR scope exclusions, topic/category validation, transcript construction rules, early live-agent exclusions, and asynchronous scoring through skills.

Batch evaluations are performed using the following logic:

Trigger

  • Table: Evaluation set [sn_na_conv_eval_evaluation_set]
  • Condition: State changes to In Progress and Evaluation type = Conversation

Inputs

  • Evaluation Set record with:
    • Query filter: A query that targets conversations to be evaluated (for example, sys_cs_conversation filters).
    • Evaluation type: Conversation
    • State: In Progress (to start)
  • LLM/Skills: Chat Topic Classifier, plus the evaluation skills listed after this.

High-level behavior

  • Reads the query filter and randomly samples up to 100 conversations.
  • Skips already-evaluated conversations.
  • Excludes HR-scoped interactions.
  • Uses Chat Topic Classifier to validate evaluation eligibility and extracts Topic and Category.
  • Builds a transcript with controlled inclusion of Knowledge articles and catalog sources, and applies early live agent exclusions.
  • Creates an Evaluation record and asynchronously invokes all selected evaluation skills, writing scores and rationale to metrics.

Sequence of execution:

Action 1: If the query filter isn’t empty

  • Purpose: Guard clause.
  • Logic: Look up the Evaluation Set record and check the query filter field.
  • If the query filter is present: Proceed to Action 2.
  • If empty: Stop and optionally log No query provided.

Action 2: Randomize conversations

  • Purpose: Select a bounded, random sample of conversations from the provided query.
  • Logic:
    • Execute the query to get matching conversation records.
    • Randomly select up to 100 conversations.
      • If >100 matches, cap at 100.
      • If <100, select all.
    • Validate the query; if invalid, return false and an empty or partial array.
  • Outputs:
    • success: true/false
    • conversation_ids: array of sys_ids (max 100)
  • If success = true: Proceed to Action 3; otherwise, stop and log the validation error.

Action 3: Look up the evaluation table to check prior evaluation

  • Purpose: Avoid duplicate evaluations.
  • Logic: For each conversation sys_id, check sn_na_conv_eval_evaluation for existing records indicating that it's already evaluated or is in progress (implementation choice: state not in canceled/failed).
  • If not previously evaluated: Proceed to Action 4 for that conversation.
  • If already evaluated: Skip this conversation, optionally log Already evaluated.

Action 4: Look up the interaction record

  • Purpose: Enforce HR scope exclusion.
  • Logic: Resolve the interaction related to the conversation. If its application scope contains hr, skip the conversation.
  • If the scope doesn’t contain hr: Proceed to Action 5.

Action 5: buildTranscript

  • Purpose: Construct the final, minute-level transcript and determine downstream skill set and guardrails.
  • Steps:
    • Aggregate all conversation messages.
    • Tag user messages as [User]: and virtual agent messages as [Virtual Agent]:.
    • Knowledge articles:
      • If genius results reference Knowledge articles, query the Knowledge article and replace the genius snippet with the entire article body.
      • Annotate with [Virtual Agent]: Help articles for user query: and wrap content within Article_Start and Article_End.
      • Constraints:
        • If the KB is HR-scoped or inaccessible, don't evaluate (skip conversation).
        • Truncate the article body to a maximum of 10,000 words.
        • If the KB content source is attached files (PDF/Word/Txt), fall back to the genius result instead of full file content.
    • Catalog Items:
      • If genius results reference catalog items, query sc_cat_item and build a string: catalog name, short description, description.
      • Annotate with [Virtual Agent]: Please choose one of the below options: and include citation order.
    • Live Agent Exclusions:
      • If the first user message requests a live agent, skip evaluation.
      • If a live agent is invoked within the first 120 words, skip evaluation.
  • Outputs:
    • ExecuteEvaluation: true/false (post-guardrail outcome)
    • Chat transcript
    • Knowledge articles referred
    • Catalog items referred
    • First live agent occurrence: Sys_id of the conversation message (if present)
    • Skills to invoke:
      • Coherence Chat Evaluation
      • Conciseness Chat Eval
      • Context Retention
      • Inadequate Slot Filling Chat Eval
      • Intent Accuracy Chat Eval
      • Smooth Flowing Conversation Chat Eval
      • Truthfulness Hallucination Chat Eval
    • Additional logs
  • If ExecuteEvaluation = true: Proceed to Action 7; otherwise, skip the conversation.

Action 6: If Block

  • Purpose: Branch to record creation.
  • Logic: If ExecuteEvaluation from Action 6 is true, go to Action 8.

Action 7: Chat Classifier Eval

  • Purpose: Validate whether the conversation should be evaluated and extract high-level labels.
  • Logic:
    • Build a lightweight transcript from sys_cs_message for classification input.
    • Invoke Chat topic classifier skill with the transcript.
    • Receive:
      • Execute evaluation: true/false
      • Topic Name
      • Category: IT or HR
  • If Execute evaluation = true: Proceed to Action 6.
  • If false: Skip conversation and log the classifier decision.

Action 8: Create or Update evaluation record

  • Purpose: Persist an evaluation entry for this conversation.
  • Table: sn_na_conv_eval_evaluation
  • Field population:
    • Document conversation: Conversation reference
    • State: processing
    • Topic: from Action 5
    • Category: from Action 5
    • KB Referred: from Action 6
    • Catalog Referred: from Action 6
    • First live agent occurrence: from Action 6
    • Type: chat summarization
    • User: initiating user for the conversation
    • Message log: Additional logs from Action 6
  • On success: Proceed to Action 9.

Action 9: For Loop over skills

  • Purpose: Execute each selected evaluation skill.
  • For each skill in the list from Action 6:

    • Action 10: invokeApiDefinition

      • Inputs: Skill Name, Conversation, Transcript, Evaluation Id
      • Behavior:
        • Invoke the Now Assist skill asynchronously.
        • The post-processor writes results into sys_generative_ai_response_validator.
        • Extract JSON response fields:
          • Score
          • Reason for Score
          • Examples supporting the reasoning
        • Create child metric records in sn_na_conv_eval_evaluation_metrics linked to the parent evaluation.

    • Action 11: Wait

      Pause seven seconds before proceeding to the next skill to manage rate limits or throttling.

Search documentation
Search document titles and file paths on the brazil branch