Custom Criteria

Unlike managed criteria, which are predefined and standardized, custom criteria allow you to create evaluation metrics aligned perfectly with your unique business objectives. By specifying a custom name, checklist items, and priority levels, you can measure any aspect of conversation performance—such as compliance with a newly introduced policy, alignment with brand voice, or handling of a specific type of customer query.

Who Benefits:

• Project Managers & Team Leads: Craft criteria that target current priorities or new training focuses, tailoring feedback to evolving team goals.
• Sales & CX Teams: Introduce custom checks for domain-specific issues (e.g., product feature explanations) or emerging customer concerns.
• Business Managers & Strategists: Align criteria with strategic initiatives, ensuring that evaluations reflect key performance indicators and tactical objectives.
• Compliance Officers: Rapidly introduce criteria for new regulatory changes, ensuring that calls meet evolving standards.

Value Proposition:

• Tailored Evaluations: Align assessments with your unique goals rather than relying solely on off-the-shelf metrics.
• Dynamic Adjustments: Easily adjust priorities, add or remove checklist items, and refine definitions as business needs shift.
• Efficient Integration: Structured JSON output feeds directly into analytics tools, enabling quick iteration and refinement.

Data Dictionary

Objective:

Create a custom criterion with a name, a set of checklist items each with a priority (high, medium, low), and optionally a description and tags. Evaluate a conversation against this criterion by scoring each item, computing a weighted score, and returning a standardized JSON output.

Required JSON Schema:

{
  "criteria": [
    {
      "name": "<Custom Criterion Name>",
      "score": <number>,
      "summary": "<string>",
      "feedback": {
        // optional suggestions or notes
      }
    }
  ]
}

Data Dictionary:

If Criterion Not Applicable or No Data:
If no relevant aspects match the custom criterion or no scoring is possible, return a zero or omit the criterion. If no criteria apply at all, return an empty criteria array.

Determining Score & Thresholds

Confidence and Scoring:

• Define Your Checklist & Priorities:
  A custom criterion might be stored in a configuration or database, for example:

  JSON

  {
    "id": "Custom.Brand_Voice_Alignment",
    "name": "Brand Voice Alignment",
    "type": "custom",
    "description": "Evaluates how well the agent adheres to the brand's approved terminology and tone.",
    "tags": ["brand", "custom"],
    "checklist": [
      { "text": "Use of approved terminology?", "priority": "high" },
      { "text": "Avoidance of restricted phrases?", "priority": "medium" },
      { "text": "Consistent tone of voice?", "priority": "high" },
      { "text": "Correct handling of brand-specific product names?", "priority": "low" }
    ]
  }

• Priority Weights (example):

  • High = 3
  • Medium = 2
  • Low = 1

• Scoring Method:

  1. Assign partial scores (0.0-1.0) to each item based on transcript evidence.
  2. Compute weighted average using priorityWeights.
  3. Convert weighted average (0.0-1.0) into a 0.0 to 100.0 scale.
  4. Return the final score, a summary, and optional feedback.

Iterative Improvement & Calibration:

• Initial Testing: Validate scoring against known calls or scenarios. Adjust checklist items or priorities as needed.
• Refine Priorities: As organizational goals change, update priority weights or add/remove items.
• Stakeholder Feedback: Involve team leads or compliance officers to refine criteria and ensure relevance.
• Continuous Updates: As language patterns, product lines, or messaging strategies evolve, revise custom criteria definitions seamlessly.

Prompt Construction & Instructions

Role Specification & Reiteration:

• Present the system as a “highly experienced assistant” evaluating a custom criterion.
• Reiterate scoring methods, priority weights, and formatting rules.
• Mention that the criterion definition (name, checklist) can be provided at runtime.

No Hallucination:

• Only score based on the transcript and the provided custom criterion definition.
• If uncertain, assign moderate or low partial scores or omit uncertain fields.

Strict Formatting:

• Return only JSON.
• If no improvement needed, feedback can be empty or omitted.

Prompt for Implementation

System Message (Role: System):
"You are a highly experienced assistant evaluating a custom criterion. You have:

• A transcript
• A custom criterion definition (name, checklist with priorities, optional description/tags)
• Instructions specifying scoringMethod and priorityWeights

Your task:

1. For each checklist item, assign a partial score (0.0-1.0) based on the transcript.
2. Compute a weighted average using provided priorityWeights.
3. Convert that average to 0.0-100.0 for score.
4. Provide a summary and optional feedback.
5. Return only JSON, no extra commentary.

Instructions & Variables (example):

{
  "instructions": {
    "scoringMethod": "weighted_average",
    "priorityWeights": {
      "high": 3,
      "medium": 2,
      "low": 1
    }
  },
  "criterion": {
    "id": "Custom.Brand_Voice_Alignment",
    "name": "Brand Voice Alignment",
    "type": "custom",
    "description": "Evaluates adherence to brand-approved tone and terminology.",
    "checklist": [
      { "text": "Use approved terminology?", "priority": "high" },
      { "text": "Avoid restricted phrases?", "priority": "medium" },
      { "text": "Consistent tone?", "priority": "high" },
      { "text": "Correct product name usage?", "priority": "low" }
    ]
  }
}

User Message (Role: User):
"Analyze the following transcript against the 'Brand Voice Alignment' custom criterion:

[TRANSCRIPT_JSON][TRANSCRIPT_JSON]"

Expected Output Format:

{
  "criteria": [
    {
      "name": "Brand Voice Alignment",
      "score": <number>,
      "summary": "<string>",
      "feedback": {
        // optional suggestions
      }
    }
  ]
}