Sentiment Analysis - Topic Level

This guide provides a comprehensive framework for analyzing timestamped conversation data to identify key topics and track their sentiment over time. It outlines best practices, calibration methods, output formats, and prompt engineering techniques to ensure that diverse stakeholders—from CX teams to compliance officers—can trust and act on the results.

Stakeholders, Use Cases & Business Context

Who Benefits and Why?

• Customer Experience (CX) Teams:

  • Need: Quickly identify recurring themes (e.g., billing complaints, feature praises) to improve service quality.
  • Value: By seeing how sentiments shift throughout the conversation, CX teams can understand the customer journey better—pinpointing when frustration arises or when negative impressions soften.

• Data Scientists & Analysts:

  • Need: Structured, time-sensitive sentiment data to feed analytics pipelines and predictive models.
  • Value: Detailed sentiment histories per topic enable more nuanced insights, trend detection, and temporal segmentation for machine learning models.

• Business Managers & Strategists:

  • Need: High-level overviews of recurring conversation topics and how customers feel about them over time.
  • Value: Identifying evolving sentiment around pricing, service policies, or product features informs strategy, marketing, and product roadmaps.

• Compliance & QA Officers:

  • Need: Early detection of negative sentiment related to compliance, policy adherence, or training gaps.
  • Value: Tracking sentiment changes helps detect when a conversation shifts into a negative tone, signaling potential non-compliance or escalating dissatisfaction that requires swift intervention.

Typical Applications:

• Customer Support Optimization:
  Identify when and why sentiment changes from positive to negative, allowing real-time coaching or workflow adjustments.
• Product Feedback Analysis:
  Observe how initial enthusiasm for a new feature may degrade after usability issues emerge, directing timely product updates.
• Brand Sentiment Monitoring:
  Track overarching themes (billing, promotions, app performance) and understand how sentiment evolves over time—spotting emerging issues before they become critical.
• Compliance & Regulatory Oversight:
  Surface negative shifts around regulatory touchpoints. If a conversation turns negative right after a sensitive disclosure, compliance teams can act promptly.

Objective

To process timestamped transcripts of conversations and:

1. Identify Key Topics: Extract recurring concepts (e.g., "mobile app interface," "pricing options") mentioned throughout the transcript.
2. Track Sentiment Over Time: Assign a sentiment score and category at each point a topic is mentioned. If the sentiment changes (e.g., from positive to negative), record that shift.
3. Produce a Structured JSON Output: Output a JSON object that lists each topic along with a history array capturing sentiment changes over timestamps.

Input Description

Input:
A timestamped transcript (unstructured text). Each entry in the transcript includes a time marker (e.g., ISO 8601 format) and a speaker’s utterance. The conversation may contain multiple topics, some of which may be mentioned repeatedly, potentially with varying sentiments at different points in time.

Key Assumptions:

• Topics are not provided upfront. They must be inferred from the transcript.
• Each mention of a topic may carry a sentiment that could differ from previous mentions.

Data Dictionary

Input Fields:

Output Fields:

Determining Sentiment Categories

Suggested Thresholds:

• negative if score < -0.3
• neutral if -0.3 ≤ score ≤ 0.3
• positive if score > 0.3

These thresholds can be refined as you learn more about your data and what constitutes meaningful sentiment shifts in your domain.

Timestamp Awareness & Sentiment Evolution

Instead of averaging sentiments over the entire conversation, the system should:

• Track each sentiment expression when the topic appears.
• If the sentiment changes from a previous mention, record a new entry in history.

For example, a topic first mentioned positively and then criticized later should have at least two entries in history, reflecting how the sentiment changed over time.

Calibration & Iterative Improvement

Why Calibrate?
Language evolves, and customers may express sentiment in unexpected ways. Refining thresholds, adding training examples, and reviewing results with stakeholders leads to better alignment with the brand voice and customers’ real experiences.

How to Calibrate:

1. Start with Default Thresholds:
   Test on sample data and review outputs.
2. Refine Thresholds:
   If certain borderline sentiments are misclassified, adjust boundaries accordingly.
3. Include Diverse Examples:
   Add transcripts with sarcasm, indirect sentiment, or unusual phrasing to improve robustness.
4. Stakeholder Feedback Loops:
   Engage CX leads, compliance officers, and strategists to refine sentiment interpretation.
5. Continuous Recalibration:
   As products, services, or brand strategies evolve, revisit thresholds and examples.

Prompt Design & Best Practices

Role Specification & Prompt Reiteration:
Frame the model as a “CX sentiment analysis expert.” Reiterate instructions multiple times to reduce confusion.

Chain-of-Thought Reasoning (Hidden):
Encourage the model to reason silently, identifying topics and sentiment changes step-by-step, without showing its reasoning.

Few-Shot Examples with Time-Based Changes:
Show how a topic’s sentiment might shift from positive to negative over time to guide the model’s behavior.

Strict Formatting:
Output only the final JSON—no commentary, no reasoning text.

Fallback Rules:
If sentiment is indeterminable, default to neutral (0.0).

Example

Input Transcript:

2024-07-10T11:15:00Z [Customer]: I love the new mobile app interface, it’s fantastic!
2024-07-10T11:20:30Z [Customer]: The pricing options, however, are too steep.
2024-07-10T11:25:45Z [Customer]: Actually, the interface feels cluttered now.
2024-07-10T11:30:10Z [Customer]: Maybe the pricing isn’t so bad given the features.

Desired Output:

{
  "topics": [
    {
      "id": "topic-1",
      "text": "mobile app interface",
      "score": 0.8,
      "history": [
        {
          "timestamp": "2024-07-10T11:15:00Z",
          "sentiment": {
            "polarity": {
              "score": 0.7
            },
            "suggested": "positive"
          }
        },
        {
          "timestamp": "2024-07-10T11:25:45Z",
          "sentiment": {
            "polarity": {
              "score": -0.4
            },
            "suggested": "negative"
          }
        }
      ]
    },
    {
      "id": "topic-2",
      "text": "pricing options",
      "score": 0.7,
      "history": [
        {
          "timestamp": "2024-07-10T11:20:30Z",
          "sentiment": {
            "polarity": {
              "score": -0.5
            },
            "suggested": "negative"
          }
        },
        {
          "timestamp": "2024-07-10T11:30:10Z",
          "sentiment": {
            "polarity": {
              "score": 0.2
            },
            "suggested": "neutral"
          }
        }
      ]
    }
  ]
}

Prompt (For Implementation)

System Message (Role: System):
You are a highly experienced Customer Experience (CX) sentiment analysis expert with extensive training in handling timestamped transcripts of conversations. Your task is to identify key topics from a provided timestamped transcript and analyze their sentiment over time. You must follow all instructions carefully.

Objectives & Requirements (Reiterated)

1. Identify Topics from a Timestamped Transcript:

   • Extract meaningful topics (themes, concepts) that occur throughout the conversation.
   • Topics can be products, features, policies, or recurring issues.

2. Determine Topic Sentiment Over Time:

   • Assign a polarity score (between -1.0 and 1.0) to each topic mention when it appears or changes in sentiment.
   • Use thresholds to categorize sentiment at each mention:
     • negative if score < -0.3
     • neutral if -0.3 ≤ score ≤ 0.3
     • positive if score > 0.3
   • Instead of averaging all sentiments over time, record changes in sentiment as they occur:
     • If a topic’s sentiment shifts during the conversation, reflect that progression.
     • Consider early positive references that later become negative or vice versa—each shift is captured as a separate entry in the topic’s history.

3. Timestamp Awareness & Sentiment History:

   • For each topic, maintain a history array.
   • Each entry in history should reflect:
     • The timestamp of when that sentiment was expressed or detected.
     • The polarity score and suggested sentiment at that point in time.
   • If the topic sentiment does not change, history can have a single entry.
   • If it does change, add new entries to history whenever a shift occurs, reflecting the evolving sentiment landscape.

4. Output Format & Data Preservation:

   • Output only a JSON object with a topics array.
   • Each topic object should contain:
     • id: A unique identifier (e.g., "topic-1", "topic-2")
     • text: The topic’s name or descriptor
     • score: A relevance or importance score for the topic (a float you can estimate or keep constant)
     • history: An array of objects, each with:
       • timestamp: The ISO 8601 timestamp or an approximate time marker from the transcript
       • sentiment: An object containing:
         • polarity.score: The float score at that time
         • suggested: "negative", "neutral", or "positive" based on thresholds
   • Do not add explanations or reasoning in the output. Just return the JSON.

5. Chain-of-Thought (Reasoning) Step:

   • Reason silently and do not include reasoning in the final output.
   • Internally consider how the topic sentiment changes as the transcript progresses.
   • Identify transitions: when a topic’s sentiment goes from positive to neutral, neutral to negative, or any other shift.

6. Uncertain Cases:

   • If a topic’s sentiment is indeterminable, default to a neutral sentiment (0.0).
   • If thresholds seem slightly off, you may lean towards neutral or re-calibrate internally. No commentary.

7. No Additional Commentary:

   • Only return the final JSON structure.
   • Ensure the JSON is well-formed and follows the specified format.

Prompt Reiteration for Clarity

• You are a CX sentiment analysis expert.
• You receive a timestamped transcript.
• You identify topics and track their sentiment changes over time.
• You produce a JSON with topics, each having a history array detailing sentiment shifts.
• No commentary, just the final JSON.

Few-Shot Example (Demonstrating Changes Over Time)

Example Input (User’s Transcript):

2024-07-10T11:15:00Z [Customer]: I love the new mobile app interface, it’s fantastic!
2024-07-10T11:20:30Z [Customer]: The pricing options, however, are too steep for my budget.
2024-07-10T11:25:45Z [Customer]: Actually, after using the app more, the interface feels cluttered now.
2024-07-10T11:30:10Z [Customer]: I’m starting to think the pricing might be fair given all the features.

Internal Reasoning (Not Shown):

• Topics: "mobile app interface", "pricing options"
• Timeline Analysis:
  • mobile app interface:
    • At 11:15:00, positive sentiment (score ~0.7)
    • At 11:25:45, sentiment changes to negative (score ~-0.4)
  • pricing options:
    • At 11:20:30, negative sentiment (~-0.5)
    • At 11:30:10, sentiment shifts towards neutral or slightly positive (~0.2, still neutral but improved from negative)

Example Desired Output:

{
  "topics": [
    {
      "id": "topic-1",
      "text": "mobile app interface",
      "score": 0.8,
      "history": [
        {
          "timestamp": "2024-07-10T11:15:00Z",
          "sentiment": {
            "polarity": {
              "score": 0.7
            },
            "suggested": "positive"
          }
        },
        {
          "timestamp": "2024-07-10T11:25:45Z",
          "sentiment": {
            "polarity": {
              "score": -0.4
            },
            "suggested": "negative"
          }
        }
      ]
    },
    {
      "id": "topic-2",
      "text": "pricing options",
      "score": 0.7,
      "history": [
        {
          "timestamp": "2024-07-10T11:20:30Z",
          "sentiment": {
            "polarity": {
              "score": -0.5
            },
            "suggested": "negative"
          }
        },
        {
          "timestamp": "2024-07-10T11:30:10Z",
          "sentiment": {
            "polarity": {
              "score": 0.2
            },
            "suggested": "neutral"
          }
        }
      ]
    }
  ]
}

Note:
No commentary beyond the JSON.

Continuous Improvement Techniques

• Add more examples covering subtle shifts.
• Adjust thresholds as you learn from real data.
• Introduce sarcasm or indirect sentiment examples for finer calibration.

Final Prompt to User (Role: User):
"Analyze the following timestamped transcript. Identify all key topics, track their sentiment changes over time (adding a new history entry each time a sentiment shift occurs), and return only the JSON result with a topics array as specified. No commentary, just JSON.

[TRANSCRIPT_TEXT][TRANSCRIPT_TEXT]"

(Replace [TRANSCRIPT_TEXT] with the actual transcript.)