Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most common and impactful tasks in business automation. Whether you're routing customer support tickets, moderating content, or categorizing documents, getting classification right directly affects operational efficiency and user experience.

Traditional machine learning approaches to classification often struggle with complex business rules, limited training data, and the need for explainable results. Large Language Models (LLMs) like Claude offer a powerful alternative—they can handle nuanced decision-making, work with few examples, and provide natural language justifications for their classifications.

In this guide, you'll build a production-ready classification system that categorizes insurance support tickets into 10 distinct categories. You'll learn how to progressively improve accuracy from a baseline of ~70% to over 95% by combining three key techniques:

• Prompt engineering to define clear classification rules
• Retrieval-Augmented Generation (RAG) to provide relevant examples
• Chain-of-thought reasoning to improve decision quality

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ installed with basic familiarity
• An Anthropic API key
• A VoyageAI API key (optional—embeddings can be pre-computed)
• Basic understanding of classification problems

Setup and Installation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, set up your environment and API clients:

import os
import anthropic
from getpass import getpass
Set your API keys
os.environ["ANTHROPIC_API_KEY"] = getpass("Enter your Anthropic API key: ")
os.environ["VOYAGE_API_KEY"] = getpass("Enter your VoyageAI API key (optional): ")
Initialize Claude client
client = anthropic.Anthropic()
Set model name
MODEL_NAME = "claude-3-opus-20240229"

Problem Definition: Insurance Support Ticket Classifier

Insurance companies receive thousands of support tickets daily, covering topics like billing, policy administration, claims, and coverage questions. Manually categorizing these tickets is slow and error-prone.

We'll classify tickets into 10 categories. Here are a few examples:

Step 1: Data Preparation

We'll split our data into training and test sets. The training data will be used to build the classification model, while the test data evaluates performance.

import pandas as pd
from sklearn.model_selection import train_test_split
Load your dataset (example structure)
df = pd.read_csv("insurance_tickets.csv")
Split into train and test
X_train, X_test, y_train, y_test = train_test_split(
    df["ticket_text"], 
    df["category"], 
    test_size=0.2, 
    random_state=42
)
print(f"Training samples: {len(X_train)}")
print(f"Test samples: {len(X_test)}")

Step 2: Baseline Classification with Prompt Engineering

Let's start with a simple prompt that defines the classification task. This is our baseline.

def classify_ticket_baseline(ticket_text, categories):
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{categories}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Expected accuracy: ~70% — This baseline works but misses many edge cases.

Step 3: Improving Accuracy with RAG (Retrieval-Augmented Generation)

To boost accuracy, we'll retrieve the most similar examples from our training data and include them in the prompt. This gives Claude concrete reference points.

First, create embeddings for your training data:

import voyageai
vo = voyageai.Client()
Generate embeddings for training data
train_embeddings = vo.embed(
    X_train.tolist(), 
    model="voyage-2"
).embeddings

Now, build a retrieval function and an enhanced classifier:

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
def retrieve_examples(query, k=3):
    """Retrieve k most similar training examples."""
    query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
    similarities = cosine_similarity([query_embedding], train_embeddings)[0]
    top_indices = np.argsort(similarities)[-k:][::-1]
    
    examples = []
    for idx in top_indices:
        examples.append({
            "ticket": X_train.iloc[idx],
            "category": y_train.iloc[idx]
        })
    return examples
def classify_ticket_with_rag(ticket_text, categories):
    # Retrieve similar examples
    examples = retrieve_examples(ticket_text, k=3)
    
    # Format examples for the prompt
    examples_text = "\n\nHere are similar examples from our database:\n"
    for ex in examples:
        examples_text += f"- Ticket: {ex['ticket']}\n  Category: {ex['category']}\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{categories}
{examples_text}
Ticket to classify: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Expected accuracy: ~85-90% — RAG provides concrete context that helps Claude make better decisions.

Step 4: Chain-of-Thought Reasoning for 95%+ Accuracy

For the final improvement, we'll add chain-of-thought reasoning. Instead of asking Claude to output just the category, we ask it to reason step-by-step before arriving at a conclusion.

def classify_ticket_cot(ticket_text, categories):
    # Retrieve similar examples
    examples = retrieve_examples(ticket_text, k=3)
    
    examples_text = "\n\nHere are similar examples from our database:\n"
    for ex in examples:
        examples_text += f"- Ticket: {ex['ticket']}\n  Category: {ex['category']}\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{categories}
{examples_text}
Ticket to classify: {ticket_text}
First, think step-by-step about which category fits best. Consider:
What is the main topic of the ticket?
What specific action or information is being requested?
Which category definition matches most closely?

Then, output your final answer as:
Category: [exact category name]

Reasoning:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}]
    )
    
    # Parse the response to extract the category
    full_response = response.content[0].text.strip()
    # Extract category line (assumes format "Category: ...")
    for line in full_response.split("\n"):
        if line.startswith("Category:"):
            return line.replace("Category:", "").strip()
    return full_response

Expected accuracy: 95%+ — Chain-of-thought reasoning forces Claude to analyze the ticket systematically before classifying.

Step 5: Evaluation

Now let's evaluate our system on the test set:

from sklearn.metrics import accuracy_score, classification_report
predictions = []
for ticket in X_test:
    pred = classify_ticket_cot(ticket, category_definitions)
    predictions.append(pred)
accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {accuracy:.2%}")
print("\nClassification Report:")
print(classification_report(y_test, predictions))

Performance Comparison

Key Takeaways

• Start simple, then layer complexity: Begin with a baseline prompt, then add RAG, then chain-of-thought. Each layer adds measurable accuracy gains.
• RAG dramatically improves classification with limited data: By retrieving similar examples, you effectively give Claude a "memory" of past correct classifications without fine-tuning.
• Chain-of-thought reasoning is your secret weapon: For complex classification tasks, asking Claude to reason step-by-step before outputting a category can boost accuracy by 10% or more.
• Explainability is built-in: Unlike traditional ML classifiers, Claude can provide natural language explanations for its decisions, making it easier to audit and debug.
• This pattern generalizes: The techniques in this guide—prompt engineering, RAG, and chain-of-thought—apply to any classification problem, not just insurance tickets.