Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most practical applications of large language models (LLMs) in enterprise settings. Traditional machine learning classifiers struggle with complex business rules, limited training data, and the need for explainable results. Claude excels in all these areas.

In this guide, you'll build a production-ready classification system that categorizes insurance support tickets into 10 distinct categories. You'll learn how to progressively improve accuracy from roughly 70% to over 95% by combining three powerful techniques: prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ with basic familiarity
• Anthropic API key (get one here)
• VoyageAI API key (optional — embeddings are pre-computed in the cookbook)
• Basic understanding of classification problems

Setup and Installation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, load your API keys and set up the Claude client:

import os
from anthropic import Anthropic
Load API keys from environment
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
Initialize Claude client
client = Anthropic(api_key=anthropic_api_key)
MODEL_NAME = "claude-3-opus-20240229"

Why Use Claude for Classification?

Traditional machine learning classifiers require large amounts of labeled data and struggle with nuanced business rules. LLMs like Claude offer several advantages:

• Handle complex business rules that are difficult to encode in traditional ML
• Work with limited training data — sometimes just a few examples per class
• Provide natural language explanations for every classification decision
• Adapt quickly to new categories without retraining

Problem Definition: Insurance Support Ticket Classifier

We'll build a system that categorizes insurance support tickets into 10 categories. Here are the category definitions (synthetically generated by Claude 3 Opus for this example):

• Billing Inquiries — Questions about invoices, charges, fees, and premiums
• Policy Administration — Requests for policy changes, updates, or cancellations
• Claims Assistance — Questions about the claims process and filing procedures
• Coverage Explanations — Questions about what is covered under specific policy types
• Account Management — Requests for account updates, password resets, or login issues
• Fraud and Security — Reports of suspicious activity or identity theft concerns
• Agent and Broker Support — Questions about agent assignments or broker communications
• Complaints and Escalations — Formal complaints or requests for supervisor intervention
• General Inquiries — Miscellaneous questions not fitting other categories
• Policy Documentation — Requests for policy documents, certificates, or ID cards

Step 1: Data Preparation

We'll split our data into training and test sets. The training data will be used to build the classification model, while the test data will evaluate its performance.

import pandas as pd
from sklearn.model_selection import train_test_split
Load your dataset (replace with your actual data)
df = pd.read_csv('insurance_tickets.csv')
Split into train and test sets
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
print(f"Training samples: {len(train_df)}")
print(f"Test samples: {len(test_df)}")

Step 2: Basic Prompt Engineering (70% Accuracy)

Let's start with a simple prompt that defines the task and categories:

def classify_ticket_basic(ticket_text: str) -> str:
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
Billing Inquiries
Policy Administration
Claims Assistance
Coverage Explanations
Account Management
Fraud and Security
Agent and Broker Support
Complaints and Escalations
General Inquiries
Policy Documentation

Respond with ONLY the category name.
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

This basic approach typically achieves around 70% accuracy. The main issues are:

• Ambiguous tickets that could fit multiple categories
• Lack of examples to guide the model
• No reasoning step to work through complex cases

Step 3: Adding Few-Shot Examples (80% Accuracy)

We can improve accuracy by including examples in the prompt. This is called few-shot prompting:

def classify_ticket_few_shot(ticket_text: str) -> str:
    examples = """
Example 1:
Ticket: "Why was I charged $150 for a late fee on my auto policy?"
Category: Billing Inquiries
Example 2:
Ticket: "I need to add my new car to my existing policy."
Category: Policy Administration
Example 3:
Ticket: "Someone filed a claim using my policy number without my permission."
Category: Fraud and Security
"""
    
    prompt = f"""You are an insurance support ticket classifier.
Classify the following ticket into exactly one of these categories:
[Category definitions here]
Here are some examples:
{examples}
Ticket: {ticket_text}
Category:"""
    
    # ... API call same as before

Adding 3-5 well-chosen examples typically boosts accuracy to around 80%. The key is selecting examples that cover edge cases and ambiguous scenarios.

Step 4: Implementing RAG for Dynamic Examples (90% Accuracy)

Static examples in the prompt are limited. For better results, we'll implement Retrieval-Augmented Generation (RAG) to dynamically fetch the most relevant examples for each ticket.

import voyageai
from sklearn.metrics.pairwise import cosine_similarity
Initialize embedding model
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Create embeddings for training data
def get_embeddings(texts):
    result = vo.embed(texts, model="voyage-2")
    return result.embeddings
Pre-compute embeddings for training data
train_embeddings = get_embeddings(train_df['ticket_text'].tolist())
def find_similar_examples(query: str, k: int = 3):
    query_embedding = get_embeddings([query])[0]
    similarities = cosine_similarity([query_embedding], train_embeddings)[0]
    top_indices = similarities.argsort()[-k:][::-1]
    return train_df.iloc[top_indices]
def classify_ticket_rag(ticket_text: str) -> str:
    # Find similar examples
    similar = find_similar_examples(ticket_text)
    
    # Build dynamic examples string
    examples = ""
    for _, row in similar.iterrows():
        examples += f"Ticket: {row['ticket_text']}\nCategory: {row['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier.
Classify the following ticket into exactly one of these categories:
[Category definitions here]
Here are the most relevant examples:
{examples}
Ticket: {ticket_text}
Category:"""
    
    # ... API call

RAG brings accuracy to approximately 90%. The model now has contextually relevant examples for every query.

Step 5: Chain-of-Thought Reasoning (95%+ Accuracy)

The final improvement comes from adding chain-of-thought (CoT) reasoning. Instead of jumping straight to a category, Claude first explains its reasoning:

def classify_ticket_cot(ticket_text: str) -> dict:
    # Find similar examples (same as before)
    similar = find_similar_examples(ticket_text)
    
    examples = ""
    for _, row in similar.iterrows():
        examples += f"Ticket: {row['ticket_text']}\nCategory: {row['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier.
Classify the following ticket into exactly one of these categories:
[Category definitions here]
Here are the most relevant examples:
{examples}
First, think through your reasoning step by step. Then provide your final answer.
Ticket: {ticket_text}
Reasoning:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text.strip()
    
    # Parse reasoning and final category
    # (In practice, you'd use structured output or parsing)
    return {
        "reasoning": full_response,
        "category": extract_category(full_response)
    }

Chain-of-thought reasoning pushes accuracy above 95% because:

• Claude works through ambiguous cases systematically
• The reasoning step catches edge cases
• You get explainable results for compliance and auditing

Testing and Evaluation

Let's evaluate our final system:

def evaluate_classifier(classifier_func, test_df):
    correct = 0
    total = len(test_df)
    
    for _, row in test_df.iterrows():
        predicted = classifier_func(row['ticket_text'])
        if predicted == row['category']:
            correct += 1
    
    accuracy = correct / total * 100
    print(f"Accuracy: {accuracy:.2f}%")
    return accuracy
Test the final classifier
final_accuracy = evaluate_classifier(classify_ticket_cot, test_df)

Putting It All Together

Here's the complete pipeline:

• Data Preparation — Split data into train/test sets
• Embedding Generation — Create vector embeddings for training data
• Similarity Search — Find relevant examples for each query
• Prompt Construction — Build prompt with category definitions + dynamic examples
• Chain-of-Thought Classification — Claude reasons through the classification
• Evaluation — Measure accuracy and iterate

Key Takeaways

• Start simple, then layer complexity: Begin with basic prompt engineering (70% accuracy), add few-shot examples (80%), implement RAG (90%), and finish with chain-of-thought reasoning (95%+).
• RAG dramatically improves accuracy: Dynamically fetching relevant examples for each query is far more effective than static examples in the prompt.
• Chain-of-thought reasoning is essential for complex cases: Having Claude explain its reasoning catches edge cases and provides audit trails.
• Claude excels where traditional ML struggles: Complex business rules, limited training data, and the need for explainable results are all areas where Claude outperforms traditional classifiers.
• Always evaluate systematically: Use a held-out test set and measure accuracy to validate improvements before deploying to production.