Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Large Language Models (LLMs) have transformed the classification landscape, particularly for problems involving complex business rules, limited training data, or the need for explainable results. In this guide, you'll build a production-ready classification system that categorizes insurance support tickets into 10 distinct categories with 95%+ accuracy.

By combining prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning, you'll learn how to progressively improve your classifier's performance while maintaining interpretability—a critical requirement in regulated industries like insurance.

Prerequisites

Before diving in, ensure you have:

• Python 3.11+ with basic familiarity
• Anthropic API key (get one here)
• VoyageAI API key (optional—embeddings can be pre-computed)
• Basic understanding of classification problems

Setup and Installation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, set up your environment variables and initialize the Claude client:

import os
from anthropic import Anthropic
Load API keys from environment
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
Set your model
MODEL_NAME = "claude-3-opus-20240229"

Understanding the Problem: Insurance Support Ticket Classification

Insurance companies receive thousands of support tickets daily, covering everything from billing inquiries to claims assistance. Manual categorization is slow, error-prone, and expensive. An automated classifier must handle:

• Complex business rules (e.g., "Is a premium adjustment a billing issue or policy administration?")
• Ambiguous language (e.g., "My payment didn't go through" could be billing or technical)
• Explainable decisions (regulatory requirements demand transparency)

The 10 Ticket Categories

• Billing Inquiries – Invoices, charges, fees, premiums
• Policy Administration – Changes, updates, cancellations, renewals
• Claims Assistance – Filing procedures, documentation, status
• Coverage Explanations – What's covered, limits, exclusions
• Account Management – Login issues, profile updates, contact changes
• Underwriting – Risk assessment, policy issuance, eligibility
• Fraud & Compliance – Suspicious activity, regulatory questions
• Agent Support – Commission questions, agent portal issues
• Product Information – Policy types, riders, benefits
• General Inquiry – Anything not fitting above categories

Step 1: Data Preparation

Proper data preparation is crucial. You'll need:

• Training data: Labeled examples for few-shot learning
• Test data: Unseen examples for evaluation

import pandas as pd
from sklearn.model_selection import train_test_split
Load your dataset
Assuming a CSV with columns: 'ticket_text' and 'category'
df = pd.read_csv("insurance_tickets.csv")
Split into training and test sets
train_df, test_df = train_test_split(
    df, test_size=0.2, random_state=42, stratify=df['category']
)
print(f"Training samples: {len(train_df)}")
print(f"Test samples: {len(test_df)}")

Step 2: Prompt Engineering for Baseline Classification

Start with a well-structured prompt that defines categories clearly. This is your baseline—expect around 70% accuracy.

def classify_ticket_baseline(ticket_text, categories):
    """Basic classification without examples."""
    prompt = f"""You are an insurance support ticket classifier. 
Categorize the following ticket into exactly one of these categories:
{categories}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Why this works: Clear category definitions reduce ambiguity. However, without examples, Claude may struggle with edge cases.

Step 3: Adding Few-Shot Examples

Improve accuracy by including 3-5 representative examples per category:

def classify_ticket_fewshot(ticket_text, categories, examples):
    """Classification with few-shot examples."""
    example_text = "\n\nHere are examples of correctly classified tickets:\n"
    for ex in examples:
        example_text += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Categorize the following ticket into exactly one of these categories:
{categories}
{example_text}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

This typically boosts accuracy to 80-85%.

Step 4: Implementing Retrieval-Augmented Generation (RAG)

For maximum accuracy (95%+), dynamically retrieve the most relevant examples for each query using vector embeddings.

Create Embeddings for Your Training Data

import voyageai
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Generate embeddings for all training examples
train_texts = train_df['ticket_text'].tolist()
train_embeddings = vo.embed(
    train_texts, 
    model="voyage-2", 
    input_type="document"
).embeddings

Build a Simple Vector Store

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
class VectorStore:
    def __init__(self, texts, embeddings, labels):
        self.texts = texts
        self.embeddings = np.array(embeddings)
        self.labels = labels
    
    def search(self, query_embedding, k=5):
        similarities = cosine_similarity(
            [query_embedding], self.embeddings
        )[0]
        top_indices = np.argsort(similarities)[-k:][::-1]
        return [
            {
                'text': self.texts[i],
                'label': self.labels[i],
                'score': similarities[i]
            }
            for i in top_indices
        ]
vector_store = VectorStore(train_texts, train_embeddings, train_df['category'].tolist())

Classify with RAG

def classify_ticket_rag(ticket_text, categories, vector_store, k=3):
    """Classification with RAG-based example retrieval."""
    # Get query embedding
    query_embedding = vo.embed(
        [ticket_text], 
        model="voyage-2", 
        input_type="query"
    ).embeddings[0]
    
    # Retrieve most similar examples
    retrieved = vector_store.search(query_embedding, k=k)
    
    # Build prompt with retrieved examples
    example_text = "\n\nHere are the most relevant examples:\n"
    for ex in retrieved:
        example_text += f"Ticket: {ex['text']}\nCategory: {ex['label']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Categorize the following ticket into exactly one of these categories:
{categories}
{example_text}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Step 5: Adding Chain-of-Thought Reasoning

For explainable results, instruct Claude to reason step-by-step before outputting the category:

def classify_ticket_cot(ticket_text, categories, vector_store, k=3):
    """Classification with chain-of-thought reasoning."""
    query_embedding = vo.embed(
        [ticket_text], 
        model="voyage-2", 
        input_type="query"
    ).embeddings[0]
    
    retrieved = vector_store.search(query_embedding, k=k)
    
    example_text = "\n\nHere are the most relevant examples:\n"
    for ex in retrieved:
        example_text += f"Ticket: {ex['text']}\nCategory: {ex['label']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Categorize the following ticket into exactly one of these categories.
First, think step-by-step about which category fits best. 
Consider the key topics, keywords, and intent of the ticket.
Then, output your final answer as: "Category: [category_name]"
Categories:
{categories}
{example_text}
Ticket: {ticket_text}
Reasoning:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text.strip()
    # Extract category from the response
    if "Category:" in full_response:
        return full_response.split("Category:")[-1].strip()
    return full_response

Step 6: Testing and Evaluation

Run your classifier against the test set and measure accuracy:

def evaluate_classifier(classifier_fn, test_df, categories, vector_store):
    correct = 0
    total = len(test_df)
    
    for idx, row in test_df.iterrows():
        predicted = classifier_fn(
            row['ticket_text'], 
            categories, 
            vector_store
        )
        if predicted == row['category']:
            correct += 1
    
    accuracy = correct / total
    return accuracy
Evaluate each approach
baseline_acc = evaluate_classifier(classify_ticket_baseline, test_df, categories, None)
rag_acc = evaluate_classifier(classify_ticket_rag, test_df, categories, vector_store)
cot_acc = evaluate_classifier(classify_ticket_cot, test_df, categories, vector_store)
print(f"Baseline accuracy: {baseline_acc:.1%}")
print(f"RAG accuracy: {rag_acc:.1%}")
print(f"Chain-of-thought + RAG accuracy: {cot_acc:.1%}")

Expected Results

Best Practices for Production

• Monitor for drift: Regularly evaluate your classifier on new data to catch performance degradation
• Log reasoning: Store chain-of-thought outputs for audit trails and debugging
• Handle edge cases: Add a "Confidence" field to flag low-confidence classifications for human review
• Optimize retrieval: Experiment with k (number of retrieved examples) and embedding models
• Cache embeddings: Pre-compute and store embeddings to reduce API costs

Key Takeaways

• Start simple, iterate fast: Begin with a well-structured prompt, then add few-shot examples, RAG, and chain-of-thought progressively
• RAG dramatically improves accuracy: Dynamic example retrieval outperforms static few-shot learning by providing contextually relevant examples
• Explainability matters: Chain-of-thought reasoning not only improves accuracy but also provides audit trails—critical for regulated industries
• 95%+ accuracy is achievable: By combining prompt engineering, RAG, and structured reasoning, you can build production-grade classifiers with limited training data
• Cost vs. accuracy tradeoffs: RAG adds embedding costs but reduces the number of tokens needed per classification, often resulting in net savings for high-volume systems