Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most powerful and practical applications of large language models (LLMs). Whether you're routing customer support tickets, moderating content, or categorizing documents, getting classification right can dramatically improve operational efficiency.

In this guide, you'll learn how to build a production-ready classification system using Claude that achieves 95%+ accuracy on a complex, multi-class insurance support ticket classification task. We'll start with a simple prompt and progressively layer in advanced techniques: prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.

By the end, you'll have a reusable framework for building high-accuracy classifiers that handle complex business rules, work with limited training data, and provide explainable results.

Why Use Claude for Classification?

Traditional machine learning classifiers require large labeled datasets, extensive feature engineering, and struggle with nuanced or evolving business rules. Claude excels here because:

• Handles complex business logic without explicit programming
• Works with limited training data by leveraging pre-trained knowledge
• Provides natural language explanations for every classification decision
• Easily adapts to new categories or rule changes

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ installed
• An Anthropic API key
• Basic familiarity with Python and classification concepts
• (Optional) A VoyageAI API key for embeddings (pre-computed embeddings are available)

Step 1: Setup and Data Preparation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Now, let's set up our environment and load the API keys:

import os
import anthropic
Load API keys from environment
ANTHROPIC_API_KEY = os.environ.get("ANTHROPIC_API_KEY")
VOYAGE_API_KEY = os.environ.get("VOYAGE_API_KEY")
Initialize Claude client
client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)
Set model name
MODEL_NAME = "claude-3-opus-20240229"

Understanding the Problem

We're building a classifier for an insurance company's support ticket system. The tickets need to be categorized into 10 distinct categories, including:

• Billing Inquiries – Questions about invoices, charges, premiums
• Policy Administration – Policy changes, cancellations, renewals
• Claims Assistance – Claims process, documentation, status
• Coverage Explanations – What's covered, limits, exclusions
• (And 6 more categories covering the full insurance domain)

Each category has specific definitions and examples. The challenge? Many tickets span multiple categories or use industry jargon.

Step 2: Baseline Classification with Prompt Engineering

Let's start with a simple prompt and see where we land:

def classify_ticket_baseline(ticket_text: str) -> str:
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
Billing Inquiries
Policy Administration
Claims Assistance
Coverage Explanations

Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~70% accuracy. Not bad for a baseline, but we can do much better.

Step 3: Improving Accuracy with Structured Prompts

The key to better classification is providing Claude with clear category definitions and few-shot examples. Here's an improved approach:

def classify_ticket_structured(ticket_text: str, examples: list) -> str:
    # Build few-shot examples
    example_text = ""
    for ex in examples:
        example_text += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an expert insurance ticket classifier.
CATEGORY DEFINITIONS:
Billing Inquiries: Questions about invoices, charges, fees, premiums, payment methods, due dates.
Policy Administration: Requests for policy changes, cancellations, renewals, adding/removing coverage.
Claims Assistance: Questions about claims process, filing procedures, claim status, payout timelines.
Coverage Explanations: Questions about what's covered, limits, exclusions, deductibles.

EXAMPLES:
{example_text}

CLASSIFY THE FOLLOWING TICKET:
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~82% accuracy. Better, but we're still missing context.

Step 4: Retrieval-Augmented Generation (RAG) for Dynamic Examples

Static examples in prompts are limited. What if we could dynamically retrieve the most relevant examples for each ticket? That's where RAG comes in.

Building the Vector Database

import voyageai
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
Initialize VoyageAI client
vo = voyageai.Client(api_key=VOYAGE_API_KEY)
Generate embeddings for training data
def get_embeddings(texts: list) -> np.ndarray:
    result = vo.embed(texts, model="voyage-2")
    return np.array(result.embeddings)
Store embeddings in a simple vector database
train_embeddings = get_embeddings(train_texts)

Retrieving Relevant Examples

def retrieve_similar_examples(query: str, k: int = 3) -> list:
    # Get query embedding
    query_embedding = get_embeddings([query])[0]
    
    # Compute similarities
    similarities = cosine_similarity([query_embedding], train_embeddings)[0]
    
    # Get top-k indices
    top_indices = np.argsort(similarities)[-k:][::-1]
    
    return [train_data[i] for i in top_indices]

RAG-Enhanced Classification

def classify_ticket_rag(ticket_text: str) -> str:
    # Retrieve relevant examples
    similar_examples = retrieve_similar_examples(ticket_text, k=3)
    
    # Build prompt with retrieved examples
    prompt = f"""You are an expert insurance ticket classifier.
CATEGORY DEFINITIONS:
[Same definitions as above]
RELEVANT EXAMPLES:
"""
    for ex in similar_examples:
        prompt += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt += f"CLASSIFY THE FOLLOWING TICKET:\nTicket: {ticket_text}\n\nCategory:"
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~90% accuracy. The dynamic examples make a significant difference.

Step 5: Chain-of-Thought Reasoning for 95%+ Accuracy

The final piece of the puzzle is chain-of-thought (CoT) reasoning. Instead of asking Claude to jump straight to a category, we ask it to explain its reasoning first.

def classify_ticket_cot(ticket_text: str) -> dict:
    # Retrieve relevant examples
    similar_examples = retrieve_similar_examples(ticket_text, k=3)
    
    prompt = f"""You are an expert insurance ticket classifier.
CATEGORY DEFINITIONS:
[Same definitions as above]
RELEVANT EXAMPLES:
"""
    for ex in similar_examples:
        prompt += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt += f"""CLASSIFY THE FOLLOWING TICKET:
Ticket: {ticket_text}
First, think step-by-step:
What is the main topic of this ticket?
Which category definition best matches?
Are there any edge cases or ambiguities?

Reasoning:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text.strip()
    
    # Extract the final category (usually the last line)
    lines = full_response.split('\n')
    category = lines[-1].strip() if lines else "Unknown"
    
    return {
        "category": category,
        "reasoning": full_response
    }

Result: 95%+ accuracy. The reasoning step forces Claude to carefully consider the evidence before deciding.

Testing and Evaluation

To properly evaluate your classifier, use a held-out test set:

def evaluate_classifier(test_data: list, classifier_fn) -> dict:
    correct = 0
    total = len(test_data)
    
    for item in test_data:
        predicted = classifier_fn(item['text'])
        if isinstance(predicted, dict):
            predicted = predicted['category']
        if predicted.strip().lower() == item['category'].strip().lower():
            correct += 1
    
    accuracy = correct / total
    return {
        "accuracy": accuracy,
        "correct": correct,
        "total": total
    }
Run evaluation
results = evaluate_classifier(test_data, classify_ticket_cot)
print(f"Accuracy: {results['accuracy']:.2%}")

Key Takeaways

• Start simple, then iterate: Begin with a basic prompt, measure accuracy, and progressively add complexity (structured prompts → few-shot → RAG → chain-of-thought).
• RAG dramatically improves accuracy: Dynamically retrieving the most relevant examples for each query is far more effective than static few-shot examples.
• Chain-of-thought reasoning is a game-changer: Asking Claude to explain its reasoning before outputting a category consistently boosts accuracy by 5-10%.
• Explainability is built-in: Unlike traditional ML classifiers, Claude provides natural language justifications for every decision, making it easier to audit and debug.
• This framework is reusable: The same techniques apply to any classification problem – content moderation, document routing, intent detection, and more.

Next Steps

Ready to build your own classifier? Start by:

• Defining your categories with clear, unambiguous definitions
• Collecting 50-100 labeled examples per category
• Implementing the RAG + chain-of-thought pipeline shown above
• Iterating based on error analysis

For more advanced use cases, consider fine-tuning Claude on your specific domain or adding a confidence threshold for uncertain classifications.