Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most practical applications of Large Language Models (LLMs) in enterprise settings. Whether you're routing customer support tickets, categorizing documents, or flagging compliance issues, getting classification right—and explainable—is critical.

In this guide, you'll learn how to build a production-ready classification system using Anthropic's Claude. We'll walk through a real-world example: an Insurance Support Ticket Classifier that categorizes customer inquiries into 10 distinct categories. You'll see how to progressively improve accuracy from a baseline of ~70% to over 95% by combining prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ with basic familiarity
• Anthropic API key (get one here)
• VoyageAI API key (optional—embeddings can be pre-computed)
• Basic understanding of classification problems

Why LLMs for Classification?

Traditional machine learning classifiers struggle with:

• Complex business rules that are hard to encode as features
• Limited or low-quality training data
• Explainability—black-box models don't tell you why a decision was made

LLMs like Claude solve these problems. They can understand nuanced business logic from natural language descriptions, work effectively with few-shot examples, and provide natural language justifications for every classification.

Step 1: Problem Definition and Data Preparation

Our example comes from the insurance industry. Customer support tickets cover topics like billing, policy administration, claims assistance, and coverage explanations. Manually categorizing these is slow and error-prone.

Category Definitions

Here are the 10 categories we'll use:

• Billing Inquiries – Questions about invoices, charges, fees, premiums
• Policy Administration – Policy changes, renewals, cancellations
• Claims Assistance – Claims process, documentation, status
• Coverage Explanations – What's covered, limits, exclusions
• Account Management – Login issues, profile updates, contact changes
• Underwriting Questions – Risk assessment, policy issuance
• Agent Support – Agent tools, commission inquiries
• Fraud Reporting – Suspicious activity, identity theft concerns
• Compliance & Regulatory – Legal requirements, regulatory filings
• General Inquiries – Miscellaneous questions

Setting Up Your Environment

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Then load your API keys and set up the client:

import os
from anthropic import Anthropic
Load API keys from environment
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
Set model name
MODEL_NAME = "claude-3-opus-20240229"

Step 2: Baseline Classification with Prompt Engineering

Let's start simple. We'll create a basic prompt that asks Claude to classify a ticket based on the category definitions.

Designing the Prompt Template

def create_classification_prompt(ticket_text: str, categories: list) -> str:
    category_descriptions = "\n".join(
        [f"{i+1}. {cat['name']}: {cat['description']}" 
         for i, cat in enumerate(categories)]
    )
    
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{category_descriptions}
Ticket: {ticket_text}
Respond with only the category number and name, e.g., "1. Billing Inquiries".
"""
    return prompt

Running the Baseline

def classify_ticket(ticket_text: str, categories: list) -> str:
    prompt = create_classification_prompt(ticket_text, categories)
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    
    return response.content[0].text
Test it
ticket = "I was charged twice for my premium this month. Can you refund the duplicate?"
result = classify_ticket(ticket, categories)
print(result)  # Should output: 1. Billing Inquiries

Baseline accuracy: ~70%. Not bad, but we can do much better.

Step 3: Improving Accuracy with Few-Shot Examples

The biggest leap in accuracy comes from providing relevant examples. Instead of just describing categories, we show Claude actual tickets and their correct classifications.

Building a Few-Shot Example Store

# Example tickets with correct classifications
examples = [
    {
        "ticket": "My premium went up $50 this month. Why?",
        "category": "1. Billing Inquiries"
    },
    {
        "ticket": "I need to add my spouse to my auto policy.",
        "category": "2. Policy Administration"
    },
    {
        "ticket": "How do I file a claim for hail damage?",
        "category": "3. Claims Assistance"
    },
    # Add 5-10 more diverse examples
]

Retrieving Relevant Examples with RAG

For maximum accuracy, we don't just use random examples—we retrieve the most similar ones using vector embeddings. This is Retrieval-Augmented Generation (RAG).

import voyageai
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Embed all example tickets
example_texts = [ex["ticket"] for ex in examples]
example_embeddings = vo.embed(example_texts, model="voyage-2").embeddings
def find_similar_examples(query: str, k: int = 3):
    query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
    
    # Compute cosine similarity
    similarities = [
        np.dot(query_embedding, emb) / 
        (np.linalg.norm(query_embedding) * np.linalg.norm(emb))
        for emb in example_embeddings
    ]
    
    # Get top-k indices
    top_indices = np.argsort(similarities)[-k:][::-1]
    return [examples[i] for i in top_indices]

Enhanced Prompt with RAG

def create_rag_prompt(ticket_text: str, categories: list, examples: list) -> str:
    example_block = "\n\n".join([
        f"Example {i+1}:\nTicket: {ex['ticket']}\nCategory: {ex['category']}"
        for i, ex in enumerate(examples)
    ])
    
    prompt = f"""You are an insurance support ticket classifier. 
Here are some examples of correctly classified tickets:
{example_block}
Now classify this new ticket. Use the examples above as guidance.
Categories:
{category_descriptions}
Ticket: {ticket_text}
First, think step by step about which category fits best. Then respond with only the category number and name.
"""
    return prompt

Accuracy after RAG: ~85-90%. Significant improvement.

Step 4: Chain-of-Thought Reasoning for 95%+ Accuracy

The final piece is chain-of-thought (CoT) reasoning. Instead of asking Claude to jump straight to an answer, we ask it to reason step by step.

def create_cot_prompt(ticket_text: str, categories: list, examples: list) -> str:
    prompt = f"""You are an insurance support ticket classifier.
Here are examples:
{example_block}
Categories:
{category_descriptions}
Ticket: {ticket_text}
Let's think through this step by step:
What is the customer's main issue or request?
Which category best matches this issue?
Why do other categories not fit?

After your reasoning, provide your final answer on a new line starting with "Category:".
"""
    return prompt

Full Classification Pipeline

def classify_with_cot(ticket_text: str) -> dict:
    # 1. Retrieve similar examples
    similar = find_similar_examples(ticket_text, k=3)
    
    # 2. Build prompt with CoT
    prompt = create_cot_prompt(ticket_text, categories, similar)
    
    # 3. Get response from Claude
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text
    
    # 4. Parse the category from the response
    lines = full_response.split("\n")
    category_line = [l for l in lines if l.startswith("Category:")][0]
    
    return {
        "category": category_line.replace("Category:", "").strip(),
        "reasoning": full_response
    }

Final accuracy: 95%+. And you get a full explanation for every classification.

Step 5: Evaluation and Iteration

To measure your system's performance:

from sklearn.metrics import accuracy_score, classification_report
def evaluate(test_tickets, test_labels):
    predictions = []
    for ticket in test_tickets:
        result = classify_with_cot(ticket)
        predictions.append(result["category"])
    
    accuracy = accuracy_score(test_labels, predictions)
    print(f"Accuracy: {accuracy:.2%}")
    print(classification_report(test_labels, predictions))
    
    return accuracy

Best Practices for Production

• Start simple: Begin with a basic prompt, then add examples, then add RAG, then add CoT.
• Diversify your examples: Include edge cases and ambiguous tickets.
• Monitor drift: Re-evaluate your system monthly as new ticket types emerge.
• Use structured output: Request JSON format for easier parsing in production.
• Cache embeddings: Pre-compute and store embeddings for your example database.

Key Takeaways

• LLMs excel at complex classification with nuanced business rules and limited training data, outperforming traditional ML approaches in these scenarios.
• RAG dramatically improves accuracy by providing relevant few-shot examples retrieved via vector similarity search.
• Chain-of-thought reasoning pushes accuracy above 95% while providing explainable results—critical for regulated industries like insurance.
• Start with a simple prompt and iterate: Each layer (basic prompt → few-shot → RAG → CoT) adds measurable improvement.
• Production systems need monitoring: Re-evaluate periodically and update your example database as new patterns emerge.