Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most practical and high-impact applications of large language models (LLMs) in business. Whether you're routing support tickets, moderating content, or categorizing customer feedback, getting classification right can save hours of manual work and improve response times dramatically.

In this guide, you'll build a production-ready classification system using Claude that categorizes insurance support tickets into 10 distinct categories. We'll start with a simple prompt-based approach (hitting around 70% accuracy) and progressively layer in advanced techniques—including retrieval-augmented generation (RAG) and chain-of-thought reasoning—to push accuracy above 95%.

By the end, you'll have a reusable framework for building high-accuracy classifiers that handle complex business rules, work with limited training data, and provide explainable results.

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ installed with basic familiarity
• Anthropic API key (get one here)
• VoyageAI API key (optional—embeddings can be pre-computed)
• Basic understanding of classification problems

Setup: Installing Dependencies

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Then, load your API keys and set your model name:

import os
from anthropic import Anthropic
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
MODEL_NAME = "claude-3-opus-20240229"  # or claude-3-sonnet for faster results

The Problem: Insurance Support Ticket Classification

Insurance companies receive thousands of support tickets daily—billing questions, claims assistance, policy changes, and more. Manually categorizing these is slow, error-prone, and expensive.

We'll classify tickets into 10 categories:

• Billing Inquiries – Invoices, charges, fees, premiums
• Policy Administration – Changes, updates, cancellations, renewals
• Claims Assistance – Filing procedures, documentation, status
• Coverage Explanations – What's covered, limits, exclusions
• Account Management – Login issues, profile updates, password resets
• Underwriting – Risk assessment, policy issuance, documentation
• Fraud & Compliance – Suspicious activity, regulatory questions
• Agent Support – Commission questions, licensing, tools
• Product Information – Plan details, benefits, comparisons
• General Inquiry – Anything not fitting above

Step 1: Basic Prompt-Based Classification (70% Accuracy)

Let's start simple. We'll ask Claude to classify a ticket using only the category definitions in the prompt.

def classify_ticket_basic(ticket_text: str) -> str:
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
Billing Inquiries
Policy Administration
Claims Assistance
Coverage Explanations
Account Management
Underwriting
Fraud & Compliance
Agent Support
Product Information
General Inquiry

Respond with ONLY the category number and name.
Ticket: {ticket_text}"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: This approach typically achieves ~70% accuracy. Why? Because category definitions alone don't capture edge cases, ambiguous phrasing, or domain-specific nuances. For example, "I need to update my payment method" could be Billing or Account Management depending on context.

Step 2: Adding Few-Shot Examples (80% Accuracy)

To improve, we can provide a few labeled examples in the prompt. This gives Claude reference points for ambiguous cases.

def classify_ticket_few_shot(ticket_text: str) -> str:
    examples = """
Example 1: "Why was I charged $50 extra this month?" -> 1. Billing Inquiries
Example 2: "I need to cancel my auto policy effective next week" -> 2. Policy Administration
Example 3: "How do I file a claim for my damaged roof?" -> 3. Claims Assistance
Example 4: "Does my plan cover annual checkups?" -> 4. Coverage Explanations
"""
    
    prompt = f"""You are an insurance support ticket classifier. Use these examples as reference:
{examples}
Now classify this ticket:
{ticket_text}
Respond with ONLY the category number and name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: Accuracy jumps to ~80%. But we're limited by the prompt's context window—we can only include a handful of examples. For a 10-class problem, we need more.

Step 3: Retrieval-Augmented Generation (RAG) for Dynamic Examples (90% Accuracy)

Instead of hardcoding examples, we'll store all our training data in a vector database and retrieve the most relevant examples for each new ticket. This is the key to scaling.

Build the Vector Database

import voyageai
import numpy as np
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Sample training data (ticket_text, category)
training_data = [
    ("Why was my premium increased?", "Billing Inquiries"),
    ("I want to add roadside assistance to my policy", "Policy Administration"),
    # ... 100+ more examples
]
Generate embeddings
texts = [item[0] for item in training_data]
embeddings = vo.embed(texts, model="voyage-2").embeddings
Store in a simple numpy array for demo (use Pinecone/Weaviate in production)
embedding_matrix = np.array(embeddings)

Retrieve and Classify

from sklearn.metrics.pairwise import cosine_similarity
def retrieve_examples(query: str, k: int = 5):
    query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
    similarities = cosine_similarity([query_embedding], embedding_matrix)[0]
    top_indices = np.argsort(similarities)[-k:][::-1]
    return [training_data[i] for i in top_indices]
def classify_ticket_rag(ticket_text: str) -> str:
    # Retrieve most similar examples
    similar_examples = retrieve_examples(ticket_text, k=5)
    examples_str = "\n".join([f"{text} -> {cat}" for text, cat in similar_examples])
    
    prompt = f"""You are an insurance support ticket classifier. Here are the most relevant examples:
{examples_str}
Now classify this ticket:
{ticket_text}
Respond with ONLY the category number and name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: Accuracy reaches ~90%. By dynamically retrieving the most relevant examples, Claude gets better context for each classification.

Step 4: Chain-of-Thought Reasoning (95%+ Accuracy)

For the final push, we'll add chain-of-thought (CoT) reasoning. Instead of jumping straight to a category, Claude first explains its reasoning step-by-step. This reduces errors from jumping to conclusions.

def classify_ticket_cot(ticket_text: str) -> dict:
    similar_examples = retrieve_examples(ticket_text, k=5)
    examples_str = "\n".join([f"{text} -> {cat}" for text, cat in similar_examples])
    
    prompt = f"""You are an insurance support ticket classifier. Follow these steps:
Read the ticket carefully
Identify key phrases and keywords
Compare with the relevant examples below
Explain your reasoning step by step
Output the final category

Relevant examples:
{examples_str}
Ticket: {ticket_text}
First, provide your reasoning in <reasoning> tags. Then, output the category in <category> tags."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text.strip()
    # Parse reasoning and category from XML tags
    reasoning = full_response.split("<reasoning>")[1].split("</reasoning>")[0].strip()
    category = full_response.split("<category>")[1].split("</category>")[0].strip()
    
    return {"category": category, "reasoning": reasoning}

Result: 95%+ accuracy. The chain-of-thought step forces Claude to articulate its logic, catching mistakes like confusing "payment method update" (Account Management) with "billing dispute" (Billing Inquiries).

Testing and Evaluation

To properly evaluate your classifier, split your data into training and test sets:

from sklearn.model_selection import train_test_split
Assuming you have X (ticket texts) and y (labels)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
Build vector DB from X_train only
Then evaluate on X_test
def evaluate_classifier(classify_fn, X_test, y_test):
    correct = 0
    for ticket, true_label in zip(X_test, y_test):
        predicted = classify_fn(ticket)
        if predicted == true_label:
            correct += 1
    accuracy = correct / len(X_test)
    print(f"Accuracy: {accuracy:.2%}")
    return accuracy

Best Practices for Production

• Use a dedicated vector database – For production, use Pinecone, Weaviate, or Chroma instead of in-memory numpy arrays.
• Cache embeddings – Pre-compute and store embeddings to avoid re-querying the embedding API.
• Monitor confidence – Track cases where Claude is uncertain (e.g., when it asks for clarification) and route them to human reviewers.
• Iterate on edge cases – Continuously add misclassified examples to your training data.
• Use structured output – With Claude's tool use feature, you can enforce JSON output for easier parsing.

Key Takeaways

• Start simple, then layer complexity – A basic prompt gets ~70% accuracy. Add few-shot examples for ~80%, RAG for ~90%, and chain-of-thought for 95%+.
• RAG scales your training data – By retrieving relevant examples dynamically, you can leverage hundreds of labeled examples without blowing up your prompt.
• Chain-of-thought reduces errors – Forcing Claude to explain its reasoning catches subtle misclassifications and improves accuracy by 5-10%.
• This framework is reusable – The same pattern (basic prompt → few-shot → RAG → CoT) works for any classification problem, from content moderation to medical coding.
• Explainability is built-in – With chain-of-thought, every classification comes with a human-readable explanation, which is critical for regulated industries like insurance.

Now you have a production-ready classification system that can handle complex business rules, work with limited training data, and provide explainable results. Go ahead and adapt this framework to your own use cases!