Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most common and impactful tasks in business automation. Whether you're routing support tickets, categorizing customer feedback, or flagging compliance issues, getting classification right directly affects operational efficiency and customer satisfaction.

Traditional machine learning approaches often struggle with complex business rules, limited training data, or the need for explainable results. Large Language Models (LLMs) like Claude offer a powerful alternative—they can handle nuanced categories, provide natural language justifications, and adapt quickly to new requirements.

In this guide, you'll build a production-ready insurance support ticket classifier that categorizes tickets into 10 distinct categories. You'll learn how to progressively improve accuracy from a baseline of ~70% to over 95% by combining three key techniques:

• Prompt engineering to define clear classification rules
• Retrieval-Augmented Generation (RAG) to provide relevant examples
• Chain-of-thought reasoning to improve complex decision-making

By the end, you'll have a reusable framework for building high-accuracy classification systems with Claude.

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ installed
• An Anthropic API key
• Basic familiarity with Python and API calls
• (Optional) A VoyageAI API key for generating embeddings

Step 1: Setup and Data Preparation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Now, set up your environment and load your API keys:

import os
from anthropic import Anthropic
Load API keys from environment variables
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
Set your model
MODEL_NAME = "claude-3-opus-20240229"

Understanding the Data

For this guide, we'll use a synthetically generated dataset of insurance support tickets. The data covers 10 categories, including:

• Billing Inquiries – Questions about invoices, charges, fees, and premiums
• Policy Administration – Requests for policy changes, updates, or cancellations
• Claims Assistance – Questions about the claims process and filing procedures
• Coverage Explanations – Clarification on what is covered under specific policies
• And 6 more categories covering the full spectrum of insurance support

Each ticket is a short text description of a customer issue, paired with its correct category label.

Step 2: Baseline Classification with Prompt Engineering

Let's start with a simple approach: asking Claude to classify a ticket using only a prompt with category definitions.

def classify_ticket_baseline(ticket_text, categories):
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{categories}
Ticket: {ticket_text}
Respond with only the category name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

This baseline approach typically achieves around 70-75% accuracy. The main limitations are:

• No examples to guide Claude's understanding
• Ambiguous tickets can be misclassified
• No mechanism to handle edge cases

Step 3: Improving Accuracy with Retrieval-Augmented Generation (RAG)

To boost accuracy, we'll implement RAG. The idea is simple: for each new ticket, retrieve the most similar examples from our training data and include them in the prompt. This gives Claude concrete reference points.

Building the Vector Database

First, we'll create embeddings for our training data and store them in a vector database:

import voyageai
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
Initialize VoyageAI client
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Generate embeddings for training data
def get_embeddings(texts):
    result = vo.embed(texts, model="voyage-2")
    return result.embeddings
Store embeddings in a simple dictionary
train_embeddings = {}
for idx, row in train_data.iterrows():
    train_embeddings[idx] = {
        "text": row["ticket_text"],
        "label": row["category"],
        "embedding": get_embeddings([row["ticket_text"]])[0]
    }

Retrieving Similar Examples

Now, when a new ticket comes in, we find the most similar examples:

def find_similar_examples(query, train_embeddings, k=3):
    query_embedding = get_embeddings([query])[0]
    
    similarities = []
    for idx, data in train_embeddings.items():
        sim = cosine_similarity([query_embedding], [data["embedding"]])[0][0]
        similarities.append((sim, data))
    
    # Return top-k most similar examples
    similarities.sort(reverse=True, key=lambda x: x[0])
    return [s[1] for s in similarities[:k]]

Enhanced Classification with RAG

Finally, we include the retrieved examples in our prompt:

def classify_ticket_with_rag(ticket_text, categories, train_embeddings):
    # Retrieve similar examples
    examples = find_similar_examples(ticket_text, train_embeddings, k=3)
    
    # Format examples for the prompt
    examples_text = ""
    for i, ex in enumerate(examples, 1):
        examples_text += f"Example {i}:\nTicket: {ex['text']}\nCategory: {ex['label']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{categories}
Here are some examples of correctly classified tickets:
{examples_text}
Ticket: {ticket_text}
Respond with only the category name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

With RAG, accuracy typically jumps to 85-90%. The examples help Claude understand the nuances of each category.

Step 4: Achieving 95%+ with Chain-of-Thought Reasoning

To push accuracy even higher, we'll add chain-of-thought (CoT) reasoning. Instead of asking Claude to output just the category, we ask it to think step-by-step before making a decision.

def classify_ticket_cot(ticket_text, categories, train_embeddings):
    # Retrieve similar examples
    examples = find_similar_examples(ticket_text, train_embeddings, k=3)
    
    examples_text = ""
    for i, ex in enumerate(examples, 1):
        examples_text += f"Example {i}:\nTicket: {ex['text']}\nCategory: {ex['label']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{categories}
Here are some examples of correctly classified tickets:
{examples_text}
Ticket: {ticket_text}
First, think step-by-step about which category fits best. Consider:
What is the main topic of the ticket?
Which category definition matches most closely?
Are there any edge cases or ambiguities?

Then, provide your final answer in this format:
Reasoning: [your step-by-step reasoning]
Category: [category name]"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    
    # Parse the response to extract the category
    full_response = response.content[0].text.strip()
    category_line = [line for line in full_response.split('\n') if line.startswith('Category:')]
    return category_line[0].replace('Category:', '').strip() if category_line else full_response

This approach consistently achieves 95%+ accuracy. The chain-of-thought reasoning helps Claude handle ambiguous cases and provides transparency into its decision-making process.

Step 5: Testing and Evaluation

Let's evaluate our system on a test dataset:

def evaluate_classifier(test_data, classifier_fn, **kwargs):
    correct = 0
    total = len(test_data)
    
    for idx, row in test_data.iterrows():
        predicted = classifier_fn(row["ticket_text"], **kwargs)
        if predicted == row["category"]:
            correct += 1
    
    accuracy = correct / total * 100
    return accuracy
Test the CoT classifier
accuracy = evaluate_classifier(
    test_data, 
    classify_ticket_cot,
    categories=category_definitions,
    train_embeddings=train_embeddings
)
print(f"Accuracy: {accuracy:.2f}%")

Best Practices for Production

When deploying your classifier, keep these tips in mind:

• Monitor accuracy over time – As new tickets come in, periodically re-evaluate your model's performance.
• Update your vector database – Add correctly classified tickets to your training data to improve retrieval quality.
• Handle edge cases – Create a catch-all category for tickets that don't fit existing categories.
• Log reasoning – Store the chain-of-thought output for auditability and debugging.
• Set confidence thresholds – If Claude's reasoning shows low confidence, flag the ticket for human review.

Key Takeaways

• Start simple, then iterate – Begin with basic prompt engineering, then layer in RAG and chain-of-thought reasoning to progressively improve accuracy.
• RAG dramatically improves performance – Providing relevant examples from your training data helps Claude understand nuanced category boundaries, boosting accuracy by 15-20%.
• Chain-of-thought reasoning adds transparency – Asking Claude to explain its reasoning not only improves accuracy but also makes the system auditable and easier to debug.
• This approach works with limited data – Unlike traditional ML classifiers that require thousands of examples per category, Claude can achieve high accuracy with just dozens of well-chosen examples.
• The framework is reusable – You can adapt this pattern to any classification task, from customer support routing to content moderation to document categorization.