Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most practical applications of large language models (LLMs) in business today. While traditional machine learning approaches struggle with complex business rules, limited training data, and the need for explainable results, Claude excels in all these areas.

In this guide, you'll build a production-ready insurance support ticket classifier that categorizes customer inquiries into 10 distinct categories. We'll start with a simple prompt-based approach (achieving ~70% accuracy) and progressively refine it using retrieval-augmented generation (RAG) and chain-of-thought reasoning to reach 95%+ accuracy.

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ with basic familiarity
• Anthropic API key – get one here
• VoyageAI API key (optional – embeddings are pre-computed in the cookbook)
• Basic understanding of classification problems

Setup and Installation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, load your API keys and set up the Claude client:

import os
from anthropic import Anthropic
Load API keys from environment
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
Set model name
MODEL_NAME = "claude-3-opus-20240229"

Problem Definition: Insurance Support Ticket Classifier

Insurance companies receive thousands of support tickets daily. Manually categorizing them is slow, error-prone, and expensive. Our goal is to automate this process with high accuracy.

We'll classify tickets into 10 categories:

• Billing Inquiries – Questions about invoices, charges, fees, premiums
• Policy Administration – Policy changes, cancellations, renewals
• Claims Assistance – Claims process, documentation, status
• Coverage Explanations – What's covered, limits, exclusions
• Account Management – Login issues, profile updates
• Underwriting – Risk assessment, policy issuance
• Fraud Reporting – Suspicious activity, identity theft
• Compliance – Regulatory questions, legal requirements
• Agent Support – Agent tools, commission questions
• General Inquiry – Anything not fitting above

Step 1: Basic Prompt Engineering (70% Accuracy)

Let's start with a straightforward prompt that defines the task and categories:

def classify_ticket_basic(ticket_text: str) -> str:
    prompt = f"""You are an insurance support ticket classifier. Categorize the following ticket into one of these categories:
Billing Inquiries
Policy Administration
Claims Assistance
Coverage Explanations
Account Management
Underwriting
Fraud Reporting
Compliance
Agent Support
General Inquiry

Respond with ONLY the category number and name.
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: This approach typically achieves around 70% accuracy. The main issues are ambiguity in edge cases and inconsistent handling of tickets that span multiple categories.

Step 2: Adding Chain-of-Thought Reasoning (85% Accuracy)

By asking Claude to reason step-by-step before outputting a classification, we dramatically improve accuracy:

def classify_ticket_cot(ticket_text: str) -> str:
    prompt = f"""You are an insurance support ticket classifier. For the given ticket, follow these steps:
Identify the main topic and key entities mentioned
Determine which category best matches the primary intent
If multiple categories apply, choose the most specific one
Output only the category number and name

Categories:
Billing Inquiries
Policy Administration
Claims Assistance
Coverage Explanations
Account Management
Underwriting
Fraud Reporting
Compliance
Agent Support
General Inquiry

Ticket: {ticket_text}
Let's think step by step:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: Chain-of-thought reasoning pushes accuracy to ~85%. Claude can now disambiguate between similar categories by reasoning about intent.

Step 3: Implementing Retrieval-Augmented Generation (RAG) (95%+ Accuracy)

To reach production-level accuracy, we need to provide Claude with relevant examples from our training data. This is where RAG comes in.

Create a Vector Database

First, generate embeddings for your training data:

import voyageai
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Generate embeddings for training examples
train_texts = [example["ticket"] for example in training_data]
train_embeddings = vo.embed(train_texts, model="voyage-2").embeddings

Build a Retrieval Function

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
def retrieve_similar_examples(query: str, k: int = 3):
    # Embed the query
    query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
    
    # Compute similarities
    similarities = cosine_similarity([query_embedding], train_embeddings)[0]
    
    # Get top-k indices
    top_indices = np.argsort(similarities)[-k:][::-1]
    
    # Return the most similar examples
    return [training_data[i] for i in top_indices]

Augment the Prompt with Retrieved Examples

def classify_ticket_rag(ticket_text: str) -> str:
    # Retrieve similar examples
    similar_examples = retrieve_similar_examples(ticket_text, k=3)
    
    # Format examples for the prompt
    examples_text = ""
    for i, ex in enumerate(similar_examples, 1):
        examples_text += f"Example {i}:\nTicket: {ex['ticket']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Use the following examples as reference for how to classify tickets.
Reference Examples:
{examples_text}
Now classify this ticket:
Ticket: {ticket_text}
Follow these steps:
Compare this ticket to the reference examples
Identify the primary intent
Output only the category number and name

Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: With RAG, accuracy jumps to 95%+. The retrieved examples act as a dynamic few-shot learning mechanism, adapting to each query's specific context.

Testing and Evaluation

To properly evaluate your classifier, split your data into training and test sets:

from sklearn.model_selection import train_test_split
Assuming you have a list of tickets with their true categories
tickets = [item["ticket"] for item in all_data]
categories = [item["category"] for item in all_data]
X_train, X_test, y_train, y_test = train_test_split(
    tickets, categories, test_size=0.2, random_state=42
)
Evaluate the RAG classifier
correct = 0
total = len(X_test)
for ticket, true_category in zip(X_test, y_test):
    predicted = classify_ticket_rag(ticket)
    if predicted == true_category:
        correct += 1
accuracy = correct / total
print(f"Accuracy: {accuracy:.2%}")

Best Practices for Production

• Monitor accuracy drift – Re-evaluate your classifier periodically as new ticket types emerge
• Log misclassifications – Use them to improve your retrieval database
• Set confidence thresholds – Flag low-confidence classifications for human review
• Cache embeddings – Avoid recomputing embeddings for the same queries
• Use async API calls – For high-throughput systems, batch your classification requests

Key Takeaways

• Start simple, then iterate – Basic prompt engineering gets you to ~70% accuracy; chain-of-thought adds another 15%; RAG pushes you past 95%
• RAG is your secret weapon – By retrieving relevant examples dynamically, you overcome the limitations of static few-shot prompts and handle edge cases gracefully
• Explainability matters – Claude's natural language reasoning makes classifications auditable and trustworthy, which is critical in regulated industries like insurance
• Limited data is not a blocker – Unlike traditional ML, LLMs can achieve high accuracy with as few as 50-100 labeled examples when combined with RAG
• Production readiness – The techniques in this guide are immediately applicable to real-world systems, not just academic exercises