Building a High-Accuracy Insurance Ticket Classifier with Claude: From 70% to 95%+ Accuracy

Classification is one of the most powerful and practical applications of large language models (LLMs) like Claude. Traditional machine learning classifiers often struggle with complex business rules, limited training data, and the need for explainable results. Claude excels in all these areas.

In this guide, you'll build a production-ready classification system that categorizes insurance support tickets into 10 distinct categories. You'll learn how to progressively improve accuracy from a baseline of ~70% to over 95% by combining three key techniques: prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.

Why Use Claude for Classification?

Before diving into the code, let's understand why Claude is an excellent choice for classification tasks:

• Handles complex business rules: Claude can understand nuanced category definitions and edge cases that are difficult to encode in traditional ML models.
• Works with limited data: Unlike traditional classifiers that require hundreds or thousands of labeled examples per category, Claude can perform well with just a handful of high-quality examples.
• Provides explanations: Claude can justify its classifications in natural language, making it easy to audit and debug.
• Adapts quickly: You can update category definitions or add new categories by simply modifying the prompt — no model retraining required.

Prerequisites

To follow along, you'll need:

• Python 3.11+ installed
• An Anthropic API key
• Basic familiarity with Python and classification concepts
• (Optional) A VoyageAI API key for generating embeddings

Step 1: Setup and Data Preparation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, set up your environment and load the API keys:

import os
from anthropic import Anthropic
Load API keys from environment variables
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
Set your model
MODEL_NAME = "claude-3-opus-20240229"

Understanding the Dataset

For this guide, we'll use a synthetically generated dataset of insurance support tickets. The data covers 10 categories commonly found in insurance customer service:

• Billing Inquiries — Questions about invoices, charges, fees, and premiums
• Policy Administration — Requests for policy changes, updates, or cancellations
• Claims Assistance — Questions about the claims process and filing procedures
• Coverage Explanations — Questions about what is covered under specific policy types
• Account Management — Login issues, profile updates, and account access
• Underwriting Questions — Risk assessment, policy issuance, and eligibility
• Fraud Reporting — Suspected fraudulent activity or identity theft
• Compliance and Regulatory — Questions about insurance regulations and legal requirements
• Agent Support — Requests from insurance agents regarding their clients
• General Inquiries — Miscellaneous questions not covered by other categories

Step 2: Baseline Classification with Prompt Engineering

Let's start with a simple approach: asking Claude to classify tickets using only a prompt with category definitions.

def classify_ticket_baseline(ticket_text, category_definitions):
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{category_definitions}
Ticket: {ticket_text}
Respond with only the category name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

This baseline approach typically achieves around 70-75% accuracy. The main issues are:

• Ambiguous tickets that could fit multiple categories
• Lack of examples to guide Claude's understanding
• No way to handle edge cases or subtle distinctions

Step 3: Improving Accuracy with Few-Shot Examples

Adding a few carefully selected examples to the prompt can significantly boost performance. This technique is called few-shot prompting.

def classify_ticket_fewshot(ticket_text, category_definitions, examples):
    examples_text = ""
    for example in examples:
        examples_text += f"Ticket: {example['text']}\nCategory: {example['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{category_definitions}
Here are some examples:
{examples_text}
Ticket: {ticket_text}
Respond with only the category name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

With 3-5 well-chosen examples per category, accuracy typically jumps to 80-85%.

Step 4: Retrieval-Augmented Generation (RAG) for Dynamic Examples

Instead of hardcoding examples, we can use a vector database to retrieve the most relevant examples for each ticket. This is where RAG shines.

Building the Vector Database

import voyageai
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
Initialize VoyageAI client
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Generate embeddings for your training data
def get_embeddings(texts):
    result = vo.embed(texts, model="voyage-2")
    return result.embeddings
Store embeddings in a simple list (use a proper vector DB in production)
training_embeddings = get_embeddings([ex["text"] for ex in training_data])

Retrieving Relevant Examples

def retrieve_similar_examples(query, k=3):
    query_embedding = get_embeddings([query])[0]
    
    # Calculate similarity scores
    similarities = cosine_similarity([query_embedding], training_embeddings)[0]
    
    # Get top-k indices
    top_indices = np.argsort(similarities)[-k:][::-1]
    
    return [training_data[i] for i in top_indices]

RAG-Enhanced Classification

def classify_ticket_rag(ticket_text, category_definitions):
    # Retrieve relevant examples
    similar_examples = retrieve_similar_examples(ticket_text, k=3)
    
    # Build prompt with retrieved examples
    examples_text = ""
    for ex in similar_examples:
        examples_text += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{category_definitions}
Here are similar tickets and their categories:
{examples_text}
Ticket: {ticket_text}
Respond with only the category name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

With RAG, accuracy typically reaches 88-92%.

Step 5: Chain-of-Thought Reasoning for Maximum Accuracy

To push accuracy beyond 95%, we add chain-of-thought (CoT) reasoning. Instead of asking Claude to output just the category, we ask it to reason step-by-step before giving the final answer.

def classify_ticket_cot(ticket_text, category_definitions):
    # Retrieve relevant examples
    similar_examples = retrieve_similar_examples(ticket_text, k=3)
    
    examples_text = ""
    for ex in similar_examples:
        examples_text += f"Ticket: {ex['text']}\nCategory: {ex['category']}\nReasoning: {ex['reasoning']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{category_definitions}
Here are similar tickets, their categories, and reasoning:
{examples_text}
Ticket: {ticket_text}
First, reason step-by-step about which category this ticket belongs to. Then, provide your final answer in the format:
Category: [category name]
Reasoning:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text.strip()
    # Parse the category from the response
    category = full_response.split("Category:")[-1].strip().split("\n")[0]
    return category

This approach consistently achieves 95%+ accuracy on the insurance ticket dataset.

Step 6: Testing and Evaluation

To properly evaluate your classifier, split your data into training and test sets:

from sklearn.model_selection import train_test_split
Split data
train_data, test_data = train_test_split(
    all_tickets, 
    test_size=0.2, 
    random_state=42,
    stratify=[t["category"] for t in all_tickets]
)
Evaluate on test set
correct = 0
total = len(test_data)
for ticket in test_data:
    predicted = classify_ticket_cot(ticket["text"], category_definitions)
    if predicted == ticket["category"]:
        correct += 1
accuracy = correct / total
print(f"Accuracy: {accuracy:.2%}")

Best Practices for Production

• Use a proper vector database like Pinecone, Weaviate, or Chroma for production-scale RAG.
• Cache embeddings to avoid regenerating them for every query.
• Monitor confidence scores — ask Claude to output a confidence level and flag low-confidence classifications for human review.
• Implement fallback logic — if Claude's confidence is below a threshold, route to a human agent.
• Regularly update your example database with new, high-quality examples to maintain accuracy.

Key Takeaways

• Start simple, then layer complexity: Begin with prompt engineering, add few-shot examples, then RAG, and finally chain-of-thought reasoning. Each layer adds meaningful accuracy improvements.
• RAG dramatically improves accuracy: By dynamically retrieving the most relevant examples for each query, you can boost accuracy by 15-20 percentage points over baseline prompting.
• Chain-of-thought reasoning pushes accuracy past 95%: Asking Claude to reason step-by-step before outputting a classification reduces errors on ambiguous cases.
• Claude excels with limited data: Unlike traditional ML classifiers, Claude can achieve high accuracy with just a handful of examples per category.
• Explainability is built-in: Claude can provide natural language justifications for its classifications, making it easy to audit and debug your system.