Building a High-Accuracy Classification System with Claude: From 70% to 95%+ Accuracy

Classification is a cornerstone of many business workflows, from routing customer support tickets to categorizing documents. Traditional machine learning approaches often struggle with complex business rules, limited training data, and the need for explainable results. Large Language Models (LLMs) like Claude offer a powerful alternative.

In this guide, you'll learn how to build a production-ready classification system using Claude that achieves 95%+ accuracy on a complex insurance support ticket classification task. You'll progress through three key techniques: prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ with basic familiarity
• Anthropic API key (get one here)
• VoyageAI API key (optional — embeddings are pre-computed)
• Understanding of classification problems

Setup

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Then, load your API keys and set up the client:

import os
from anthropic import Anthropic
Load API keys from environment
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
Set your model
MODEL_NAME = "claude-3-opus-20240229"

Problem Definition: Insurance Support Ticket Classifier

We'll build a classifier that categorizes insurance support tickets into 10 categories. The dataset and labels are synthetically generated by Claude 3 Opus, but they reflect real-world complexity.

Category Definitions

• Billing Inquiries — Questions about invoices, charges, fees, premiums, payment methods, and due dates.
• Policy Administration — Requests for policy changes, updates, cancellations, renewals, or adding/removing coverage.
• Claims Assistance — Questions about the claims process, filing procedures, documentation, claim status, and payout timelines.
• Coverage Explanations — Questions about what is covered, coverage limits, exclusions, deductibles, and out-of-pocket expenses.
• Account Management — Login issues, password resets, account updates, and profile management.
• Product Information — Questions about insurance products, plan options, and policy features.
• Complaints — Dissatisfaction with service, complaints about agents, or negative feedback.
• Fraud Reporting — Reporting suspected fraud, identity theft, or suspicious claims.
• General Inquiry — Miscellaneous questions not fitting other categories.
• Cancellation Requests — Requests to cancel policies or terminate coverage.

Step 1: Baseline Classification with Prompt Engineering

Let's start with a simple zero-shot classification prompt. This will give us a baseline to improve upon.

def classify_ticket_zero_shot(ticket_text, categories):
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
Categories:
{chr(10).join([f'{i+1}. {cat}' for i, cat in enumerate(categories)])}
Ticket: {ticket_text}
Respond with only the category name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Expected accuracy: ~70% — This approach works for simple cases but fails on ambiguous tickets or those requiring nuanced understanding of business rules.

Step 2: Improving with Few-Shot Examples and RAG

To boost accuracy, we'll implement Retrieval-Augmented Generation (RAG). The idea is simple: for each ticket, retrieve the most similar examples from a labeled dataset and include them in the prompt as few-shot examples.

Building the Vector Database

First, we'll create embeddings for our labeled training data using VoyageAI:

import voyageai
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Create embeddings for training data
train_texts = [example["text"] for example in training_data]
train_embeddings = vo.embed(train_texts, model="voyage-2").embeddings

Retrieving Relevant Examples

Now, when a new ticket comes in, we find the most similar examples:

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
def retrieve_similar_examples(query, train_embeddings, training_data, k=5):
    query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
    similarities = cosine_similarity([query_embedding], train_embeddings)[0]
    top_indices = np.argsort(similarities)[-k:][::-1]
    return [training_data[i] for i in top_indices]

Augmented Prompt with Examples

Finally, we build a prompt that includes the retrieved examples:

def classify_with_rag(ticket_text, categories, examples):
    example_str = ""
    for ex in examples:
        example_str += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Use the following examples as reference:
{example_str}
Now classify this ticket:
Ticket: {ticket_text}
Categories:
{chr(10).join([f'- {cat}' for cat in categories])}
Respond with only the category name."""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Expected accuracy: ~85-90% — RAG significantly improves performance by providing relevant context.

Step 3: Chain-of-Thought Reasoning for 95%+ Accuracy

To push accuracy even higher, we'll add chain-of-thought (CoT) reasoning. Instead of asking Claude to output just the category, we ask it to reason step-by-step before giving the final answer.

def classify_with_cot(ticket_text, categories, examples):
    example_str = ""
    for ex in examples:
        example_str += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Use the following examples as reference:
{example_str}
Now classify this ticket step by step:
Ticket: {ticket_text}
Categories:
{chr(10).join([f'- {cat}' for cat in categories])}
First, think through the reasoning:
What is the main topic of this ticket?
Which category best matches this topic?
Are there any edge cases or overlaps with other categories?

Then, provide your final answer in the format:
Category: [category name]"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Expected accuracy: 95%+ — Chain-of-thought reasoning helps Claude handle ambiguous cases, edge cases, and tickets that span multiple categories.

Testing and Evaluation

To evaluate your classifier, run it against a held-out test set:

def evaluate_classifier(classifier_fn, test_data, categories):
    correct = 0
    total = len(test_data)
    
    for item in test_data:
        predicted = classifier_fn(item["text"], categories, retrieve_similar_examples(item["text"], train_embeddings, training_data))
        if predicted == item["category"]:
            correct += 1
    
    accuracy = correct / total * 100
    print(f"Accuracy: {accuracy:.2f}%")
    return accuracy

Key Takeaways

• Start simple, then iterate: Begin with zero-shot prompting, then add few-shot examples via RAG, and finally incorporate chain-of-thought reasoning for maximum accuracy.
• RAG bridges the gap: Retrieving relevant examples from a vector database dramatically improves classification accuracy without requiring fine-tuning.
• Chain-of-thought reasoning unlocks 95%+ accuracy: By asking Claude to reason step-by-step, you handle edge cases and ambiguous tickets that stump simpler approaches.
• Explainability is built-in: Unlike traditional ML classifiers, Claude provides natural language explanations for its decisions, making it easier to audit and debug.
• Works with limited data: This approach excels when you have only hundreds (not thousands) of labeled examples, making it ideal for real-world business scenarios.

Next Steps

• Experiment with different embedding models (e.g., text-embedding-3-small, voyage-2)
• Add a confidence threshold to flag uncertain classifications for human review
• Implement a feedback loop where corrections improve future classifications
• Explore multi-label classification for tickets that span multiple categories

By combining prompt engineering, RAG, and chain-of-thought reasoning, you can build classification systems that rival or exceed traditional machine learning approaches — with less data, more flexibility, and full explainability.