Building a High-Accuracy Classification System with Claude: From 70% to 95%+ Accuracy

Classification is one of the most common and impactful use cases for Large Language Models (LLMs). Whether you're routing customer support tickets, moderating content, or categorizing documents, getting classification right can dramatically improve operational efficiency. However, achieving high accuracy—especially with complex business rules and limited training data—requires more than just a simple prompt.

In this guide, you'll learn how to build a production-ready classification system using Claude AI that progressively improves accuracy from a baseline of ~70% to over 95%. We'll use a real-world example: categorizing insurance support tickets into 10 distinct categories.

Prerequisites

• Python 3.11+ with basic familiarity
• An Anthropic API key
• A VoyageAI API key (optional—embeddings can be pre-computed)
• Basic understanding of classification problems

Setup and Installation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, set up your environment variables and initialize the Claude client:

import os
from anthropic import Anthropic
Load API keys from environment
ANTHROPIC_API_KEY = os.environ.get("ANTHROPIC_API_KEY")
VOYAGE_API_KEY = os.environ.get("VOYAGE_API_KEY")
Initialize Claude client
client = Anthropic(api_key=ANTHROPIC_API_KEY)
MODEL_NAME = "claude-3-opus-20240229"

The Challenge: Insurance Support Ticket Classification

Insurance companies receive thousands of support tickets daily. Manually categorizing these tickets is slow, expensive, and error-prone. The categories include:

• Billing Inquiries – Questions about invoices, charges, fees, and premiums
• Policy Administration – Requests for policy changes, updates, or cancellations
• Claims Assistance – Questions about the claims process and filing procedures
• Coverage Explanations – Questions about what is covered under specific policy types
• And 6 more categories (total of 10)

The challenge? Many tickets span multiple categories, contain ambiguous language, or reference complex business rules that traditional ML models struggle to handle.

Step 1: Baseline Classification with Prompt Engineering

Let's start with a simple approach: asking Claude to classify tickets using a well-structured prompt.

def classify_ticket(ticket_text, categories):
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{categories}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~70% accuracy. Not bad for a baseline, but far from production-ready. The main issues are ambiguity in edge cases and inconsistent handling of multi-topic tickets.

Step 2: Improving with Few-Shot Examples

Adding examples to your prompt (few-shot learning) can significantly boost accuracy. The key is selecting the right examples for each query.

def classify_with_examples(ticket_text, categories, examples):
    example_text = ""
    for ex in examples:
        example_text += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{categories}
Here are some examples:
{example_text}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~80% accuracy. Better, but we're still missing context for edge cases.

Step 3: Retrieval-Augmented Generation (RAG) for Dynamic Examples

Instead of hardcoding examples, use a vector database to retrieve the most semantically similar examples for each query. This is where RAG shines.

import voyageai
import numpy as np
Initialize VoyageAI for embeddings
vo = voyageai.Client(api_key=VOYAGE_API_KEY)
Create embeddings for your training data
def create_embeddings(texts):
    result = vo.embed(texts, model="voyage-2")
    return result.embeddings
Find similar examples for a given query
def find_similar_examples(query, training_data, k=3):
    query_embedding = create_embeddings([query])[0]
    
    # Calculate cosine similarity
    similarities = []
    for item in training_data:
        item_embedding = item['embedding']
        similarity = np.dot(query_embedding, item_embedding)
        similarities.append(similarity)
    
    # Get top-k indices
    top_indices = np.argsort(similarities)[-k:][::-1]
    return [training_data[i] for i in top_indices]

Now integrate this into your classification function:

def classify_with_rag(ticket_text, categories, training_data):
    # Retrieve relevant examples
    similar_examples = find_similar_examples(ticket_text, training_data, k=5)
    
    # Build prompt with retrieved examples
    example_text = ""
    for ex in similar_examples:
        example_text += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{categories}
Relevant examples:
{example_text}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~90% accuracy. The dynamic retrieval of relevant examples makes a significant difference.

Step 4: Chain-of-Thought Reasoning for Explainable Results

To push accuracy above 95%, add chain-of-thought (CoT) reasoning. This forces Claude to "think through" the classification step by step, reducing errors and providing explainable results.

def classify_with_cot(ticket_text, categories, training_data):
    similar_examples = find_similar_examples(ticket_text, training_data, k=5)
    
    example_text = ""
    for ex in similar_examples:
        example_text += f"Ticket: {ex['text']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. 
Classify the following ticket into exactly one of these categories:
{categories}
Relevant examples:
{example_text}
Ticket: {ticket_text}
First, think through the classification step by step:
What is the main topic of this ticket?
Which category best matches this topic?
Are there any edge cases or ambiguities?

Then, provide your final answer in this format:
Category: [category name]
Reasoning: [brief explanation]"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: 95%+ accuracy. The combination of RAG and chain-of-thought reasoning creates a robust, explainable classification system.

Testing and Evaluation

To properly evaluate your system, split your data into training and test sets:

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
Split data
X_train, X_test, y_train, y_test = train_test_split(
    tickets, labels, test_size=0.2, random_state=42
)
Evaluate
predictions = []
for ticket in X_test:
    result = classify_with_cot(ticket, categories, training_data)
    predicted_category = extract_category(result)
    predictions.append(predicted_category)
accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {accuracy:.2%}")
print(classification_report(y_test, predictions))

Key Takeaways

• Start simple, then iterate: Begin with basic prompt engineering (70% accuracy), then layer in few-shot examples (80%), RAG (90%), and chain-of-thought reasoning (95%+) for progressive improvement.
• RAG is a game-changer for classification: Dynamically retrieving similar examples from your training data provides context that static prompts cannot match, especially for edge cases.
• Chain-of-thought reasoning adds explainability: By forcing Claude to "think aloud," you not only improve accuracy but also gain insight into why a classification was made—critical for auditing and debugging.
• LLMs excel where traditional ML struggles: Complex business rules, ambiguous language, and limited training data are exactly the scenarios where LLM-based classification outperforms traditional approaches.
• Always test rigorously: Use proper train/test splits and evaluation metrics to measure real-world performance before deploying to production.