Building a High-Accuracy Classification System with Claude: From 70% to 95%+ Accuracy

Classification is one of the most common and impactful use cases for large language models (LLMs). Whether you're routing customer support tickets, moderating content, or categorizing documents, getting classification right can dramatically improve operational efficiency. But achieving production-grade accuracy—consistently above 95%—requires more than just a simple prompt.

In this guide, you'll learn how to build a robust classification system using Claude that progressively improves from ~70% to 95%+ accuracy. We'll use a real-world example: categorizing insurance support tickets into 10 distinct categories. You'll master three key techniques: prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.

Prerequisites

Before diving in, make sure you have:

• Python 3.11+ installed
• An Anthropic API key
• Basic familiarity with Python and API calls
• Understanding of classification concepts

Setup and Installation

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, set up your API keys and initialize the Claude client:

import os
from anthropic import Anthropic
Load API keys from environment variables
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
Initialize the Claude client
client = Anthropic(api_key=anthropic_api_key)
MODEL_NAME = "claude-3-opus-20240229"

The Problem: Insurance Support Ticket Classification

Insurance companies receive thousands of support tickets daily. Manually categorizing these tickets is slow, error-prone, and expensive. Our goal is to build an automated system that classifies tickets into categories like:

• Billing Inquiries – Questions about invoices, charges, premiums
• Policy Administration – Policy changes, cancellations, renewals
• Claims Assistance – Claims process, documentation, status
• Coverage Explanations – What's covered, limits, exclusions
• (and 6 more categories)

Step 1: Baseline Classification with a Simple Prompt

Let's start with the simplest approach: a direct prompt asking Claude to classify each ticket.

def classify_ticket(ticket_text, categories):
    prompt = f"""Classify the following insurance support ticket into one of these categories:
{categories}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~70% accuracy. Not bad for a baseline, but far from production-ready. The model struggles with ambiguous tickets and edge cases.

Step 2: Improving with Structured Prompts and Few-Shot Examples

The first improvement is to provide clear category definitions and a few examples. This gives Claude a better understanding of each category's boundaries.

def classify_with_examples(ticket_text, categories_with_definitions, examples):
    prompt = f"""You are an expert insurance ticket classifier. Classify the following ticket into exactly one category.
Category Definitions:
{categories_with_definitions}
Examples:
{examples}
Ticket to classify: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~82% accuracy. Adding definitions and examples helps, but we're still missing context for edge cases.

Step 3: Retrieval-Augmented Generation (RAG) for Dynamic Examples

Instead of hardcoding examples, we can use a vector database to retrieve the most relevant examples for each ticket. This is the core of RAG: dynamically augmenting the prompt with similar cases.

Building the Vector Database

import voyageai
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Generate embeddings for all training examples
def get_embeddings(texts):
    result = vo.embed(texts, model="voyage-2")
    return result.embeddings
Store embeddings with their categories
training_embeddings = get_embeddings(training_tickets)

Retrieving Similar Examples

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
def get_similar_examples(query, k=3):
    query_embedding = get_embeddings([query])[0]
    similarities = cosine_similarity([query_embedding], training_embeddings)[0]
    top_k_indices = np.argsort(similarities)[-k:][::-1]
    
    examples = []
    for idx in top_k_indices:
        examples.append({
            "ticket": training_tickets[idx],
            "category": training_categories[idx],
            "similarity": similarities[idx]
        })
    return examples

Classifying with RAG

def classify_with_rag(ticket_text, categories_with_definitions):
    # Retrieve similar examples
    similar_examples = get_similar_examples(ticket_text, k=3)
    
    # Format examples for the prompt
    examples_text = "\n\n".join([
        f"Example {i+1}:\nTicket: {ex['ticket']}\nCategory: {ex['category']}"
        for i, ex in enumerate(similar_examples)
    ])
    
    prompt = f"""You are an expert insurance ticket classifier. Classify the following ticket into exactly one category.
Category Definitions:
{categories_with_definitions}
Here are similar tickets and their categories for reference:
{examples_text}
Ticket to classify: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

Result: ~90% accuracy. The dynamic examples dramatically improve performance, especially for edge cases.

Step 4: Adding Chain-of-Thought Reasoning

The final improvement is to ask Claude to reason step-by-step before giving the final classification. This reduces errors from jumping to conclusions.

def classify_with_cot(ticket_text, categories_with_definitions):
    similar_examples = get_similar_examples(ticket_text, k=3)
    
    examples_text = "\n\n".join([
        f"Example {i+1}:\nTicket: {ex['ticket']}\nCategory: {ex['category']}"
        for i, ex in enumerate(similar_examples)
    ])
    
    prompt = f"""You are an expert insurance ticket classifier. Classify the following ticket into exactly one category.
Category Definitions:
{categories_with_definitions}
Here are similar tickets and their categories for reference:
{examples_text}
Ticket to classify: {ticket_text}
First, think step-by-step about which category fits best. Consider:
What is the main topic of the ticket?
Which category definition matches best?
How does this compare to the similar examples?

Then, provide your final answer in this format:
Reasoning: [your step-by-step reasoning]
Category: [exact category name]"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    
    # Parse the response to extract the category
    full_response = response.content[0].text.strip()
    category_line = [line for line in full_response.split('\n') if line.startswith('Category:')]
    return category_line[0].replace('Category:', '').strip() if category_line else full_response

Result: 95%+ accuracy. Chain-of-thought reasoning catches subtle distinctions and reduces misclassifications.

Testing and Evaluation

To properly evaluate your system, split your data into training and test sets:

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
Split data
X_train, X_test, y_train, y_test = train_test_split(
    tickets, categories, test_size=0.2, random_state=42
)
Build vector database from training data
training_embeddings = get_embeddings(X_train)
Test the classifier
predictions = []
for ticket in X_test:
    pred = classify_with_cot(ticket, category_definitions)
    predictions.append(pred)
Calculate accuracy
accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {accuracy:.2%}")
print(classification_report(y_test, predictions))

Best Practices for Production

• Monitor confidence scores: Track how often Claude is uncertain or asks for clarification
• Handle edge cases: Create a catch-all category for truly ambiguous tickets
• Iterate on examples: As you encounter misclassifications, add them as new examples to your vector database
• Use temperature 0: For classification tasks, always use temperature=0 for deterministic outputs
• Validate output format: Always parse and validate the returned category against your allowed list

Key Takeaways

• Start simple, then layer complexity: Begin with a basic prompt, then add few-shot examples, RAG, and chain-of-thought progressively
• RAG dramatically improves accuracy: Retrieving similar examples dynamically gives Claude the context it needs for edge cases
• Chain-of-thought reasoning catches subtle distinctions: Asking Claude to think step-by-step before classifying reduces errors by 5-10%
• You can achieve 95%+ accuracy without fine-tuning: With the right prompt engineering and RAG, Claude can match or exceed fine-tuned models
• Always validate and monitor in production: Classification systems need ongoing evaluation to maintain accuracy as new patterns emerge