Building a High-Accuracy Classification System with Claude: From 70% to 95%+ Accuracy

Classification is a cornerstone of many business workflows—from routing customer support tickets to categorizing documents and moderating content. Traditional machine learning approaches often struggle with complex business rules, limited training data, and the need for explainable results. Enter Claude AI: a large language model that can handle these challenges with remarkable ease.

In this guide, you'll learn how to build a production-ready classification system using Claude, progressively improving accuracy from a baseline of 70% to over 95% by combining three powerful techniques: prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning. We'll use a real-world example: classifying insurance support tickets into 10 categories.

By the end, you'll have a reusable framework for building classification systems that are accurate, explainable, and adaptable to your own business needs.

Prerequisites

• Python 3.11+ with basic familiarity
• An Anthropic API key (get one here)
• A VoyageAI API key (optional—embeddings can be pre-computed)
• Basic understanding of classification problems

Setup: Installing Dependencies

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Next, load your API keys and set your model name:

import os
from anthropic import Anthropic
Load API keys from environment variables
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
voyage_api_key = os.environ.get("VOYAGE_API_KEY")
Initialize the Anthropic client
client = Anthropic(api_key=anthropic_api_key)
Set the model name
MODEL_NAME = "claude-3-opus-20240229"

Why Use LLMs for Classification?

Large language models like Claude have revolutionized classification by overcoming key limitations of traditional ML:

• Complex business rules: LLMs can understand nuanced, multi-condition logic that's hard to encode in feature engineering.
• Limited training data: Few-shot learning works well with just 10–50 examples per class.
• Explainability: Claude can provide natural language justifications for its decisions, building trust and enabling audit trails.

Step 1: Data Preparation

We'll start by preparing our training and test datasets. The training data is used to build the classification model (via few-shot examples), while the test data evaluates performance.

For this guide, we'll use synthetically generated insurance support tickets covering 10 categories:

• Billing Inquiries
• Policy Administration
• Claims Assistance
• Coverage Explanations
• Account Management
• Underwriting
• Fraud & Compliance
• Agent Support
• Product Information
• General Feedback

import pandas as pd
from sklearn.model_selection import train_test_split
Load your dataset (replace with your actual data)
df = pd.read_csv("insurance_tickets.csv")
Split into train and test sets
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
print(f"Training samples: {len(train_df)}")
print(f"Test samples: {len(test_df)}")

Step 2: Prompt Engineering for Baseline Classification

Prompt engineering is the art of crafting instructions that guide Claude to produce accurate, consistent outputs. For classification, your prompt should include:

• System instructions: Define the task and output format.
• Category definitions: Clear, detailed descriptions of each class.
• Few-shot examples: Representative samples for each category.
• The query: The input to classify.

Here's a baseline prompt template:

def build_classification_prompt(query, categories, examples):
    """Build a prompt for classifying a support ticket."""
    
    # Format category definitions
    category_descriptions = "\n".join([
        f"{i+1}. {cat['name']}: {cat['description']}"
        for i, cat in enumerate(categories)
    ])
    
    # Format few-shot examples
    example_text = ""
    for ex in examples:
        example_text += f"Ticket: {ex['ticket']}\nCategory: {ex['category']}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Your task is to categorize each ticket into one of the following categories:
{category_descriptions}
Here are some examples:
{example_text}
Now classify this ticket:
Ticket: {query}
Category:"""
    
    return prompt
Example usage
categories = [
    {"name": "Billing Inquiries", "description": "Questions about invoices, charges, fees, and premiums"},
    {"name": "Claims Assistance", "description": "Questions about the claims process and filing procedures"},
    # ... add all 10 categories
]
examples = [
    {"ticket": "Why was I charged $50 extra this month?", "category": "Billing Inquiries"},
    {"ticket": "How do I file a claim for water damage?", "category": "Claims Assistance"},
    # ... add more examples
]
query = "I need to update my policy after getting married"
prompt = build_classification_prompt(query, categories, examples)

Now let's classify using Claude:

def classify_ticket(query, categories, examples):
    """Classify a single ticket using Claude."""
    prompt = build_classification_prompt(query, categories, examples)
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    
    return response.content[0].text.strip()
Test it
result = classify_ticket(query, categories, examples)
print(f"Predicted category: {result}")

This baseline approach typically achieves around 70% accuracy—decent, but not production-ready.

Step 3: Implementing Retrieval-Augmented Generation (RAG)

To boost accuracy, we'll use RAG to dynamically retrieve the most relevant few-shot examples for each query. This ensures Claude sees examples that are semantically similar to the input, improving classification quality.

Create a Vector Database

We'll use VoyageAI to generate embeddings and store them in a simple vector index:

import voyageai
import numpy as np
vo = voyageai.Client(api_key=voyage_api_key)
Generate embeddings for training examples
def get_embeddings(texts):
    result = vo.embed(texts, model="voyage-2")
    return result.embeddings
Create a vector database (simple in-memory index)
class VectorDB:
    def __init__(self, texts, embeddings):
        self.texts = texts
        self.embeddings = np.array(embeddings)
    
    def search(self, query_embedding, k=5):
        # Cosine similarity search
        similarities = np.dot(self.embeddings, query_embedding) / (
            np.linalg.norm(self.embeddings, axis=1) * np.linalg.norm(query_embedding)
        )
        top_k_indices = np.argsort(similarities)[-k:][::-1]
        return [self.texts[i] for i in top_k_indices]
Build the database
train_texts = train_df["ticket"].tolist()
train_embeddings = get_embeddings(train_texts)
db = VectorDB(train_texts, train_embeddings)

Augment the Prompt with Retrieved Examples

Now modify your classification function to retrieve relevant examples dynamically:

def classify_with_rag(query, categories, db, k=5):
    """Classify a ticket using RAG to retrieve relevant examples."""
    # Get query embedding
    query_embedding = get_embeddings([query])[0]
    
    # Retrieve similar examples
    similar_examples = db.search(query_embedding, k=k)
    
    # Build prompt with retrieved examples
    example_text = ""
    for ex in similar_examples:
        # You'll need to map back to the original category
        category = train_df[train_df["ticket"] == ex]["category"].iloc[0]
        example_text += f"Ticket: {ex}\nCategory: {category}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Categories:
{format_categories(categories)}
Relevant examples:
{example_text}
Classify this ticket:
Ticket: {query}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=50,
        messages=[{"role": "user", "content": prompt}]
    )
    
    return response.content[0].text.strip()

With RAG, accuracy typically jumps to 85–90%.

Step 4: Adding Chain-of-Thought Reasoning

Chain-of-thought (CoT) reasoning asks Claude to explain its reasoning before outputting the final category. This reduces errors by forcing the model to think step-by-step.

def classify_with_cot(query, categories, db, k=5):
    """Classify using RAG + chain-of-thought reasoning."""
    query_embedding = get_embeddings([query])[0]
    similar_examples = db.search(query_embedding, k=k)
    
    example_text = ""
    for ex in similar_examples:
        category = train_df[train_df["ticket"] == ex]["category"].iloc[0]
        example_text += f"Ticket: {ex}\nCategory: {category}\n\n"
    
    prompt = f"""You are an insurance support ticket classifier. Categories:
{format_categories(categories)}
Relevant examples:
{example_text}
Classify this ticket by first reasoning step-by-step, then output the final category.
Ticket: {query}
Reasoning:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text.strip()
    
    # Parse the final category from the response
    # Assuming format: "Reasoning: ... \nCategory: X"
    lines = full_response.split("\n")
    for line in lines:
        if line.startswith("Category:"):
            return line.replace("Category:", "").strip()
    
    return full_response  # Fallback

With CoT reasoning added, accuracy can reach 95%+.

Step 5: Testing and Evaluation

Let's evaluate our system on the test set:

from sklearn.metrics import accuracy_score, classification_report
Classify all test tickets
predictions = []
for ticket in test_df["ticket"]:
    pred = classify_with_cot(ticket, categories, db)
    predictions.append(pred)
Calculate accuracy
accuracy = accuracy_score(test_df["category"], predictions)
print(f"Accuracy: {accuracy:.2%}")
Detailed report
print(classification_report(test_df["category"], predictions))

Putting It All Together: The Complete Pipeline

Here's the full classification pipeline:

def build_classification_pipeline(train_data, categories):
    """Build a complete classification pipeline."""
    # Step 1: Generate embeddings for training data
    train_texts = train_data["ticket"].tolist()
    train_embeddings = get_embeddings(train_texts)
    
    # Step 2: Create vector database
    db = VectorDB(train_texts, train_embeddings)
    
    # Step 3: Define classification function
    def classify(query):
        # Retrieve similar examples
        query_embedding = get_embeddings([query])[0]
        similar_examples = db.search(query_embedding, k=5)
        
        # Build prompt with CoT
        prompt = build_cot_prompt(query, categories, similar_examples, train_data)
        
        # Get classification from Claude
        response = client.messages.create(
            model=MODEL_NAME,
            max_tokens=200,
            messages=[{"role": "user", "content": prompt}]
        )
        
        return parse_category(response.content[0].text.strip())
    
    return classify
Use the pipeline
classifier = build_classification_pipeline(train_df, categories)
result = classifier("I need to cancel my policy due to moving abroad")
print(f"Category: {result}")

Best Practices for Production

• Iterate on category definitions: Clear, non-overlapping definitions are critical.
• Curate few-shot examples: Choose diverse, high-quality examples for each category.
• Monitor and retrain: Periodically update your vector database with new examples.
• Add confidence thresholds: If Claude's confidence is low, route to a human reviewer.
• Log all classifications: Maintain an audit trail for compliance and improvement.

Key Takeaways

• LLMs excel at complex classification: Claude handles nuanced business rules, limited data, and provides explainable results that traditional ML struggles with.
• Combine three techniques for 95%+ accuracy: Start with prompt engineering (70% accuracy), add RAG (85–90%), and finish with chain-of-thought reasoning (95%+).
• RAG dynamically retrieves relevant examples: By embedding training data and retrieving similar examples for each query, you provide Claude with the most useful context.
• Chain-of-thought reasoning reduces errors: Asking Claude to reason step-by-step before outputting a category improves accuracy and provides transparency.
• This framework is reusable: Adapt the pipeline to any classification task—support tickets, document routing, content moderation, and more.