Building a High-Accuracy Classification System with Claude: From 70% to 95%+ Accuracy

Classification is a cornerstone of many business workflows—from routing support tickets to moderating content. Traditional machine learning approaches often struggle with complex business rules, limited training data, and the need for explainable results. Enter Claude: a powerful LLM that can handle these challenges with elegance.

In this guide, you'll build a production-ready classification system that categorizes insurance support tickets into 10 distinct categories. You'll learn how to progressively improve classification accuracy from a baseline of ~70% to over 95% by combining prompt engineering, retrieval-augmented generation (RAG), and chain-of-thought reasoning.

What You'll Learn

• How to design effective classification prompts for Claude
• How to implement RAG to augment Claude with relevant examples
• How to use chain-of-thought reasoning for complex classifications
• How to evaluate and iterate on your classification system
• How to achieve explainable, transparent classifications

Prerequisites

• Python 3.11+ with basic familiarity
• An Anthropic API key (get one here)
• A VoyageAI API key (optional—embeddings can be pre-computed)
• Basic understanding of classification problems

Setup

First, install the required packages:

pip install anthropic voyageai pandas matplotlib scikit-learn numpy

Then, set up your API keys and initialize the Claude client:

import os
from anthropic import Anthropic
Load API keys from environment
anthropic_api_key = os.environ.get("ANTHROPIC_API_KEY")
client = Anthropic(api_key=anthropic_api_key)
Set model name
MODEL_NAME = "claude-3-opus-20240229"

Step 1: Data Preparation

Proper data preparation is critical. You'll need a labeled dataset of support tickets with their correct categories. For this guide, we'll use a synthetically generated dataset of insurance support tickets.

import pandas as pd
from sklearn.model_selection import train_test_split
Load your dataset
df = pd.read_csv("insurance_tickets.csv")
Split into training and test sets
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
print(f"Training samples: {len(train_df)}")
print(f"Test samples: {len(test_df)}")

Step 2: Define Category Definitions

Claude needs clear, detailed category definitions to classify accurately. Here are the 10 categories for our insurance support ticket system:

• Billing Inquiries – Questions about invoices, charges, fees, premiums, payment methods, and due dates.
• Policy Administration – Requests for policy changes, updates, cancellations, renewals, or reinstatements.
• Claims Assistance – Questions about the claims process, filing procedures, documentation, and payout timelines.
• Coverage Explanations – Questions about what is covered, coverage limits, exclusions, and deductibles.
• Account Management – Login issues, profile updates, password resets, and multi-factor authentication.
• Documentation Requests – Requests for policy documents, certificates, ID cards, or claim forms.
• Agent Assistance – Requests to speak with an agent, complaints about service, or escalation requests.
• Fraud Concerns – Reports of suspicious activity, identity theft, or potential fraud.
• Third-Party Coordination – Questions about coordination with other insurers, providers, or legal entities.
• General Inquiries – Miscellaneous questions that don't fit other categories.

Step 3: Baseline Prompt Engineering

Start with a simple zero-shot prompt to establish a baseline:

def classify_ticket_zero_shot(ticket_text, categories):
    prompt = f"""You are an insurance support ticket classifier. Classify the following ticket into one of these categories:
{categories}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

This baseline typically achieves ~70-75% accuracy. Let's improve it.

Step 4: Implement Retrieval-Augmented Generation (RAG)

RAG dramatically improves accuracy by providing Claude with relevant examples from your training data. Here's how to implement it:

import voyageai
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
Initialize VoyageAI client
vo = voyageai.Client(api_key=os.environ.get("VOYAGE_API_KEY"))
Generate embeddings for training data
train_embeddings = vo.embed(
    train_df["ticket_text"].tolist(),
    model="voyage-2",
    input_type="document"
).embeddings
def retrieve_similar_examples(query, k=3):
    """Retrieve k most similar examples from training data."""
    query_embedding = vo.embed(
        [query],
        model="voyage-2",
        input_type="query"
    ).embeddings[0]
    
    similarities = cosine_similarity([query_embedding], train_embeddings)[0]
    top_indices = np.argsort(similarities)[-k:][::-1]
    
    examples = []
    for idx in top_indices:
        examples.append({
            "ticket": train_df.iloc[idx]["ticket_text"],
            "category": train_df.iloc[idx]["category"]
        })
    return examples
def classify_with_rag(ticket_text, categories):
    # Retrieve similar examples
    examples = retrieve_similar_examples(ticket_text, k=3)
    
    # Format examples
    examples_text = "\n\n".join([
        f"Example {i+1}:\nTicket: {ex['ticket']}\nCategory: {ex['category']}"
        for i, ex in enumerate(examples)
    ])
    
    prompt = f"""You are an insurance support ticket classifier. Use the following examples as reference:
{examples_text}
Now classify this ticket into one of these categories:
{categories}
Ticket: {ticket_text}
Category:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text.strip()

With RAG, accuracy typically jumps to ~85-90%.

Step 5: Add Chain-of-Thought Reasoning

Chain-of-thought (CoT) reasoning forces Claude to explain its decision-making process, which improves accuracy and provides transparency:

def classify_with_cot(ticket_text, categories):
    # Retrieve similar examples
    examples = retrieve_similar_examples(ticket_text, k=3)
    
    examples_text = "\n\n".join([
        f"Example {i+1}:\nTicket: {ex['ticket']}\nCategory: {ex['category']}"
        for i, ex in enumerate(examples)
    ])
    
    prompt = f"""You are an insurance support ticket classifier. Use the following examples as reference:
{examples_text}
Now classify this ticket. First, think step-by-step about which category fits best, then provide your final answer.
Categories:
{categories}
Ticket: {ticket_text}
Let's think step by step:"""
    
    response = client.messages.create(
        model=MODEL_NAME,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}]
    )
    
    full_response = response.content[0].text.strip()
    
    # Extract the final category (assumes it's on the last line)
    lines = full_response.split("\n")
    final_category = lines[-1].strip()
    
    return final_category, full_response

With CoT reasoning, accuracy reaches 95%+.

Step 6: Evaluation

Test your system on the held-out test set:

from sklearn.metrics import accuracy_score, classification_report
predictions = []
for ticket in test_df["ticket_text"]:
    pred, _ = classify_with_cot(ticket, category_definitions)
    predictions.append(pred)
accuracy = accuracy_score(test_df["category"], predictions)
print(f"Accuracy: {accuracy:.2%}")
print(classification_report(test_df["category"], predictions))

Best Practices for Production

• Cache embeddings: Pre-compute and store embeddings to avoid repeated API calls.
• Use temperature 0: For deterministic classification, set temperature=0.
• Validate output format: Use structured output (JSON mode) to ensure parseable results.
• Monitor confidence: Track cases where Claude is uncertain and route them for human review.
• Iterate on categories: Refine category definitions based on misclassifications.

Key Takeaways

• Start simple, then iterate: Begin with zero-shot prompting, then add RAG and chain-of-thought reasoning to progressively improve accuracy from ~70% to 95%+.
• RAG is a game-changer: Providing relevant examples from your training data dramatically improves classification accuracy without requiring fine-tuning.
• Chain-of-thought reasoning adds transparency: CoT not only improves accuracy but also provides explainable results that help debug misclassifications.
• Claude excels at complex business rules: LLMs handle nuanced, multi-faceted classification problems that traditional ML approaches struggle with.
• Production readiness requires validation: Always validate output formats, monitor confidence, and have a fallback for uncertain cases.