Building Production-Ready RAG Systems with Claude: From Basic to Advanced

Retrieval Augmented Generation (RAG) is transforming how enterprises leverage Claude for domain-specific tasks. While Claude excels at general knowledge, it needs RAG to answer questions about your internal documentation, customer support history, or proprietary data. This guide walks you through building a production-ready RAG system, complete with proper evaluation and optimization techniques.

Why RAG Matters for Enterprise Applications

Claude's training data has a cutoff date, and it doesn't know your company's internal processes. RAG bridges this gap by:

• Grounding responses in your verified documentation
• Reducing hallucinations by providing relevant context
• Enabling real-time updates without retraining models
• Maintaining data privacy by keeping sensitive info in your vector database

Prerequisites and Setup

Before diving in, ensure you have:

• An Anthropic API key
• A Voyage AI API key for embeddings
• Python 3.8+ environment

Required Libraries

pip install anthropic voyageai pandas numpy matplotlib scikit-learn

Initialize Your Vector Database

For this guide, we'll use an in-memory vector database. In production, consider hosted solutions like Pinecone, Weaviate, or Chroma.

import voyageai
from anthropic import Anthropic
import numpy as np
Initialize clients
vo = voyageai.Client(api_key="your-voyage-api-key")
anthropic = Anthropic(api_key="your-anthropic-api-key")
class InMemoryVectorDB:
    def __init__(self):
        self.documents = []
        self.embeddings = []
    
    def add_documents(self, documents):
        texts = [doc['content'] for doc in documents]
        embeddings = vo.embed(texts, model="voyage-2").embeddings
        self.documents.extend(documents)
        self.embeddings.extend(embeddings)
    
    def search(self, query, k=3):
        query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
        similarities = [
            np.dot(query_embedding, doc_emb) / 
            (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb))
            for doc_emb in self.embeddings
        ]
        top_indices = np.argsort(similarities)[-k:][::-1]
        return [self.documents[i] for i in top_indices]

Level 1: Basic RAG Pipeline

Let's start with a "naive" RAG implementation. This three-step process forms the foundation:

• Chunk documents by headings
• Embed each chunk using Voyage AI
• Retrieve relevant chunks via cosine similarity

Document Chunking Strategy

def chunk_by_headings(document):
    """Split document by markdown headings"""
    chunks = []
    current_heading = None
    current_content = []
    
    for line in document.split('\n'):
        if line.startswith('#'):
            if current_heading:
                chunks.append({
                    'heading': current_heading,
                    'content': '\n'.join(current_content)
                })
            current_heading = line
            current_content = []
        else:
            current_content.append(line)
    
    if current_heading:
        chunks.append({
            'heading': current_heading,
            'content': '\n'.join(current_content)
        })
    
    return chunks

Query Execution

def basic_rag_query(query, vector_db, k=3):
    # Retrieve relevant chunks
    relevant_chunks = vector_db.search(query, k=k)
    
    # Construct context
    context = "\n\n".join([
        f"[{chunk['heading']}]\n{chunk['content']}"
        for chunk in relevant_chunks
    ])
    
    # Generate response with Claude
    response = anthropic.messages.create(
        model="claude-3-sonnet-20240229",
        max_tokens=1000,
        messages=[{
            "role": "user",
            "content": f"Based on the following documentation, answer the question.\n\nDocumentation:\n{context}\n\nQuestion: {query}"
        }]
    )
    
    return response.content[0].text

Building a Robust Evaluation System

"Vibes-based" evaluation won't cut it for production. You need quantitative metrics. Let's build an evaluation suite that measures both retrieval and end-to-end performance.

Creating a Test Dataset

Generate 100+ synthetic QA pairs with:

• A question
• Ground truth relevant chunks
• A correct answer

# Example evaluation sample
{
    "question": "How do I handle rate limits in Claude API?",
    "relevant_chunks": [
        "rate_limiting.md#overview",
        "rate_limiting.md#best-practices"
    ],
    "correct_answer": "Implement exponential backoff and monitor your usage..."
}

Key Metrics Explained

#### Retrieval Metrics

Precision measures how many retrieved chunks are actually relevant:

Precision = True Positives / (True Positives + False Positives)

• High precision = fewer irrelevant chunks
• Our system retrieves minimum 3 chunks, which can lower precision

Recall measures how many relevant chunks we captured:

Recall = True Positives / (True Positives + False Negatives)

• High recall = comprehensive coverage
• Critical for complex questions needing multiple sources

F1 Score balances precision and recall:

F1 = 2  (Precision  Recall) / (Precision + Recall)

Mean Reciprocal Rank (MRR) measures how early the first relevant result appears:

MRR = (1/Q) * Σ(1/rank_of_first_relevant)

• Higher MRR = better ranking quality
• Crucial for user experience

#### End-to-End Metric Accuracy measures if Claude's final answer is correct given the retrieved context.

Implementing the Evaluation

def evaluate_retrieval(query, expected_chunks, retrieved_chunks):
    """Calculate retrieval metrics for a single query"""
    retrieved_set = set(retrieved_chunks)
    expected_set = set(expected_chunks)
    
    true_positives = len(retrieved_set & expected_set)
    
    precision = true_positives / len(retrieved_set) if retrieved_set else 0
    recall = true_positives / len(expected_set) if expected_set else 0
    f1 = 2  (precision  recall) / (precision + recall) if (precision + recall) > 0 else 0
    
    # Calculate MRR
    mrr = 0
    for rank, chunk in enumerate(retrieved_chunks, 1):
        if chunk in expected_set:
            mrr = 1 / rank
            break
    
    return {
        "precision": precision,
        "recall": recall,
        "f1": f1,
        "mrr": mrr
    }

Level 2: Summary Indexing

Basic RAG struggles with long documents where relevant information spans multiple sections. Summary indexing solves this by creating concise summaries of each document chunk.

Implementation

def create_summary_index(documents, anthropic_client):
    """Create summary for each document chunk"""
    summary_index = []
    
    for doc in documents:
        response = anthropic_client.messages.create(
            model="claude-3-haiku-20240307",
            max_tokens=200,
            messages=[{
                "role": "user",
                "content": f"Summarize this document chunk in 2-3 sentences:\n\n{doc['content']}"
            }]
        )
        
        summary_index.append({
            "original": doc,
            "summary": response.content[0].text
        })
    
    return summary_index

Hybrid Retrieval

def hybrid_retrieval(query, summary_index, vector_db, k=3):
    # Search both original content and summaries
    summary_texts = [item['summary'] for item in summary_index]
    summary_embeddings = vo.embed(summary_texts, model="voyage-2").embeddings
    
    query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
    
    # Combine scores from both searches
    scores = []
    for i, (doc, summary_emb) in enumerate(zip(vector_db.documents, summary_embeddings)):
        doc_similarity = np.dot(query_embedding, vector_db.embeddings[i])
        summary_similarity = np.dot(query_embedding, summary_emb)
        combined_score = 0.7  doc_similarity + 0.3  summary_similarity
        scores.append(combined_score)
    
    top_indices = np.argsort(scores)[-k:][::-1]
    return [vector_db.documents[i] for i in top_indices]

Level 3: Re-Ranking with Claude

Re-ranking uses Claude to evaluate and reorder retrieved chunks, significantly improving MRR.

Implementation

def rerank_with_claude(query, chunks, anthropic_client, top_k=3):
    """Use Claude to re-rank retrieved chunks"""
    chunk_texts = [f"Chunk {i+1}:\n{chunk['content']}" for i, chunk in enumerate(chunks)]
    
    prompt = f"""Given the query: "{query}"
Rank these document chunks by relevance (1 = most relevant):
{'\n\n'.join(chunk_texts)}
Return the chunk numbers in order of relevance, separated by commas."""
    
    response = anthropic_client.messages.create(
        model="claude-3-haiku-20240307",
        max_tokens=100,
        messages=[{"role": "user", "content": prompt}]
    )
    
    # Parse ranking
    ranking = [int(x.strip()) - 1 for x in response.content[0].text.split(',')]
    ranked_chunks = [chunks[i] for i in ranking[:top_k]]
    
    return ranked_chunks

Performance Results

After implementing these optimizations, here are the improvements over basic RAG:

Production Considerations

• Rate Limiting: Tier 2+ API access recommended for full evaluation runs
• Cost Management: Use Claude Haiku for summaries and re-ranking, Sonnet for final answers
• Vector Database: Migrate from in-memory to Pinecone/Weaviate for production
• Caching: Cache embeddings and summaries to reduce API calls
• Monitoring: Track retrieval metrics in production to detect drift

Key Takeaways

• Separate retrieval and generation evaluation: Measure your pipeline's components independently to identify bottlenecks
• Summary indexing improves recall: By creating searchable summaries, you capture relevant content that might be missed by keyword matching alone
• Re-ranking with Claude boosts MRR significantly: From 0.74 to 0.87, showing that LLM-based re-ranking dramatically improves result quality
• Start simple, then optimize: Begin with basic RAG, establish baseline metrics, then iteratively add advanced techniques
• Use appropriate Claude models: Haiku for cost-effective preprocessing, Sonnet for high-quality final responses