Building Production-Ready RAG Systems with Claude: From Basic to Advanced

Retrieval Augmented Generation (RAG) is one of the most powerful patterns for extending Claude's capabilities to your specific business context. While Claude excels at general knowledge tasks, it needs RAG to answer questions about your internal documentation, customer support history, or proprietary data.

In this guide, we'll walk through building a complete RAG system using Claude and Voyage AI embeddings, then systematically improve it using advanced techniques. By the end, you'll understand how to achieve significant performance gains—from 71% to 81% end-to-end accuracy—using the same evaluation-driven approach used in production systems.

Understanding the RAG Architecture

Before diving into code, let's understand what we're building. A RAG system has three core components:

• Ingestion Pipeline: Chunks documents, generates embeddings, and stores them in a vector database
• Retrieval System: Finds relevant document chunks given a user query
• Generation System: Feeds retrieved context to Claude to produce an answer

The magic of RAG is that it combines the flexibility of LLMs with the precision of information retrieval. Claude doesn't need to memorize your data—it just needs to read the right context at inference time.

Prerequisites and Setup

You'll need:

• An Anthropic API key (Tier 2 or above recommended for evaluation)
• A Voyage AI API key for embeddings
• Python 3.8+ with anthropic, voyageai, pandas, numpy, matplotlib, and scikit-learn

import anthropic
import voyageai
import numpy as np
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

Initialize clients
claude = anthropic.Anthropic(api_key="your-anthropic-key")
vo = voyageai.Client(api_key="your-voyage-key")

Level 1: Building a Basic RAG Pipeline

Let's start with what's often called "Naive RAG"—a straightforward implementation that demonstrates the core concepts.

Step 1: Chunk Your Documents

Chunking strategy matters enormously. For this guide, we'll chunk by heading, keeping content under each subheading together. This preserves semantic coherence better than fixed-length chunks.

def chunk_by_headings(document):
    chunks = []
    current_heading = None
    current_content = []
    
    for line in document.split('\n'):
        if line.startswith('##') or line.startswith('###'):
            if current_heading and current_content:
                chunks.append({
                    'heading': current_heading,
                    'content': '\n'.join(current_content)
                })
            current_heading = line
            current_content = []
        else:
            current_content.append(line)
    
    # Don't forget the last chunk
    if current_heading and current_content:
        chunks.append({
            'heading': current_heading,
            'content': '\n'.join(current_content)
        })
    
    return chunks

Step 2: Generate Embeddings

We'll use Voyage AI's embeddings, which are optimized for retrieval tasks. Each chunk gets converted into a vector that captures its semantic meaning.

def embed_chunks(chunks):
    texts = [chunk['content'] for chunk in chunks]
    embeddings = vo.embed(texts, model="voyage-2").embeddings
    
    for i, chunk in enumerate(chunks):
        chunk['embedding'] = embeddings[i]
    
    return chunks

Step 3: Store in a Vector Database

For this example, we'll use an in-memory vector store. In production, you'd use Pinecone, Weaviate, or similar.

class InMemoryVectorDB:
    def __init__(self):
        self.chunks = []
        self.embeddings = None
    
    def add_chunks(self, chunks):
        self.chunks.extend(chunks)
        self.embeddings = np.array([c['embedding'] for c in chunks])
    
    def search(self, query_embedding, k=3):
        similarities = cosine_similarity([query_embedding], self.embeddings)[0]
        top_indices = np.argsort(similarities)[-k:][::-1]
        return [self.chunks[i] for i in top_indices]

Step 4: Generate Answers with Claude

Now we retrieve relevant chunks and feed them to Claude as context.

def answer_with_rag(query, vector_db):
    # Embed the query
    query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
    
    # Retrieve relevant chunks
    relevant_chunks = vector_db.search(query_embedding, k=3)
    context = "\n\n".join([c['content'] for c in relevant_chunks])
    
    # Generate answer with Claude
    response = claude.messages.create(
        model="claude-3-sonnet-20240229",
        max_tokens=1000,
        messages=[{
            "role": "user",
            "content": f"Context: {context}\n\nQuestion: {query}\n\nAnswer the question based on the context provided."
        }]
    )
    
    return response.content[0].text

Building a Robust Evaluation System

Here's where most RAG tutorials stop—but we're just getting started. To build a production system, you need to measure two things independently:

• Retrieval Performance: Is the system finding the right documents?
• End-to-End Performance: Is Claude generating correct answers?

Creating an Evaluation Dataset

We synthetically generated 100 evaluation samples, each containing:

• A question
• Ground truth relevant chunks
• A correct answer

# Preview the evaluation dataset
import json

with open('evaluation/docs_evaluation_dataset.json', 'r') as f:
    eval_data = json.load(f)
print(f"Dataset size: {len(eval_data)} samples")
print(f"Sample question: {eval_data[0]['question']}")
print(f"Number of relevant chunks: {len(eval_data[0]['relevant_chunks'])}")

Key Retrieval Metrics

Precision: Of the chunks we retrieved, how many were actually relevant?

Precision = True Positives / Total Retrieved

High precision means you're not wasting Claude's context window on irrelevant information.

Recall: Of all the relevant chunks that exist, how many did we retrieve?

Recall = True Positives / Total Relevant

High recall ensures Claude has all the information it needs.

F1 Score: The harmonic mean of precision and recall.

F1 = 2  (Precision  Recall) / (Precision + Recall)

Mean Reciprocal Rank (MRR): How high did the first relevant chunk appear in our results?

def calculate_mrr(retrieved_chunks, relevant_chunks):
    for i, chunk in enumerate(retrieved_chunks):
        if chunk['id'] in relevant_chunks:
            return 1.0 / (i + 1)
    return 0.0

MRR is particularly important because Claude's attention is strongest on the first few chunks it receives.

Level 2: Summary Indexing

Our basic RAG has a problem: it retrieves individual chunks, but many questions require synthesizing information across multiple sections. Summary indexing solves this by creating higher-level summaries of document sections.

def create_summary_index(chunks):
    summary_chunks = []
    
    # Group chunks by parent section
    sections = {}
    for chunk in chunks:
        parent = chunk['heading'].split(' > ')[0]
        if parent not in sections:
            sections[parent] = []
        sections[parent].append(chunk)
    
    # Create summaries for each section
    for section_name, section_chunks in sections.items():
        combined_text = " ".join([c['content'] for c in section_chunks])
        
        # Use Claude to generate a summary
        response = claude.messages.create(
            model="claude-3-haiku-20240307",
            max_tokens=500,
            messages=[{
                "role": "user",
                "content": f"Summarize this documentation section in 2-3 sentences:\n\n{combined_text}"
            }]
        )
        
        summary = response.content[0].text
        summary_embedding = vo.embed([summary], model="voyage-2").embeddings[0]
        
        summary_chunks.append({
            'heading': section_name,
            'content': summary,
            'embedding': summary_embedding,
            'original_chunks': section_chunks
        })
    
    return summary_chunks

Now when a query comes in, we first search the summary index, then retrieve the original chunks from the most relevant sections. This gives us both breadth (via summaries) and depth (via original chunks).

Level 3: Re-Ranking with Claude

Even with summary indexing, our retrieval might surface chunks that are tangentially related but not directly useful. Re-ranking uses Claude itself to evaluate relevance before passing context to the generation step.

def rerank_with_claude(query, candidate_chunks, top_k=3):
    # Have Claude score each chunk's relevance
    scored_chunks = []
    
    for chunk in candidate_chunks:
        response = claude.messages.create(
            model="claude-3-haiku-20240307",
            max_tokens=10,
            messages=[{
                "role": "user",
                "content": f"On a scale of 0-10, how relevant is this text to the question?\n\nQuestion: {query}\n\nText: {chunk['content'][:500]}\n\nAnswer with just a number."
            }]
        )
        
        try:
            score = float(response.content[0].text.strip())
        except ValueError:
            score = 0
        
        scored_chunks.append((score, chunk))
    
    # Sort by score and return top_k
    scored_chunks.sort(reverse=True, key=lambda x: x[0])
    return [chunk for _, chunk in scored_chunks[:top_k]]

This adds latency but dramatically improves precision. The key insight: Claude is better at judging relevance than cosine similarity alone.

Results: The Performance Gains

After implementing all three levels, here's what we achieved:

The most dramatic improvement is in MRR (0.74 → 0.87), meaning the first retrieved chunk is much more likely to be relevant. This directly impacts end-to-end accuracy because Claude sees the most relevant information first.

Production Considerations

• Rate Limits: Full evaluations can hit rate limits. Use Tier 2+ accounts and consider sampling your evaluation dataset.
• Cost Management: Summary indexing and re-ranking add token costs. Balance against the accuracy gains you need.
• Vector Database: Move from in-memory to Pinecone, Weaviate, or pgvector for production.
• Chunking Strategy: Experiment with different strategies—semantic chunking, sliding windows, or recursive splitting.

Key Takeaways

• Evaluate retrieval and generation separately: You can't optimize what you don't measure. Build a ground truth dataset and track precision, recall, F1, and MRR independently from end-to-end accuracy.
• Summary indexing bridges the gap: Many questions require synthesizing information across multiple sections. Summary-level retrieval followed by chunk-level retrieval gives you both breadth and depth.
• Re-ranking with Claude beats pure vector search: Cosine similarity is fast but imprecise. Using Claude to judge relevance before generation significantly improves MRR and final answer quality.
• Start simple, then optimize: Begin with basic RAG, establish your baseline metrics, then add complexity only where it moves the needle. Our biggest gain came from summary indexing, not re-ranking.
• MRR is your most actionable metric: Improving the rank of the first relevant chunk directly improves Claude's ability to generate correct answers, since it sees the most relevant context first.