Contextual Retrieval: How to Reduce RAG Failure Rates by 35% with Claude

Retrieval Augmented Generation (RAG) is the backbone of many enterprise AI applications—from customer support bots to internal knowledge base Q&A. But there's a persistent problem: when you split documents into chunks for retrieval, individual chunks often lose the surrounding context. A code snippet like def calculate_tax(): makes little sense without knowing it belongs to a payroll module.

Contextual Retrieval solves this by prepending a short, chunk-specific context to each piece before embedding. The result? A 35% reduction in top-20 retrieval failure rates across diverse datasets, according to Anthropic's internal testing.

In this guide, you'll learn how to build a Contextual Retrieval system using Claude, Voyage AI embeddings, and Anthropic's prompt caching to keep costs manageable. We'll walk through a complete implementation using a dataset of 9 codebases.

What You'll Need

Before diving in, make sure you have:

• Python 3.8+ installed
• API keys for Anthropic, Voyage AI, and Cohere (for reranking)
• ~$5–10 in API credits to run the full dataset
• 30–45 minutes of focused time

We'll use a pre-chunked dataset of 9 codebases with 248 queries, each containing a "golden chunk" for evaluation. Our metric is Pass@k—whether the golden chunk appears in the top-k retrieved results.

Step 1: Establish a Baseline with Basic RAG

First, let's set up a simple RAG pipeline to measure where we start. We'll split documents into chunks, embed them with Voyage AI, and retrieve using cosine similarity.

import voyageai
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
Initialize Voyage AI client
vo = voyageai.Client(api_key="YOUR_VOYAGE_API_KEY")
Load your chunks (example structure)
chunks = load_chunks_from_json("data/codebase_chunks.json")
Embed all chunks
chunk_texts = [chunk["content"] for chunk in chunks]
embeddings = vo.embed(chunk_texts, model="voyage-2").embeddings
For a query, embed and find top-k
query = "How is tax calculated in the payroll module?"
query_embedding = vo.embed([query], model="voyage-2").embeddings[0]
similarities = cosine_similarity([query_embedding], embeddings)[0]
top_k_indices = np.argsort(similarities)[-10:][::-1]

This baseline typically achieves around 87% Pass@10 on our codebase dataset. Not bad—but we can do better.

Step 2: Implement Contextual Embeddings

The core idea is simple: before embedding each chunk, ask Claude to generate a short piece of context that explains where the chunk fits in the broader document.

import anthropic
client = anthropic.Anthropic(api_key="YOUR_ANTHROPIC_API_KEY")
def generate_chunk_context(chunk_text, full_document):
    """
    Ask Claude to generate context for a single chunk.
    """
    prompt = f"""
    Here is a chunk of text from a larger document:
    <chunk>{chunk_text}</chunk>
    
    Here is the full document for context:
    <document>{full_document}</document>
    
    Please provide a concise context (1-3 sentences) that describes:
    - What this chunk is about
    - How it fits into the larger document
    - Any important entities or concepts it references
    """
    
    response = client.messages.create(
        model="claude-3-haiku-20240307",
        max_tokens=150,
        messages=[{"role": "user", "content": prompt}]
    )
    return response.content[0].text
Apply to each chunk
contextual_chunks = []
for chunk in chunks:
    context = generate_chunk_context(chunk["content"], chunk["full_document"])
    contextual_chunks.append(f"{context}\n\n{chunk['content']}")

Now embed these contextualized chunks instead of the raw text. In our tests, this alone boosted Pass@10 from ~87% to ~95%.

Why It Works

Traditional chunking loses the forest for the trees. A chunk containing only def calculate_tax(): might match queries about "tax" but miss queries about "payroll deductions." By adding context like "This function is part of the Payroll module and calculates federal income tax based on employee W-4 data," the embedding captures both the specific function and its broader relevance.

Step 3: Optimize Costs with Prompt Caching

Generating context for every chunk sounds expensive—and it would be, without prompt caching. Anthropic's prompt caching allows you to reuse the full document across multiple chunk-context requests, dramatically reducing token costs.

# Use prompt caching to avoid re-sending the full document
response = client.messages.create(
    model="claude-3-haiku-20240307",
    max_tokens=150,
    system=[
        {
            "type": "text",
            "text": full_document,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[{"role": "user", "content": f"Generate context for this chunk: {chunk_text}"}]
)

With prompt caching, the full document is sent once and cached for subsequent requests. For a 100-chunk document, this reduces input tokens by ~99% compared to naive per-chunk requests.

Note: Prompt caching is available on Anthropic's first-party API and coming soon to AWS Bedrock and GCP Vertex AI.

Step 4: Add Contextual BM25 for Hybrid Search

Contextual retrieval isn't limited to embeddings. You can apply the same chunk-specific context to BM25—a keyword-based retrieval algorithm that excels at exact matches.

# Install BM25 via Elasticsearch or a lightweight library
from rank_bm25 import BM25Okapi
Tokenize contextualized chunks
tokenized_corpus = [chunk.split() for chunk in contextual_chunks]
bm25 = BM25Okapi(tokenized_corpus)
Query with BM25
query_tokens = query.split()
bm25_scores = bm25.get_scores(query_tokens)
bm25_top_k = np.argsort(bm25_scores)[-10:][::-1]
Combine with embedding scores (hybrid search)
Normalize both score sets and average them
combined_scores = (normalized_embedding_scores + normalized_bm25_scores) / 2

Contextual BM25 improves keyword matching because the added context includes terms that might not appear in the chunk itself. For example, a chunk about calculate_tax() might not contain the word "payroll," but the context does—so BM25 can now match queries about "payroll."

Step 5: Rerank for Precision

Even with contextual retrieval, the top-10 results may include irrelevant chunks. A reranker (also called a cross-encoder) re-scores the top candidates by comparing each one directly against the query.

import cohere
co = cohere.Client("YOUR_COHERE_API_KEY")
Get top-20 from hybrid search, then rerank to top-5
reranked = co.rerank(
    query=query,
    documents=[contextual_chunks[i] for i in top_20_indices],
    top_n=5,
    model="rerank-english-v3.0"
)
final_results = [result.document for result in reranked.results]

Reranking adds a small latency cost (typically 100–300ms) but can push Pass@5 from ~90% to ~98%.

Production Considerations

For AWS Bedrock Users

Anthropic and AWS have collaborated on a Lambda function that automates context generation. You can deploy it as a custom chunking option when configuring a Bedrock Knowledge Base. The code is available in the contextual-rag-lambda-function folder of the cookbook repository.

Cost Management

• Prompt caching is your best friend—use it for every document with 3+ chunks.
• Batch context generation by processing chunks for one document at a time.
• Choose the right model: Claude 3 Haiku is fast and cheap for context generation; use Sonnet or Opus only if you need deeper reasoning.

Evaluation

Always measure before and after. Our dataset uses Pass@k, but you should create an evaluation set from your own documents. A good rule of thumb: 50–100 queries with manually verified golden chunks.

Key Takeaways

• Contextual Embeddings reduce retrieval failure rates by up to 35% by adding chunk-specific context before embedding, solving the "lost context" problem in traditional RAG.
• Prompt caching makes this approach cost-effective by caching the full document and reusing it across multiple chunk-context requests, reducing token usage by ~99%.
• Combine Contextual Embeddings with Contextual BM25 for hybrid search that captures both semantic meaning and exact keyword matches.
• Reranking adds the final polish, pushing Pass@5 accuracy to ~98% with minimal latency overhead.
• AWS Bedrock users can deploy this as a Lambda function for seamless integration with Knowledge Bases, making production deployment straightforward.