RAG Implementation

Master Retrieval-Augmented Generation (RAG) to build LLM applications that provide accurate, grounded responses using external knowledge sources.

Use this skill when

• Building Q&A systems over proprietary documents

• Creating chatbots with current, factual information

• Implementing semantic search with natural language queries

• Reducing hallucinations with grounded responses

• Enabling LLMs to access domain-specific knowledge

• Building documentation assistants

• Creating research tools with source citation

Do not use this skill when

• You only need purely generative writing without retrieval

• The dataset is too small to justify embeddings

• You cannot store or process the source data safely

Instructions

• Define the corpus, update cadence, and evaluation targets.

• Choose embedding models and vector store based on scale.

• Build ingestion, chunking, and retrieval with reranking.

• Evaluate with grounded QA metrics and monitor drift.

Safety

• Redact sensitive data and enforce access controls.

• Avoid exposing source documents in responses when restricted.

Core Components

1. Vector Databases

Purpose: Store and retrieve document embeddings efficiently

Options:

• Pinecone: Managed, scalable, fast queries

• Weaviate: Open-source, hybrid search

• Milvus: High performance, on-premise

• Chroma: Lightweight, easy to use

• Qdrant: Fast, filtered search

• FAISS: Meta's library, local deployment

2. Embeddings

Purpose: Convert text to numerical vectors for similarity search

Models:

• text-embedding-ada-002 (OpenAI): General purpose, 1536 dims

• all-MiniLM-L6-v2 (Sentence Transformers): Fast, lightweight

• e5-large-v2: High quality, multilingual

• Instructor: Task-specific instructions

• bge-large-en-v1.5: SOTA performance

3. Retrieval Strategies

Approaches:

• Dense Retrieval: Semantic similarity via embeddings

• Sparse Retrieval: Keyword matching (BM25, TF-IDF)

• Hybrid Search: Combine dense + sparse

• Multi-Query: Generate multiple query variations

• HyDE: Generate hypothetical documents

4. Reranking

Purpose: Improve retrieval quality by reordering results

Methods:

• Cross-Encoders: BERT-based reranking

• Cohere Rerank: API-based reranking

• Maximal Marginal Relevance (MMR): Diversity + relevance

• LLM-based: Use LLM to score relevance

Quick Start

from langchain.document_loaders import DirectoryLoader from langchain.text_splitters import RecursiveCharacterTextSplitter from langchain.embeddings import OpenAIEmbeddings from langchain.vectorstores import Chroma from langchain.chains import RetrievalQA from langchain.llms import OpenAI

1. Load documents

loader = DirectoryLoader('./docs', glob="**/*.txt") documents = loader.load()

2. Split into chunks

text_splitter = RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=200, length_function=len ) chunks = text_splitter.split_documents(documents)

3. Create embeddings and vector store

embeddings = OpenAIEmbeddings() vectorstore = Chroma.from_documents(chunks, embeddings)

4. Create retrieval chain

qa_chain = RetrievalQA.from_chain_type( llm=OpenAI(), chain_type="stuff", retriever=vectorstore.as_retriever(search_kwargs={"k": 4}), return_source_documents=True )

5. Query

result = qa_chain({"query": "What are the main features?"}) print(result['result']) print(result['source_documents'])

Advanced RAG Patterns

Pattern 1: Hybrid Search

from langchain.retrievers import BM25Retriever, EnsembleRetriever

Sparse retriever (BM25)

bm25_retriever = BM25Retriever.from_documents(chunks) bm25_retriever.k = 5

Dense retriever (embeddings)

embedding_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

Combine with weights

ensemble_retriever = EnsembleRetriever( retrievers=[bm25_retriever, embedding_retriever], weights=[0.3, 0.7] )

Pattern 2: Multi-Query Retrieval

from langchain.retrievers.multi_query import MultiQueryRetriever

Generate multiple query perspectives

retriever = MultiQueryRetriever.from_llm( retriever=vectorstore.as_retriever(), llm=OpenAI() )

Single query → multiple variations → combined results

results = retriever.get_relevant_documents("What is the main topic?")

Pattern 3: Contextual Compression

from langchain.retrievers import ContextualCompressionRetriever from langchain.retrievers.document_compressors import LLMChainExtractor

compressor = LLMChainExtractor.from_llm(llm)

compression_retriever = ContextualCompressionRetriever( base_compressor=compressor, base_retriever=vectorstore.as_retriever() )

Returns only relevant parts of documents

compressed_docs = compression_retriever.get_relevant_documents("query")

Pattern 4: Parent Document Retriever

from langchain.retrievers import ParentDocumentRetriever from langchain.storage import InMemoryStore

Store for parent documents

store = InMemoryStore()

Small chunks for retrieval, large chunks for context

child_splitter = RecursiveCharacterTextSplitter(chunk_size=400) parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)

retriever = ParentDocumentRetriever( vectorstore=vectorstore, docstore=store, child_splitter=child_splitter, parent_splitter=parent_splitter )

Document Chunking Strategies

Recursive Character Text Splitter

from langchain.text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=200, length_function=len, separators=["\n\n", "\n", " ", ""] # Try these in order )

Token-Based Splitting

from langchain.text_splitters import TokenTextSplitter

splitter = TokenTextSplitter( chunk_size=512, chunk_overlap=50 )

Semantic Chunking

from langchain.text_splitters import SemanticChunker

splitter = SemanticChunker( embeddings=OpenAIEmbeddings(), breakpoint_threshold_type="percentile" )

Markdown Header Splitter

from langchain.text_splitters import MarkdownHeaderTextSplitter

headers_to_split_on = [ ("#", "Header 1"), ("##", "Header 2"), ("###", "Header 3"), ]

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

Vector Store Configurations

Pinecone

import pinecone from langchain.vectorstores import Pinecone

pinecone.init(api_key="your-api-key", environment="us-west1-gcp")

index = pinecone.Index("your-index-name")

vectorstore = Pinecone(index, embeddings.embed_query, "text")

Weaviate

import weaviate from langchain.vectorstores import Weaviate

client = weaviate.Client("http://localhost:8080")

vectorstore = Weaviate(client, "Document", "content", embeddings)

Chroma (Local)

from langchain.vectorstores import Chroma

vectorstore = Chroma( collection_name="my_collection", embedding_function=embeddings, persist_directory="./chroma_db" )

Retrieval Optimization

1. Metadata Filtering

Add metadata during indexing

chunks_with_metadata = [] for i, chunk in enumerate(chunks): chunk.metadata = { "source": chunk.metadata.get("source"), "page": i, "category": determine_category(chunk.page_content) } chunks_with_metadata.append(chunk)

Filter during retrieval

results = vectorstore.similarity_search( "query", filter={"category": "technical"}, k=5 )

2. Maximal Marginal Relevance

Balance relevance with diversity

results = vectorstore.max_marginal_relevance_search( "query", k=5, fetch_k=20, # Fetch 20, return top 5 diverse lambda_mult=0.5 # 0=max diversity, 1=max relevance )

3. Reranking with Cross-Encoder

from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

Get initial results

candidates = vectorstore.similarity_search("query", k=20)

Rerank

pairs = [[query, doc.page_content] for doc in candidates] scores = reranker.predict(pairs)

Sort by score and take top k

reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:5]

Prompt Engineering for RAG

Contextual Prompt

prompt_template = """Use the following context to answer the question. If you cannot answer based on the context, say "I don't have enough information."

Context: {context}

Question: {question}

Answer:"""

With Citations

prompt_template = """Answer the question based on the context below. Include citations using [1], [2], etc.

Context: {context}

Question: {question}

Answer (with citations):"""

With Confidence

prompt_template = """Answer the question using the context. Provide a confidence score (0-100%) for your answer.

Context: {context}

Question: {question}

Answer: Confidence:"""

Evaluation Metrics

def evaluate_rag_system(qa_chain, test_cases): metrics = { 'accuracy': [], 'retrieval_quality': [], 'groundedness': [] }

for test in test_cases:
    result = qa_chain({"query": test['question']})

    # Check if answer matches expected
    accuracy = calculate_accuracy(result['result'], test['expected'])
    metrics['accuracy'].append(accuracy)

    # Check if relevant docs were retrieved
    retrieval_quality = evaluate_retrieved_docs(
        result['source_documents'],
        test['relevant_docs']
    )
    metrics['retrieval_quality'].append(retrieval_quality)

    # Check if answer is grounded in context
    groundedness = check_groundedness(
        result['result'],
        result['source_documents']
    )
    metrics['groundedness'].append(groundedness)

return {k: sum(v)/len(v) for k, v in metrics.items()}

Resources

• references/vector-databases.md: Detailed comparison of vector DBs

• references/embeddings.md: Embedding model selection guide

• references/retrieval-strategies.md: Advanced retrieval techniques

• references/reranking.md: Reranking methods and when to use them

• references/context-window.md: Managing context limits

• assets/vector-store-config.yaml: Configuration templates

• assets/retriever-pipeline.py: Complete RAG pipeline

• assets/embedding-models.md: Model comparison and benchmarks

Best Practices

• Chunk Size: Balance between context and specificity (500-1000 tokens)

• Overlap: Use 10-20% overlap to preserve context at boundaries

• Metadata: Include source, page, timestamp for filtering and debugging

• Hybrid Search: Combine semantic and keyword search for best results

• Reranking: Improve top results with cross-encoder

• Citations: Always return source documents for transparency

• Evaluation: Continuously test retrieval quality and answer accuracy

• Monitoring: Track retrieval metrics in production

Common Issues

• Poor Retrieval: Check embedding quality, chunk size, query formulation

• Irrelevant Results: Add metadata filtering, use hybrid search, rerank

• Missing Information: Ensure documents are properly indexed

• Slow Queries: Optimize vector store, use caching, reduce k

• Hallucinations: Improve grounding prompt, add verification step