Building RAG from Scratch: 01

Understanding RAG - Architecture and Core Concepts

Series: Building RAG from Scratch
Article: 1 of 12
Level: Beginner
Prerequisites: None
Reading Time: 20-25 minutes

Introduction

Imagine you're taking an exam, but instead of having to memorize everything, you can bring your textbooks and look up information as needed. You still need to understand the questions and formulate good answers, but you have access to accurate, comprehensive information to support your responses.

This is essentially what Retrieval-Augmented Generation (RAG) does for artificial intelligence. It's a powerful technique that combines the natural language understanding and generation capabilities of Large Language Models (LLMs) with the precision and freshness of information retrieval systems.

In this first article of our comprehensive series, we'll build a solid conceptual foundation. We'll explore what RAG is, why it's become so important, how it works at a high level, and when you should (and shouldn't) use it. By the end, you'll understand RAG well enough to explain it to others and make informed decisions about using it in your projects.

What is RAG? The Simple Explanation

RAG stands for Retrieval-Augmented Generation.

Let's break this down:

Retrieval: Finding and fetching relevant information from a knowledge base or document collection
Augmented: Enhanced or supplemented
Generation: Creating responses using a language model

Put together: RAG is a technique where we enhance (augment) an AI's response generation by first retrieving relevant information from external sources.

The Restaurant Analogy

Think of it like this:

Traditional LLM (Without RAG): A chef who memorized recipes during training and can only cook from memory. If you ask for a dish they didn't learn or a recipe that was updated, they'll either say they don't know, or worse, they might make something up based on what they think the recipe should be.

LLM with RAG: The same chef, but now they have access to a constantly updated cookbook. When you ask for a dish, they first look up the current recipe, then use their cooking skills and understanding to prepare it perfectly. They can even cite which page of the cookbook they used.

The chef (LLM) is still doing the cooking (generation), but now they're consulting reliable references (retrieval) to ensure accuracy.

Why RAG Matters: The Problem It Solves

Before we dive into how RAG works, let's understand why it's needed. Large Language Models like GPT-4, Claude, or Llama are incredibly powerful, but they have significant limitations:

1. Knowledge Cutoff Problem

LLMs are trained on data up to a certain date. They have no knowledge of events, discoveries, or information that emerged after their training cutoff.

Example: If a model was trained in early 2024, it wouldn't know about:

A product launch in June 2024
Your company's Q3 financial results
Updates to a software library last month
This morning's news

2. The Hallucination Problem

When LLMs don't know something, they don't reliably say "I don't know." Instead, they might generate plausible-sounding but completely incorrect information. This is called "hallucination."

Example: Ask an LLM about a person it doesn't know:

"Tell me about Dr. Jane Smith's research on quantum computing"
The model might fabricate an entire biography, research papers, and achievements that sound convincing but are entirely fictional.

3. No Source Attribution

Traditional LLMs can't cite their sources. They generate responses from their learned parameters, but they can't tell you "I learned this from source X."

This is problematic when:

You need to verify information
Compliance requires source documentation
Users want to read more about a topic
Legal or medical applications require citations

4. Outdated Information

Information changes rapidly. By the time a model is trained, deployed, and used, some of its knowledge is already outdated.

Example:

A company's policies change monthly
Product specifications are updated
Pricing changes
APIs get new versions

5. Specialized Domain Knowledge

LLMs have broad knowledge but might lack depth in specialized domains, especially for:

Your company's internal documentation
Industry-specific terminology and practices
Proprietary information
Niche technical topics

6. The Cost of Updates

Training or fine-tuning an LLM is extremely expensive, both computationally and financially. You can't just "update" a model every time new information becomes available.

Traditional approach: Retrain the entire model = $$$$ and months of work
RAG approach: Update your knowledge base = $ and minutes of work

How RAG Solves These Problems

RAG elegantly addresses all these limitations by giving the LLM access to external, updatable knowledge:

✅ Fresh Information: Your knowledge base can be updated anytime without retraining
✅ Reduced Hallucinations: The model grounds its responses in retrieved documents
✅ Source Citations: You know exactly which documents informed the response
✅ Specialized Knowledge: Add any domain-specific information you need
✅ Cost Effective: Updating documents is far cheaper than retraining models
✅ Transparency: Users can verify information by checking source documents

RAG vs. Fine-Tuning: When to Use What?

One of the most common questions beginners ask is: "Should I use RAG or fine-tune my model?" Let's break down this crucial decision.

Fine-Tuning: What It Is

Fine-tuning means taking a pre-trained model and training it further on your specific data to adapt its behavior, style, or knowledge.

Think of it as: Teaching a chef a new cooking style or cuisine by having them practice extensively.

RAG: What It Is

RAG gives the model access to external information it can reference when generating responses.

Think of it as: Giving a chef access to cookbooks and recipe databases.

Decision Matrix

Use Case	Best Approach	Why
Learning new facts	RAG	Facts change; RAG is updatable
Changing response style/format	Fine-tuning	Style is about generation patterns
Domain-specific vocabulary	Fine-tuning	Helps model understand terminology
Up-to-date information	RAG	Can update knowledge base anytime
Citing sources	RAG	Sources are explicit in retrieval
Limited budget	RAG	Much cheaper than fine-tuning
Consistent personality/tone	Fine-tuning	Affects core generation behavior
Private/sensitive data	RAG	Data stays in your control

The Best of Both Worlds

You don't have to choose just one! Many production systems use both:

Fine-tune to teach the model your domain's vocabulary and desired response style
Use RAG to provide current, specific information

Example: A legal AI assistant might be:

Fine-tuned on legal writing style and terminology
Use RAG to retrieve specific case law and statutes

RAG Architecture: The Two-Phase System

RAG operates in two distinct phases. Understanding this separation is crucial to building effective RAG systems.

Phase 1: Indexing (Offline Preparation)

This happens before any user queries arrive. It's a one-time setup (though you'll update it as your knowledge base changes).

The Goal: Prepare your documents so they can be quickly searched.

The Process:

[Your Documents] 
    ↓
[Load & Process] → Break into chunks
    ↓
[Generate Embeddings] → Convert text to vectors
    ↓
[Store in Vector Database] → Ready for fast similarity search

Key Points:

This is done once (or when you update documents)
It's computationally intensive but happens offline
The output is a searchable knowledge base

Real-World Analogy: Like organizing a library. Before anyone can find books (retrieve), you need to catalog them, organize them by topic, and create an index system. This work happens once, then everyone benefits.

Phase 2: Query Processing (Online/Runtime)

This happens every time a user asks a question. It must be fast.

The Goal: Find relevant information and generate a response.

The Process:

[User Question]
    ↓
[Convert to Embedding] → Same format as stored documents
    ↓
[Search Vector Database] → Find most similar chunks
    ↓
[Retrieve Top K Results] → Get the best matches
    ↓
[Combine: Question + Retrieved Context]
    ↓
[Send to LLM]
    ↓
[Generate Response] → Informed by retrieved documents
    ↓
[Return to User] → With optional source citations

Key Points:

Happens in real-time for each query
Must be optimized for speed
Quality depends on both retrieval and generation

Real-World Analogy: A library patron asks a question → Librarian searches the catalog → Finds relevant books → Brings them to the patron → Patron reads and forms an answer. All of this needs to happen quickly.

Core Components: Deep Dive

Let's explore each component of a RAG system in detail.

1. Data Sources and Knowledge Base

What they are: The documents, databases, or information that your RAG system can access.

Types of sources:

Text documents: PDFs, Word docs, text files, markdown
Web content: Articles, documentation, wikis
Structured data: Databases, spreadsheets, CSV files
Code repositories: Source code, API documentation
Communication: Emails, Slack messages, meeting notes

Key consideration: Your RAG system is only as good as your data. Garbage in = garbage out.

Best practices:

Keep documents focused and well-organized
Include metadata (author, date, version, category)
Remove irrelevant or outdated information
Ensure accuracy and authority of sources

2. Document Processing and Chunking

What it is: Breaking large documents into smaller, manageable pieces.

Why it's necessary:

LLMs have context length limits (can't process entire books)
Smaller chunks enable more precise retrieval
Better matching between queries and relevant sections

The challenge: Finding the right chunk size

Too small (100 words):

✅ Very precise matching
❌ May lack context
❌ "The company was founded in" (cut off mid-sentence)

Too large (2000 words):

✅ Lots of context
❌ Less precise matching
❌ Irrelevant information mixed with relevant

Just right (500-800 words):

✅ Balanced precision and context
✅ Usually contains complete ideas
✅ Manageable for LLMs

Pro tip: Add overlap between chunks (10-20%) so information at boundaries doesn't get lost.

3. Embedding Models

What they are: AI models that convert text into numerical vectors (arrays of numbers).

Why they're needed: Computers can't directly compare the "meaning" of text, but they can compare numbers.

The magic: Similar meanings = similar vectors

Example:

"The cat sat on the mat" → [0.2, 0.8, 0.1, 0.4, ...]
"A feline rested on the rug" → [0.19, 0.79, 0.12, 0.41, ...]
"Python programming tutorial" → [0.7, 0.1, 0.9, 0.2, ...]

The first two are similar (close vectors), while the third is different (distant vector).

How it works conceptually:

The model was trained on massive amounts of text
It learned that certain words and phrases have similar meanings
It encodes this semantic understanding into vector space
Words/phrases with similar meanings cluster together in this space

Common embedding models:

OpenAI: text-embedding-3-small, text-embedding-3-large
Open-source: sentence-transformers, BGE, E5
Specialized: Models trained for specific domains (medical, legal, code)

Key insight: The same embedding model must be used for both indexing and querying, or the vectors won't be comparable.

4. Vector Databases

What they are: Specialized databases optimized for storing and searching vectors.

Why regular databases won't work:

Regular databases search for exact matches: "Find WHERE name = 'John'"
Vector databases search for similarity: "Find vectors closest to this query vector"

How they work:

Store millions of vectors efficiently
Use special algorithms (like HNSW, IVF) for fast approximate search
Can find nearest neighbors in milliseconds

Popular options:

Pinecone: Managed cloud service, easy to use
Weaviate: Open-source, feature-rich
ChromaDB: Great for local development
FAISS: Facebook's library, extremely fast
Qdrant: High performance, production-ready

What they provide:

Fast similarity search: Find top K most similar vectors
Filtering: Search within specific categories or metadata
Scaling: Handle millions to billions of vectors
Hybrid search: Combine vector and keyword search

5. Retrieval Mechanisms

What they are: The strategies and algorithms for finding relevant information.

Basic retrieval (what we start with):

Convert user query to embedding
Search vector database for similar embeddings
Return top K most similar chunks (usually 3-10)

Similarity metrics:

Cosine similarity: Measures angle between vectors (most common)
Euclidean distance: Straight-line distance between vectors
Dot product: Multiplication-based similarity

Beyond basic retrieval (covered in later articles):

Hybrid search: Combine semantic and keyword search
Re-ranking: Use a second model to re-order results
Metadata filtering: "Only search documents from 2024"
MMR (Maximal Marginal Relevance): Ensure diverse results

6. Large Language Model (LLM)

What it is: The AI model that generates the final response.

Its job in RAG:

Read the user's question
Read the retrieved context
Synthesize a coherent, accurate response
(Optionally) Cite sources

Popular LLMs for RAG:

OpenAI: GPT-4, GPT-3.5
Anthropic: Claude 3.5 Sonnet, Claude 3 Opus
Open-source: Llama 3, Mistral, Mixtral
Local: Via Ollama or LM Studio

Key considerations:

Context window: How much text can it process at once?
Cost: API calls add up quickly
Speed: Response time matters for user experience
Quality: Better models = better synthesis

The LLM's challenge in RAG: Not just generating text, but:

Understanding which retrieved information is relevant
Ignoring irrelevant parts of retrieved chunks
Combining information from multiple sources
Maintaining factual accuracy
Admitting when retrieved context doesn't contain the answer

Information Flow: Tracing a Query Through RAG

Let's follow a real example from start to finish to see how all the pieces work together.

Example Scenario

Knowledge Base: Your company's employee handbook (50 pages)
User Question: "What is the vacation policy for employees with 3 years of service?"

Step-by-Step Process

INDEXING PHASE (Done Once, Beforehand):

Load Document
- Employee handbook loaded from PDF
- 50 pages of text extracted
Chunk Document
- Broken into ~100 chunks of ~500 words each
- Chunk #42 contains: "Employees with 2-4 years of service are entitled to 15 vacation days per year..."
- Chunk #43 contains: "Vacation days must be requested at least 2 weeks in advance..."
Generate Embeddings
- Each chunk converted to a 1536-dimension vector
- Chunk #42 → [0.23, -0.45, 0.67, ...]
- Chunk #43 → [0.19, -0.42, 0.71, ...]
Store in Vector DB
- All 100 chunk embeddings stored
- Metadata added: {"source": "handbook.pdf", "page": 23, "section": "Time Off"}

QUERY PHASE (Happens in Real-Time):

User Asks Question
- Query: "What is the vacation policy for employees with 3 years of service?"
Embed Query
- Convert question to embedding: [0.21, -0.44, 0.69, ...]
- (Notice it's similar to chunk #42's embedding!)
Search Vector Database
- Find chunks with most similar embeddings
- Chunk #42: similarity score 0.89 (high!)
- Chunk #43: similarity score 0.76 (relevant too)
- Chunk #67: similarity score 0.45 (not very relevant)
Retrieve Top K
- Select top 3 chunks: #42, #43, #67
Build Prompt

   Context:
   [Chunk #42 text: "Employees with 2-4 years of service are entitled to 15 vacation days per year..."]
   [Chunk #43 text: "Vacation days must be requested at least 2 weeks in advance..."]
   [Chunk #67 text: "Sick leave policy..."]

   Question: What is the vacation policy for employees with 3 years of service?

   Instructions: Answer the question based on the provided context. 
   If the context doesn't contain the answer, say so.

LLM Generates Response

    Based on the employee handbook, employees with 3 years of service 
    fall into the 2-4 year service bracket and are entitled to 15 vacation 
    days per year. Vacation requests should be submitted at least 2 weeks 
    in advance.

    Source: Employee Handbook, page 23, Time Off section

Return to User
- User receives answer with citation
- Can click through to verify in original document

What Made This Work?

Good chunking: Vacation policy was in a cohesive chunk
Quality embeddings: Query matched relevant chunks semantically
Sufficient context: Retrieved multiple related chunks
Good prompt: Clear instructions to LLM
Source tracking: Metadata allowed citation

When to Use RAG: A Decision Framework

Now that you understand RAG, let's determine when it's the right solution.

✅ Use RAG When:

You need access to specific, current information
- Company knowledge bases
- Product documentation
- Recent research papers
- Updated policies and procedures
Information changes frequently
- News and current events
- Stock prices and market data
- Software documentation
- Inventory and availability
You need source attribution
- Legal and compliance applications
- Medical advice (with disclaimers)
- Research and academic use
- Fact-checking and verification
You have domain-specific knowledge
- Internal company information
- Specialized industry knowledge
- Proprietary data
- Custom terminology
You want to reduce hallucinations
- Critical applications where accuracy matters
- Customer-facing systems
- Decision support tools
Budget is a constraint
- Can't afford model fine-tuning
- Need cost-effective updates
- Prefer operational expenses over capital expenses

❌ Don't Use RAG When:

The model already knows the information
- General knowledge questions
- Common sense reasoning
- Basic math or logic
- Creative writing without specific facts
You need to change the model's behavior or style
- Writing in a specific voice
- Following complex formatting rules
- Adopting a persona
- (Use fine-tuning instead)
Real-time response is critical and retrieval adds too much latency
- Some high-frequency trading
- Emergency response systems
- (Though optimized RAG can be quite fast)
Your knowledge base is too small
- Less than ~50 documents
- Better to put it all in the prompt
- RAG overhead isn't worth it
Information is highly unstructured or abstract
- Pure creative tasks
- Abstract reasoning
- Philosophical discussions

Hybrid Approaches

Often, the best solution combines multiple techniques:

RAG + Prompt Engineering: Use RAG for facts, but craft prompts to control style and format

RAG + Fine-Tuning: Fine-tune for domain vocabulary and style, use RAG for current facts

RAG + Function Calling: Use RAG for documents, function calling for structured data (databases, APIs)

Common RAG Use Cases

Let's look at real-world applications to make RAG concrete.

1. Customer Support Chatbot

Knowledge Base:

Product manuals
FAQ documents
Troubleshooting guides
Knowledge base articles

Example Interaction:

User: "My device won't turn on"
RAG retrieves: Troubleshooting section about power issues
Response: "Let's check a few things. First, is the device charged? The manual indicates..."

Benefits:

Always up-to-date with latest documentation
Cites specific manual sections
Reduces support ticket volume

2. Legal Research Assistant

Knowledge Base:

Case law database
Statutes and regulations
Legal precedents
Firm's case history

Example Interaction:

Lawyer: "What's the precedent for contract disputes in California involving force majeure?"
RAG retrieves: Relevant cases and statutes
Response: "California courts have ruled in several cases. In Smith v. Jones (2019)..."

Benefits:

Searches thousands of cases instantly
Provides citations
Ensures current law is referenced

3. Internal Company Knowledge Base

Knowledge Base:

Company wiki
Project documentation
Meeting notes
Slack conversations
Email archives

Example Interaction:

Employee: "What was decided about the Q4 marketing strategy?"
RAG retrieves: Meeting notes from strategy session
Response: "In the October 15th leadership meeting, the team decided to focus on..."

Benefits:

Institutional knowledge accessible to all
Reduces repeated questions
Preserves decision history

4. Academic Research Assistant

Knowledge Base:

PubMed articles
ArXiv papers
University library
Course materials

Example Interaction:

Researcher: "What methods have been used to detect protein-protein interactions?"
RAG retrieves: Multiple research papers
Response: "Several approaches have been documented. Mass spectrometry (Chen et al., 2020)..."

Benefits:

Searches vast literature quickly
Synthesizes multiple sources
Provides citations for verification

5. Healthcare Information System

Knowledge Base:

Medical guidelines (CDC, WHO)
Drug interaction databases
Clinical protocols
Research studies

Example Interaction:

Doctor: "What are the contraindications for this medication in elderly patients?"
RAG retrieves: Drug information and studies
Response: "According to the prescribing information, caution should be exercised in patients over 65..."

Benefits:

Access to current medical guidelines
Reduces medical errors
Evidence-based recommendations

RAG Limitations and Challenges

RAG isn't perfect. Understanding its limitations helps you build better systems and set appropriate expectations.

1. Retrieval Quality Issues

The Problem: If retrieval fails, everything fails.

Wrong chunks retrieved → Wrong answer
Relevant chunks not retrieved → Incomplete answer
Too many irrelevant chunks → Confused answer

Example: Question: "What's our return policy?" Retrieved: Shipping policy (wrong section) Result: LLM generates answer about shipping, not returns

Mitigation: Test retrieval quality, use multiple retrieval strategies, implement re-ranking

2. Context Window Limitations

The Problem: Can only send limited text to LLM.

Modern LLMs have context limits (32k, 128k, 200k tokens)
Need to balance between retrieving enough context and staying within limits
More context = higher cost and slower response

Mitigation: Compress context, use better chunking, implement progressive retrieval

3. Latency

The Problem: RAG adds multiple steps, each taking time.

Embed query: ~100ms
Search database: ~50-200ms
LLM generation: ~2-10 seconds
Total: Noticeably slower than just calling LLM

Mitigation: Optimize each step, cache common queries, use faster models

4. Chunk Boundary Problems

The Problem: Important information split across chunks.

"The company was founded in [CHUNK END] [NEW CHUNK] 1995 in San Francisco"
Retrieval might get only one chunk
Context is lost

Mitigation: Use overlap between chunks, implement parent-child retrieval

5. Cost Considerations

The Problem: Every query costs money.

Embedding API calls
Vector database queries
LLM API calls
At scale, costs add up

Typical costs per query:

Embedding: $0.0001
Vector DB: $0.0001-0.001
LLM: $0.001-0.05 (varies by model)

Mitigation: Cache results, use cheaper models for simple queries, optimize token usage

6. "Lost in the Middle" Problem

The Problem: LLMs pay more attention to beginning and end of context.

Research shows middle context is often ignored
If the answer is in the middle chunk, LLM might miss it

Mitigation: Re-order chunks by relevance, use summarization, implement re-ranking

7. Requires Quality Data

The Problem: RAG is only as good as your knowledge base.

Outdated information → outdated answers
Incorrect documents → incorrect answers
Poorly organized data → poor retrieval

Mitigation: Implement data quality processes, regular updates, versioning

Key Takeaways

Let's summarize the essential points from this article:

What RAG Is

A technique that enhances LLM responses by retrieving relevant information from external sources
Combines information retrieval with generation
Operates in two phases: indexing (offline) and query processing (online)

Why RAG Matters

Solves the knowledge cutoff problem
Reduces hallucinations through grounding
Enables source attribution
Cost-effective way to keep information current
Provides access to specialized knowledge

Core Components

Data sources and knowledge base
Document chunking and processing
Embedding models (text → vectors)
Vector databases (storage and similarity search)
Retrieval mechanisms (finding relevant chunks)
LLMs (generating final responses)

When to Use RAG

Need current, specific information
Information changes frequently
Need source attribution
Have domain-specific knowledge
Want to reduce hallucinations
Budget-conscious solution

Key Challenges

Retrieval quality
Context window limits
Added latency
Chunk boundary issues
Cost at scale
Data quality requirements

What's Next?

Now that you understand RAG conceptually, you're ready to build one! In the next article, we'll:

Set up your development environment
Install necessary libraries and tools
Build your first simple RAG system
Actually run queries and see it work
Understand the code behind each component

Article 2: Setting Up Your RAG Development Environment

You'll write your first RAG code and see the concepts from this article come to life. See you there!

Review Questions

Test your understanding:

Explain RAG to someone who's never heard of it. Can you do it in 2-3 sentences?
What's the difference between RAG and fine-tuning? When would you use each?
Name the two main phases of RAG and explain what happens in each.
Why do we need embedding models? What problem do they solve?
Give three examples of when RAG would be the right solution.
What happens if retrieval fails in a RAG system?
Explain what "chunks" are and why we need them.

If you can answer these confidently, you're ready for Article 2!

Next Article: Setting Up Your RAG Development Environment
Coming Soon: Where we'll install tools, write code, and build your first working RAG system!

This is Article 1 of the "Building RAG from Scratch" series. Follow along as we progress from concepts to production-ready systems in 12 comprehensive articles.

Building RAG from Scratch: 01

Understanding RAG - Architecture and Core Concepts

Introduction

What is RAG? The Simple Explanation

The Restaurant Analogy

Why RAG Matters: The Problem It Solves

1. Knowledge Cutoff Problem

2. The Hallucination Problem

3. No Source Attribution

4. Outdated Information

5. Specialized Domain Knowledge

6. The Cost of Updates

How RAG Solves These Problems

RAG vs. Fine-Tuning: When to Use What?

Fine-Tuning: What It Is

RAG: What It Is

Decision Matrix

The Best of Both Worlds

RAG Architecture: The Two-Phase System

Phase 1: Indexing (Offline Preparation)

Phase 2: Query Processing (Online/Runtime)

Core Components: Deep Dive

1. Data Sources and Knowledge Base

2. Document Processing and Chunking

3. Embedding Models

4. Vector Databases

5. Retrieval Mechanisms

6. Large Language Model (LLM)

Information Flow: Tracing a Query Through RAG

Example Scenario

Step-by-Step Process

What Made This Work?

When to Use RAG: A Decision Framework

✅ Use RAG When:

❌ Don't Use RAG When:

Hybrid Approaches

Common RAG Use Cases

1. Customer Support Chatbot

2. Legal Research Assistant

3. Internal Company Knowledge Base

4. Academic Research Assistant

5. Healthcare Information System

RAG Limitations and Challenges

1. Retrieval Quality Issues

2. Context Window Limitations

3. Latency

4. Chunk Boundary Problems

5. Cost Considerations

6. "Lost in the Middle" Problem

7. Requires Quality Data

Key Takeaways

What RAG Is

Why RAG Matters

Core Components

When to Use RAG

Key Challenges

What's Next?

Further Reading

Review Questions

vector db

Related Topics

Building RAG from Scratch: 01