Fine-Tuning vs RAG

Should I fine-tune a model or use RAG for domain-specific knowledge?

Two approaches to adding specialized knowledge to LLMs. RAG is usually the right choice for most use cases; fine-tuning has specific advantages for style and format.

intermediate 20 min

Sources verified Dec 22

Approaches

RAG (Retrieval Augmented Generation)

moderate

Store your knowledge in a vector database. At query time, retrieve relevant documents and include them in the prompt context.

In plain English: Like having a researcher who looks up the answer in your documents before responding. They can cite their sources and always have the latest information.

Latency: 1-5s

Cost: Medium - vector DB + more tokens per request

Pros

• Knowledge is easily updateable (just update documents)
• Provides source attribution and citations
• No training costs or wait time
• Works with any model without modification
• Can handle vast knowledge bases (millions of docs)
• Factually grounded in actual documents

Cons

• Higher per-request latency (embed + search + generate)
• More tokens per request = higher API costs
• Quality depends on chunking and retrieval quality
• Requires vector database infrastructure
• Can retrieve irrelevant passages on ambiguous queries

Use When

• Knowledge changes frequently (news, documentation, inventory)
• You need source citations for compliance or trust
• You have a large knowledge base (too big to fit in context)
• You want to avoid training costs and complexity
• Factual accuracy is critical

Avoid When

• You need the model to adopt a specific writing style
• The knowledge is simple enough to fit in a system prompt
• Sub-second latency is critical
• You're optimizing for minimum cost per request

Show Code Example

rag_example.ts

// RAG approach for company policy questions
async function answerPolicyQuestion(question: string) {
  // 1. Find relevant policy documents
  const docs = await vectorDB.search(await embed(question), { topK: 3 });
  
  // 2. Generate answer with retrieved context
  const response = await llm.chat({
    messages: [
      {
        role: 'system',
        content: `Answer based on company policy documents. Cite the source document.

Policy documents:
${docs.map(d => `[${d.title}]: ${d.content}`).join('\n\n')}`
      },
      { role: 'user', content: question }
    ]
  });
  
  return response; // "According to [Vacation Policy], you accrue 2 days per month..."
}

Fine-Tuning

complex

Train a model on your specific data to internalize knowledge, style, or format. The model 'learns' your domain.

In plain English: Like hiring an employee and training them on your company's knowledge and communication style. After training, they 'just know' things, but updating their knowledge requires retraining.

Latency: 500ms-2s (same as base model)

Cost: High upfront (training), lower per-request

Pros

• Lower inference latency (no retrieval step)
• Fewer tokens per request (no context stuffing)
• Learns writing style, tone, and format
• Better at complex reasoning patterns if trained on examples
• Can learn domain-specific terminology and jargon

Cons

• High upfront training cost and complexity
• Knowledge is frozen at training time
• Updates require retraining (days, not seconds)
• No source attribution (model 'just knows')
• Risk of overfitting or catastrophic forgetting
• Less transparent — harder to debug wrong answers

Use When

• You need a specific writing style (legal, medical, brand voice)
• The knowledge is stable and rarely changes
• Latency is critical and you can't afford retrieval
• You have high volume and need to minimize per-request tokens
• You need the model to learn complex reasoning patterns

Avoid When

• Knowledge changes frequently
• You need source citations
• You don't have quality training data (hundreds of examples minimum)
• You need to explain why the model gave a specific answer

Show Code Example

fine_tuning_example.py

# Fine-tuning approach: prepare training data
training_data = [
    {
        "messages": [
            {"role": "system", "content": "You are a legal assistant for Acme Corp."},
            {"role": "user", "content": "What's the vacation policy?"},
            {"role": "assistant", "content": "Employees accrue 2 days of PTO per month..."}
        ]
    },
    # ... hundreds more examples in your style/format
]

# Upload and train (can take hours/days)
client.fine_tuning.jobs.create(
    training_file=file_id,
    model="gpt-4o-mini"
)

# After training, use the fine-tuned model
response = client.chat.completions.create(
    model="ft:gpt-4o-mini:acme-corp:policy-bot",
    messages=[{"role": "user", "content": "What's the vacation policy?"}]
)  # Responds in trained style without needing context

Hybrid (Fine-Tuned + RAG)

complex

Fine-tune for style/format, but still use RAG for factual grounding. Best of both worlds, but most complex.

In plain English: Like a trained employee who also has access to a well-organized filing cabinet. They respond in your style while checking their sources.

Latency: 1-5s

Cost: High (training + vector DB + inference)

Pros

• Consistent style AND accurate facts
• Can cite sources while maintaining brand voice
• Handles both stable and changing knowledge

Cons

• Most complex to build and maintain
• Highest total cost (training + infrastructure + inference)
• Two systems to debug when things go wrong

Use When

• Enterprise applications with strict requirements
• Need both brand consistency and factual accuracy
• Budget and team capacity for complex system

Avoid When

• Building an MVP
• Limited engineering resources
• Either pure approach would suffice

Show Code Example

hybrid_example.ts

// Hybrid: Fine-tuned model + RAG
async function answer(question: string) {
  const docs = await vectorDB.search(await embed(question));
  
  // Use fine-tuned model that knows your style
  return await fineTunedModel.chat({
    messages: [
      { role: 'system', content: `Context from docs:\n${docs.join('\n')}` },
      { role: 'user', content: question }
    ]
  });
}

Decision Factors

Factor	RAG (Retrieval Augmented Generation)	Fine-Tuning	Hybrid (Fine-Tuned + RAG)
Knowledge freshness How often does the information change?	Frequently (daily/weekly) - just update documents	Rarely (yearly) - stable domain knowledge	Mixed - some stable, some changing
Source attribution Do you need to cite sources or explain answers?	Yes - natural source attribution from retrieved docs	No - model 'just knows' without sources	Yes with consistent style
Style/format requirements Does the output need a specific style or format?	Minimal - system prompt can guide style	Critical - model learns from examples	Critical with factual accuracy needs
Latency requirements How fast must responses be?	1-5s acceptable	<1s required	1-5s acceptable
Available training data Do you have high-quality input/output examples?	Not required	Need 100s-1000s of examples	Need training data + document corpus

Real-World Scenarios

Customer support bot for a SaaS product with frequently updated documentation

Recommended: rag

Documentation changes often, and customers expect accurate answers about current features. Source citations build trust.

Legal document generator that must follow a specific firm's writing style

Recommended: fine_tuning

Legal style is stable and consistent. Documents need to match firm's established patterns. No need for citations within generated text.

Medical assistant that needs to cite sources AND maintain clinical tone

Recommended: hybrid

Medical accuracy requires grounding (liability). Clinical tone requires consistent style. Both are critical.

Internal company chatbot answering HR policy questions

Recommended: rag

Policies update periodically. Employees want to see which policy document an answer came from. Simpler than fine-tuning.

Common Misconceptions

Myth: Fine-tuning makes the model 'know' my data like RAG does

Reality: Fine-tuning is better for style/format. For factual recall, RAG is more reliable and updateable.

Myth: RAG is always better because knowledge is updateable

Reality: Fine-tuning wins for consistent style, lower latency, and stable domain knowledge.

Myth: I need thousands of examples to fine-tune

Reality: 50-100 high-quality examples can work for style transfer. More data helps but isn't always necessary.

Sources

✓ Anthropic: RAG grounds responses in retrieved documents, reducing hallucination and enabling source citations ⓘ

×

Anthropic RAG Cookbook

Anthropic

RAG (Retrieval Augmented Generation) grounds LLM responses in retrieved documents, reducing hallucination and enabling source citations. This is the middle ground between simple chat and full agents - add external knowledge without autonomous action.

Key Findings:

• RAG grounds responses in retrieved documents
• Reduces hallucination by providing authoritative context
• Embedding-based retrieval for semantic search

Credibility: high

View Source ↗

✓ OpenAI: Embeddings convert text into numerical vectors for semantic search, enabling RAG retrieval ⓘ

×

OpenAI Embeddings Documentation

OpenAI

Embeddings convert text into numerical vectors that measure semantic relatedness, enabling semantic search, clustering, recommendations, and anomaly detection. OpenAI's text-embedding-3 models are foundational for RAG (Retrieval-Augmented Generation) systems. Understanding embeddings is essential for AI development maturity - it's how AI systems 'understand' and retrieve relevant context.

Key Findings:

• Embeddings measure relatedness of text strings
• text-embedding-3-small and text-embedding-3-large models
• Used for search, clustering, recommendations, anomaly detection

Credibility: high

View Source ↗