Compare Chat vs RAG vs Agent

Compare intermediate 45 min typescript

Sources verified Dec 22, 2025

Evaluate three different AI architectures for a real-world scenario to understand when to use each approach.

1. Understand the Scenario

You're building a customer support system for an e-commerce company. You need to decide which architecture to use: simple chat, RAG, or an agentic system. Each has tradeoffs.

Learning Objectives

• Understand the capabilities and limitations of each approach
• Evaluate approaches against real requirements
• Make an informed architectural decision
• Implement prototypes of each approach

Concepts You'll Practice

Retrieval-Augmented Generation (RAG)Tool Use & Function CallingAgentic LoopsUnderstanding AI Limitations

2. Follow the Instructions

The Requirements

Your customer support system needs to:

1. Answer questions about products and policies
2. Check order status for specific customers
3. Process returns and refunds
4. Escalate complex issues to humans

Let's evaluate each architecture.

Approach 1: Simple Chat

How it works: The model answers based on its training data and a system prompt.

Pros:

• Simplest to implement
• Lowest latency
• No infrastructure needed

Cons:

• Can't access real-time data (order status)
• Will hallucinate about specific policies
• Can't take actions (process refunds)

approach1_chat.ts

// Approach 1: Simple Chat
async function simpleChat(userMessage: string) {
  return await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      {
        role: 'system',
        content: 'You are a helpful customer support agent for ShopCo.'
      },
      { role: 'user', content: userMessage }
    ]
  });
}

// ❌ "What's my order status?" - Can't check, will hallucinate
// ❌ "Process my refund" - Can't take action
// ⚠️ "What's your return policy?" - Might be outdated

Approach 2: RAG (Retrieval-Augmented Generation)

How it works: Retrieve relevant documents before generating a response.

Pros:

• Grounded in actual documentation
• Citations for trust
• Updates without retraining

Cons:

• Still can't access real-time data
• Can't take actions
• Retrieval adds latency

approach2_rag.ts

// Approach 2: RAG
async function ragChat(userMessage: string) {
  // Step 1: Retrieve relevant docs
  const docs = await vectorStore.search(userMessage, { limit: 5 });
  
  // Step 2: Generate with context
  return await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      {
        role: 'system',
        content: `Answer based on these documents:\n${docs.join('\n')}`
      },
      { role: 'user', content: userMessage }
    ]
  });
}

// ✅ "What's your return policy?" - Retrieves and cites policy
// ❌ "What's my order status?" - Still can't check
// ❌ "Process my refund" - Still can't act

Approach 3: Agent with Tools

How it works: Give the model tools to take actions and access systems.

Pros:

• Can access real-time data
• Can take actions
• Most capable

Cons:

• Most complex to build
• Highest latency (multiple API calls)
• Needs careful guardrails
• Risk of unintended actions

approach3_agent.ts

// Approach 3: Agent with Tools
const tools = [
  {
    type: 'function',
    function: {
      name: 'check_order_status',
      description: 'Check status of a customer order',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string' }
        },
        required: ['order_id']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'process_refund',
      description: 'Process a refund for an order',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string' },
          reason: { type: 'string' }
        },
        required: ['order_id', 'reason']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'search_policies',
      description: 'Search company policies',
      parameters: {
        type: 'object',
        properties: {
          query: { type: 'string' }
        },
        required: ['query']
      }
    }
  }
];

// ✅ "What's my order status?" - Calls check_order_status
// ✅ "Process my refund" - Calls process_refund (with guardrails!)
// ✅ "What's your return policy?" - Calls search_policies

Agent Guardrails Are Critical

Before letting an agent take actions like processing refunds, implement:

1. Human-in-the-loop for high-stakes actions
2. Maximum refund limits
3. Audit logging
4. Rate limiting

Your Task

Implement a hybrid solution that:

1. Uses RAG for policy questions (grounded, fast)
2. Uses tools for order status (real-time data)
3. Requires human approval for refunds (guardrails)
4. Falls back to escalation for complex issues

3. Try It Yourself

exercise_starter.ts

import OpenAI from 'openai';

const openai = new OpenAI();

// Mock data and functions
const orderDatabase: Record<string, any> = {
  'ORD-123': { status: 'shipped', tracking: 'TRK-456' },
  'ORD-789': { status: 'delivered', tracking: 'TRK-012' }
};

const policyDocs = [
  'Return Policy: Items can be returned within 30 days of delivery.',
  'Refund Policy: Refunds are processed within 5-7 business days.',
  'Shipping: Standard shipping takes 5-7 business days.'
];

// TODO: Implement tool functions
function checkOrderStatus(orderId: string): string {
  throw new Error('Not implemented');
}

function searchPolicies(query: string): string[] {
  throw new Error('Not implemented');
}

function requestRefund(orderId: string, reason: string): {
  approved: boolean;
  message: string;
} {
  // This should require human approval!
  throw new Error('Not implemented');
}

// TODO: Define tools array
const tools: OpenAI.ChatCompletionTool[] = [];

// TODO: Implement the hybrid support agent
async function supportAgent(userMessage: string): Promise<{
  response: string;
  approach_used: 'rag' | 'tool' | 'escalate';
  requires_human_review: boolean;
}> {
  throw new Error('Not implemented');
}

// Test scenarios
const testMessages = [
  "What's the status of order ORD-123?",
  "What's your return policy?",
  "I want a refund for order ORD-789",
  "My package never arrived and I'm very upset!"
];

for (const msg of testMessages) {
  supportAgent(msg).then(result => {
    console.log(`User: ${msg}`);
    console.log(`Agent: ${result.response}`);
    console.log(`Approach: ${result.approach_used}`);
    console.log(`Human review: ${result.requires_human_review}\n`);
  });
}

This typescript exercise requires local setup. Copy the code to your IDE to run.

4. Get Help (If Needed)

Reveal progressive hints

Hint 1: Define four tools: check_order_status, search_policies, request_refund, and escalate_to_human.

Hint 2: For refunds and escalations, always set requires_human_review to true.

Hint 3: Track which tool was called to determine the approach_used: search_policies → 'rag', escalate_to_human → 'escalate', others → 'tool'.

5. Check the Solution

Reveal the complete solution

exercise_solution.ts

import OpenAI from 'openai';

const openai = new OpenAI();

// Mock data
const orderDatabase: Record<string, any> = {
  'ORD-123': { status: 'shipped', tracking: 'TRK-456' },
  'ORD-789': { status: 'delivered', tracking: 'TRK-012' }
};

const policyDocs = [
  'Return Policy: Items can be returned within 30 days of delivery.',
  'Refund Policy: Refunds are processed within 5-7 business days.',
  'Shipping: Standard shipping takes 5-7 business days.'
];

// Tool implementations
function checkOrderStatus(orderId: string): string {
  const order = orderDatabase[orderId];
  if (!order) {
    return JSON.stringify({ error: 'Order not found' });
  }
  return JSON.stringify(order);
}

function searchPolicies(query: string): string {
  const relevant = policyDocs.filter(doc => 
    doc.toLowerCase().includes(query.toLowerCase())
  );
  return JSON.stringify(relevant.length > 0 ? relevant : policyDocs);
}

function requestRefund(orderId: string, reason: string): string {
  // NEVER auto-approve refunds - require human review
  return JSON.stringify({
    status: 'pending_review',
    message: `Refund request for ${orderId} submitted for human review`,
    ticket_id: `TKT-${Date.now()}`
  });
}

// Tool definitions
const tools: OpenAI.ChatCompletionTool[] = [
  {
    type: 'function',
    function: {
      name: 'check_order_status',
      description: 'Check the status of a customer order',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string', description: 'The order ID like ORD-123' }
        },
        required: ['order_id']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'search_policies',
      description: 'Search company policies about returns, refunds, shipping',
      parameters: {
        type: 'object',
        properties: {
          query: { type: 'string', description: 'Topic to search for' }
        },
        required: ['query']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'request_refund',
      description: 'Submit a refund request (requires human approval)',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string' },
          reason: { type: 'string' }
        },
        required: ['order_id', 'reason']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'escalate_to_human',
      description: 'Escalate complex issues to human support',
      parameters: {
        type: 'object',
        properties: {
          summary: { type: 'string' },
          urgency: { type: 'string', enum: ['low', 'medium', 'high'] }
        },
        required: ['summary', 'urgency']
      }
    }
  }
];

function executeTool(name: string, args: any): string {
  switch (name) {
    case 'check_order_status':
      return checkOrderStatus(args.order_id);
    case 'search_policies':
      return searchPolicies(args.query);
    case 'request_refund':
      return requestRefund(args.order_id, args.reason);
    case 'escalate_to_human':
      return JSON.stringify({
        status: 'escalated',
        message: `Issue escalated: ${args.summary}`,
        urgency: args.urgency
      });
    default:
      return JSON.stringify({ error: 'Unknown tool' });
  }
}

async function supportAgent(userMessage: string): Promise<{
  response: string;
  approach_used: 'rag' | 'tool' | 'escalate';
  requires_human_review: boolean;
}> {
  const messages: OpenAI.ChatCompletionMessageParam[] = [
    {
      role: 'system',
      content: `You are a customer support agent for ShopCo.

Guidelines:
1. For policy questions, use search_policies to find accurate information
2. For order status, use check_order_status with the order ID
3. For refund requests, use request_refund - these require human approval
4. For complex/emotional issues, use escalate_to_human
5. Always be empathetic and helpful`
    },
    { role: 'user', content: userMessage }
  ];

  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages,
    tools
  });

  const assistantMessage = response.choices[0].message;
  let approach: 'rag' | 'tool' | 'escalate' = 'tool';
  let requiresHumanReview = false;

  // Handle tool calls
  if (assistantMessage.tool_calls && assistantMessage.tool_calls.length > 0) {
    messages.push(assistantMessage);

    for (const call of assistantMessage.tool_calls) {
      const args = JSON.parse(call.function.arguments);
      const result = executeTool(call.function.name, args);

      // Track approach and human review needs
      if (call.function.name === 'search_policies') {
        approach = 'rag';
      } else if (call.function.name === 'escalate_to_human') {
        approach = 'escalate';
        requiresHumanReview = true;
      } else if (call.function.name === 'request_refund') {
        requiresHumanReview = true;
      }

      messages.push({
        role: 'tool',
        tool_call_id: call.id,
        content: result
      });
    }

    // Get final response with tool results
    const finalResponse = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages,
      tools
    });

    return {
      response: finalResponse.choices[0].message.content || '',
      approach_used: approach,
      requires_human_review: requiresHumanReview
    };
  }

  return {
    response: assistantMessage.content || '',
    approach_used: 'tool',
    requires_human_review: false
  };
}

// Test scenarios
const testMessages = [
  "What's the status of order ORD-123?",
  "What's your return policy?",
  "I want a refund for order ORD-789",
  "My package never arrived and I'm very upset!"
];

for (const msg of testMessages) {
  supportAgent(msg).then(result => {
    console.log(`User: ${msg}`);
    console.log(`Agent: ${result.response}`);
    console.log(`Approach: ${result.approach_used}`);
    console.log(`Human review: ${result.requires_human_review}\n`);
  });
}

/* Expected outputs:

User: What's the status of order ORD-123?
Agent: Your order ORD-123 has been shipped! Tracking number: TRK-456
Approach: tool
Human review: false

User: What's your return policy?
Agent: Items can be returned within 30 days of delivery.
Approach: rag
Human review: false

User: I want a refund for order ORD-789
Agent: I've submitted your refund request for review. A team member will process it within 5-7 business days.
Approach: tool
Human review: true

User: My package never arrived and I'm very upset!
Agent: I'm so sorry to hear about this frustrating experience. I've escalated this to our support team for immediate attention.
Approach: escalate
Human review: true
*/

L33: Critical: Never auto-approve financial actions

L85: Escalation tool for complex/emotional issues

L142: Track which approach was used for analytics

L145: Flag when human review is needed

Common Mistakes

Auto-approving refunds without human review

Why it's wrong: Financial actions should never be fully automated. This creates fraud risk and customer trust issues.

How to fix: Always set requires_human_review to true for refund requests and return a 'pending_review' status.

Using a single approach for all queries

Why it's wrong: Different queries have different needs. Policy questions need grounding, order status needs real-time data, refunds need human approval.

How to fix: Let the model choose the appropriate tool based on the query, and implement all three approaches.

Not handling escalation for complex issues

Why it's wrong: Some issues (angry customers, complex disputes) shouldn't be handled by AI alone.

How to fix: Add an escalate_to_human tool and instruct the model to use it for emotional or complex situations.

Test Cases

Uses tools for order status

Order lookup should use the check_order_status tool

Input: What's the status of order ORD-123?

Expected: Returns shipped status, approach: tool

Uses RAG for policy questions

Policy questions should use search_policies

Input: What's your return policy?

Expected: Returns 30-day policy, approach: rag

Requires human review for refunds

Refund requests must be flagged for human review

Input: I want a refund

Expected: requires_human_review: true

Escalates complex issues

Emotional/complex issues should be escalated

Input: I'm very upset about my order!

Expected: approach: escalate, requires_human_review: true

Sources

✓ Anthropic: RAG grounds responses in retrieved documents, reducing hallucination and enabling source citations ⓘ

×

Anthropic RAG Cookbook

Anthropic

RAG (Retrieval Augmented Generation) grounds LLM responses in retrieved documents, reducing hallucination and enabling source citations. This is the middle ground between simple chat and full agents - add external knowledge without autonomous action.

Key Findings:

• Contextual retrieval: prepend chunk with document context before embedding (67% recall improvement)
• Optimal chunk size: 256-512 tokens balances context completeness vs retrieval precision
• Hybrid search: combine BM25 keyword matching with semantic embeddings for best recall

Credibility: high

Referenced by:

Compare Chat vs RAG vs Agent

View Source ↗

✓ OWASP GenAI Security Project: Agent approaches require guardrails - human-in-the-loop for high-stakes actions ⓘ

×

OWASP Top 10 for Agentic Applications 2026

OWASP GenAI Security Project

The first OWASP security framework specifically for agentic AI systems. The 'Least Agency' principle is critical for our agentic_supervision dimension. Key risks (goal hijacking, tool misuse, rogue agents) directly inform supervision recommendations. Released December 2025, this is the authoritative security guide for AI coding agents.

Key Findings:

• ASI01 - Agent Goal Hijack: Prompt injection manipulates agent goals
• ASI02 - Tool Misuse: Agents misuse legitimate tools for data exfiltration
• ASI03 - Identity & Privilege Abuse: Confused deputy and privilege escalation

Credibility: high

Referenced by:

Guardrails Quick Reference: AI Code Checklists Build a Read-Only MCP Server Compare Chat vs RAG vs Agent

View Source ↗

✓ The New Stack: Teams with heavy AI agent use saw PR review times increase 91%, creating new review bottlenecks ⓘ

×

Is AI Creating a New Code Review Bottleneck for Senior Engineers?

The New Stack

Documents the emerging 'AI Productivity Paradox': AI increases output but creates review bottlenecks. The 91% increase in PR review times despite 21% more tasks completed shows the shifting bottleneck problem. Critical for organizational integration dimension.

Key Findings:

• Teams with heavy AI use completed 21% more tasks but PR review times increased 91%
• 67% of developers spend more time debugging AI-generated code
• 68% note increased time spent on code reviews

Credibility: medium

Referenced by:

AI System Monitoring Compare Chat vs RAG vs Agent

View Source ↗