Compare Chat vs RAG vs Agent

Compare intermediate 45 min typescript

Sources not yet verified

Evaluate three different AI architectures for a real-world scenario to understand when to use each approach.

1. Understand the Scenario

You're building a customer support system for an e-commerce company. You need to decide which architecture to use: simple chat, RAG, or an agentic system. Each has tradeoffs.

Learning Objectives

• Understand the capabilities and limitations of each approach
• Evaluate approaches against real requirements
• Make an informed architectural decision
• Implement prototypes of each approach

Concepts You'll Practice

Retrieval-Augmented Generation (RAG)Tool Use & Function CallingAgentic LoopsUnderstanding AI Limitations

2. Follow the Instructions

The Requirements

Your customer support system needs to:

1. Answer questions about products and policies
2. Check order status for specific customers
3. Process returns and refunds
4. Escalate complex issues to humans

Let's evaluate each architecture.

Approach 1: Simple Chat

How it works: The model answers based on its training data and a system prompt.

Pros:

• Simplest to implement
• Lowest latency
• No infrastructure needed

Cons:

• Can't access real-time data (order status)
• Will hallucinate about specific policies
• Can't take actions (process refunds)

// Approach 1: Simple Chat
async function simpleChat(userMessage: string) {
  return await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      {
        role: 'system',
        content: 'You are a helpful customer support agent for ShopCo.'
      },
      { role: 'user', content: userMessage }
    ]
  });
}

// ❌ "What's my order status?" - Can't check, will hallucinate
// ❌ "Process my refund" - Can't take action
// ⚠️ "What's your return policy?" - Might be outdated

Approach 2: RAG (Retrieval-Augmented Generation)

How it works: Retrieve relevant documents before generating a response.

Pros:

• Grounded in actual documentation
• Citations for trust
• Updates without retraining

Cons:

• Still can't access real-time data
• Can't take actions
• Retrieval adds latency

// Approach 2: RAG
async function ragChat(userMessage: string) {
  // Step 1: Retrieve relevant docs
  const docs = await vectorStore.search(userMessage, { limit: 5 });
  
  // Step 2: Generate with context
  return await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      {
        role: 'system',
        content: `Answer based on these documents:\n${docs.join('\n')}`
      },
      { role: 'user', content: userMessage }
    ]
  });
}

// ✅ "What's your return policy?" - Retrieves and cites policy
// ❌ "What's my order status?" - Still can't check
// ❌ "Process my refund" - Still can't act

Approach 3: Agent with Tools

How it works: Give the model tools to take actions and access systems.

Pros:

• Can access real-time data
• Can take actions
• Most capable

Cons:

• Most complex to build
• Highest latency (multiple API calls)
• Needs careful guardrails
• Risk of unintended actions

// Approach 3: Agent with Tools
const tools = [
  {
    type: 'function',
    function: {
      name: 'check_order_status',
      description: 'Check status of a customer order',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string' }
        },
        required: ['order_id']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'process_refund',
      description: 'Process a refund for an order',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string' },
          reason: { type: 'string' }
        },
        required: ['order_id', 'reason']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'search_policies',
      description: 'Search company policies',
      parameters: {
        type: 'object',
        properties: {
          query: { type: 'string' }
        },
        required: ['query']
      }
    }
  }
];

// ✅ "What's my order status?" - Calls check_order_status
// ✅ "Process my refund" - Calls process_refund (with guardrails!)
// ✅ "What's your return policy?" - Calls search_policies

⚠️ WARNING: Agent Guardrails Are Critical

Before letting an agent take actions like processing refunds, implement:

1. Human-in-the-loop for high-stakes actions
2. Maximum refund limits
3. Audit logging
4. Rate limiting

Your Task

Implement a hybrid solution that:

1. Uses RAG for policy questions (grounded, fast)
2. Uses tools for order status (real-time data)
3. Requires human approval for refunds (guardrails)
4. Falls back to escalation for complex issues

3. Try It Yourself

starter_code.ts

import OpenAI from 'openai';

const openai = new OpenAI();

// Mock data and functions
const orderDatabase: Record<string, any> = {
  'ORD-123': { status: 'shipped', tracking: 'TRK-456' },
  'ORD-789': { status: 'delivered', tracking: 'TRK-012' }
};

const policyDocs = [
  'Return Policy: Items can be returned within 30 days of delivery.',
  'Refund Policy: Refunds are processed within 5-7 business days.',
  'Shipping: Standard shipping takes 5-7 business days.'
];

// TODO: Implement tool functions
function checkOrderStatus(orderId: string): string {
  throw new Error('Not implemented');
}

function searchPolicies(query: string): string[] {
  throw new Error('Not implemented');
}

function requestRefund(orderId: string, reason: string): {
  approved: boolean;
  message: string;
} {
  // This should require human approval!
  throw new Error('Not implemented');
}

// TODO: Define tools array
const tools: OpenAI.ChatCompletionTool[] = [];

// TODO: Implement the hybrid support agent
async function supportAgent(userMessage: string): Promise<{
  response: string;
  approach_used: 'rag' | 'tool' | 'escalate';
  requires_human_review: boolean;
}> {
  throw new Error('Not implemented');
}

// Test scenarios
const testMessages = [
  "What's the status of order ORD-123?",
  "What's your return policy?",
  "I want a refund for order ORD-789",
  "My package never arrived and I'm very upset!"
];

for (const msg of testMessages) {
  supportAgent(msg).then(result => {
    console.log(`User: ${msg}`);
    console.log(`Agent: ${result.response}`);
    console.log(`Approach: ${result.approach_used}`);
    console.log(`Human review: ${result.requires_human_review}\n`);
  });
}

This typescript exercise requires local setup. Copy the code to your IDE to run.

4. Get Help (If Needed)

Reveal progressive hints

Hint 1: Define four tools: check_order_status, search_policies, request_refund, and escalate_to_human.

Hint 2: For refunds and escalations, always set requires_human_review to true.

Hint 3: Track which tool was called to determine the approach_used: search_policies → 'rag', escalate_to_human → 'escalate', others → 'tool'.

5. Check the Solution

Reveal the complete solution

solution.ts

/**
 * Key Points:
 * - Line ~33: Critical: Never auto-approve financial actions
 * - Line ~85: Escalation tool for complex/emotional issues
 * - Line ~142: Track which approach was used for analytics
 * - Line ~145: Flag when human review is needed
 */
import OpenAI from 'openai';

const openai = new OpenAI();

// Mock data
const orderDatabase: Record<string, any> = {
  'ORD-123': { status: 'shipped', tracking: 'TRK-456' },
  'ORD-789': { status: 'delivered', tracking: 'TRK-012' }
};

const policyDocs = [
  'Return Policy: Items can be returned within 30 days of delivery.',
  'Refund Policy: Refunds are processed within 5-7 business days.',
  'Shipping: Standard shipping takes 5-7 business days.'
];

// Tool implementations
function checkOrderStatus(orderId: string): string {
  const order = orderDatabase[orderId];
  if (!order) {
    return JSON.stringify({ error: 'Order not found' });
  }
  return JSON.stringify(order);
}

function searchPolicies(query: string): string {
  const relevant = policyDocs.filter(doc => 
    doc.toLowerCase().includes(query.toLowerCase())
  );
  return JSON.stringify(relevant.length > 0 ? relevant : policyDocs);
}

function requestRefund(orderId: string, reason: string): string {
  // NEVER auto-approve refunds - require human review
  return JSON.stringify({
    status: 'pending_review',
    message: `Refund request for ${orderId} submitted for human review`,
    ticket_id: `TKT-${Date.now()}`
  });
}

// Tool definitions
const tools: OpenAI.ChatCompletionTool[] = [
  {
    type: 'function',
    function: {
      name: 'check_order_status',
      description: 'Check the status of a customer order',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string', description: 'The order ID like ORD-123' }
        },
        required: ['order_id']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'search_policies',
      description: 'Search company policies about returns, refunds, shipping',
      parameters: {
        type: 'object',
        properties: {
          query: { type: 'string', description: 'Topic to search for' }
        },
        required: ['query']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'request_refund',
      description: 'Submit a refund request (requires human approval)',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string' },
          reason: { type: 'string' }
        },
        required: ['order_id', 'reason']
      }
    }
  },
  {
    type: 'function',
    function: {
      name: 'escalate_to_human',
      description: 'Escalate complex issues to human support',
      parameters: {
        type: 'object',
        properties: {
          summary: { type: 'string' },
          urgency: { type: 'string', enum: ['low', 'medium', 'high'] }
        },
        required: ['summary', 'urgency']
      }
    }
  }
];

function executeTool(name: string, args: any): string {
  switch (name) {
    case 'check_order_status':
      return checkOrderStatus(args.order_id);
    case 'search_policies':
      return searchPolicies(args.query);
    case 'request_refund':
      return requestRefund(args.order_id, args.reason);
    case 'escalate_to_human':
      return JSON.stringify({
        status: 'escalated',
        message: `Issue escalated: ${args.summary}`,
        urgency: args.urgency
      });
    default:
      return JSON.stringify({ error: 'Unknown tool' });
  }
}

async function supportAgent(userMessage: string): Promise<{
  response: string;
  approach_used: 'rag' | 'tool' | 'escalate';
  requires_human_review: boolean;
}> {
  const messages: OpenAI.ChatCompletionMessageParam[] = [
    {
      role: 'system',
      content: `You are a customer support agent for ShopCo.

Guidelines:
1. For policy questions, use search_policies to find accurate information
2. For order status, use check_order_status with the order ID
3. For refund requests, use request_refund - these require human approval
4. For complex/emotional issues, use escalate_to_human
5. Always be empathetic and helpful`
    },
    { role: 'user', content: userMessage }
  ];

  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages,
    tools
  });

  const assistantMessage = response.choices[0].message;
  let approach: 'rag' | 'tool' | 'escalate' = 'tool';
  let requiresHumanReview = false;

  // Handle tool calls
  if (assistantMessage.tool_calls && assistantMessage.tool_calls.length > 0) {
    messages.push(assistantMessage);

    for (const call of assistantMessage.tool_calls) {
      // Skip non-function calls
      if (call.type !== 'function') continue;

      const args = JSON.parse(call.function.arguments);
      const result = executeTool(call.function.name, args);

      // Track approach and human review needs
      if (call.function.name === 'search_policies') {
        approach = 'rag';
      } else if (call.function.name === 'escalate_to_human') {
        approach = 'escalate';
        requiresHumanReview = true;
      } else if (call.function.name === 'request_refund') {
        requiresHumanReview = true;
      }

      messages.push({
        role: 'tool',
        tool_call_id: call.id,
        content: result
      });
    }

    // Get final response with tool results
    const finalResponse = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages,
      tools
    });

    return {
      response: finalResponse.choices[0].message.content || '',
      approach_used: approach,
      requires_human_review: requiresHumanReview
    };
  }

  return {
    response: assistantMessage.content || '',
    approach_used: 'tool',
    requires_human_review: false
  };
}

// Test scenarios
const testMessages = [
  "What's the status of order ORD-123?",
  "What's your return policy?",
  "I want a refund for order ORD-789",
  "My package never arrived and I'm very upset!"
];

for (const msg of testMessages) {
  supportAgent(msg).then(result => {
    console.log(`User: ${msg}`);
    console.log(`Agent: ${result.response}`);
    console.log(`Approach: ${result.approach_used}`);
    console.log(`Human review: ${result.requires_human_review}\n`);
  });
}

/* Expected outputs:

User: What's the status of order ORD-123?
Agent: Your order ORD-123 has been shipped! Tracking number: TRK-456
Approach: tool
Human review: false

User: What's your return policy?
Agent: Items can be returned within 30 days of delivery.
Approach: rag
Human review: false

User: I want a refund for order ORD-789
Agent: I've submitted your refund request for review. A team member will process it within 5-7 business days.
Approach: tool
Human review: true

User: My package never arrived and I'm very upset!
Agent: I'm so sorry to hear about this frustrating experience. I've escalated this to our support team for immediate attention.
Approach: escalate
Human review: true
*/

L33: Critical: Never auto-approve financial actions

L85: Escalation tool for complex/emotional issues

L142: Track which approach was used for analytics

L145: Flag when human review is needed

Common Mistakes

Auto-approving refunds without human review

Why it's wrong: Financial actions should never be fully automated. This creates fraud risk and customer trust issues.

How to fix: Always set requires_human_review to true for refund requests and return a 'pending_review' status.

Using a single approach for all queries

Why it's wrong: Different queries have different needs. Policy questions need grounding, order status needs real-time data, refunds need human approval.

How to fix: Let the model choose the appropriate tool based on the query, and implement all three approaches.

Not handling escalation for complex issues

Why it's wrong: Some issues (angry customers, complex disputes) shouldn't be handled by AI alone.

How to fix: Add an escalate_to_human tool and instruct the model to use it for emotional or complex situations.

Test Cases

Uses tools for order status

Order lookup should use the check_order_status tool

Input: What's the status of order ORD-123?

Expected: Returns shipped status, approach: tool

Uses RAG for policy questions

Policy questions should use search_policies

Input: What's your return policy?

Expected: Returns 30-day policy, approach: rag

Requires human review for refunds

Refund requests must be flagged for human review

Input: I want a refund

Expected: requires_human_review: true

Escalates complex issues

Emotional/complex issues should be escalated

Input: I'm very upset about my order!

Expected: approach: escalate, requires_human_review: true

Sources

✓ Google DORA: Hybrid approaches that combine AI with human oversight build more trust ⓘ

×

State of AI-Assisted Software Development 2025

Google DORA

The 2025 DORA report introduces the 'AI Capabilities Model' identifying seven practices that amplify AI benefits. The core insight is that AI is an 'amplifier' - it magnifies existing organizational strengths AND weaknesses. Key stats: 89% of orgs prioritizing AI, 76% of devs using daily, but 39% have low trust. The trust research is critical: developers who trust AI more are more productive, but trust must be earned through organizational support (policies, training time, addressing concerns). The 451% adoption increase from acceptable-use policies is remarkable - clarity enables adoption.

Key Findings:

• 89% of organizations prioritizing AI integration into applications
• 76% of technologists rely on AI for parts of their daily work
• 75% of developers report positive productivity impact from AI

Credibility: high

View Source ↗