Catching a Hallucination: Trust but Verify

beginner 8 min 📋 Copy-paste ready

Sources not yet verified

hallucinationverificationtrustbeginner

Scenario

Context: You're using AI to help research an unfamiliar API or library

Goal: Learn to recognize and verify AI claims that might be hallucinated

Anti-pattern: Copying code or information from AI without verification

Tools: ChatGPTClaudeGitHub Copilot

Conversation

Red Flags for Hallucination

AI can confidently generate plausible-sounding but incorrect information, especially for:

Specific API methods or parameters
Version-specific features
URLs and documentation links
Statistics and citations

👤 You

How do I use the Node.js fs module to watch a directory for changes?

A reasonable question about a real API.

🤖 AI ✗ Bad example

You can use fs.watchDirectory() with a callback:

const fs = require('fs');

fs.watchDirectory('./my-folder', { recursive: true }, (event, filename) => {
  console.log(`${event}: ${filename}`);
});

The recursive: true option was added in Node.js 18 and watches all subdirectories.

This looks plausible but contains errors. The actual method is fs.watch(), not fs.watchDirectory(). Always verify specific API names.

Verification Strategies

👤 You

Let me verify this. Can you show me the Node.js documentation link for fs.watchDirectory()?

Asking for the source is a good first check - hallucinated APIs won't have valid docs.

🤖 AI

I apologize for the error. I was mistaken - the correct method is fs.watch(), not fs.watchDirectory().

Here's the correct usage:

const fs = require('fs');

fs.watch('./my-folder', { recursive: true }, (eventType, filename) => {
  console.log(`${eventType}: ${filename}`);
});

Documentation: https://nodejs.org/api/fs.html#fswatchfilename-options-listener

Note: The recursive option behavior varies by platform - it works fully on macOS and Windows but has limitations on Linux.

When challenged, the AI corrects itself. The verification step caught the error before it caused problems.

Verification Checklist

👤 You

What's a quick way to verify AI-provided code before using it?

Explicitly asking for verification strategies builds good habits.

🤖 AI

Quick Verification Checklist:

Check method names exist:
- Search official docs for the exact method name
- Try console.log(Object.keys(module)) to see available methods
Verify parameters:
- Official docs list all parameters
- Hover in IDE to see TypeScript signatures

Test in isolation:

// Quick REPL test before integrating
node -e "const fs = require('fs'); console.log(typeof fs.watch)"
// Should print 'function', not 'undefined'

Check version requirements:
- Search "[feature] node.js version" if AI mentions version-specific features
Validate URLs:
- Actually click documentation links before trusting them
- AI often generates plausible but non-existent URLs

These are practical, quick checks you can run in seconds.

Key Takeaways

AI confidently generates plausible-sounding incorrect information
Specific API names, parameters, and URLs are high-risk for hallucination
Ask for documentation links and actually verify them
Quick REPL tests can catch errors before they enter your codebase
When in doubt, ask the AI to verify its own claims

Try It Yourself

Prompt Template

I want to use [LIBRARY/API] to [GOAL]. Can you show me how?

Also, please include:
1. A link to the relevant official documentation
2. The minimum version required for this feature
3. Any platform-specific limitations

Variations to Try

Before using: 'Let me verify - can you confirm this method exists in the official docs?'
After getting code: 'What would happen if I called this with [edge case]? Let me check the docs.'
If something seems off: 'I couldn't find [method] in the docs. Are you sure that's the correct name?'

Sources

Tempered AI — Forged Through Practice, Not Hype

? Keyboard shortcuts

Catching a Hallucination: Trust but Verify

Scenario

Conversation

Red Flags for Hallucination

Verification Strategies

Verification Checklist

Key Takeaways

Try It Yourself

Prompt Template

Variations to Try

Related Content

Sources