Embeddings & Vector Representations

Fundamentals beginner 15 min

Sources verified Dec 22

Embeddings convert text into numerical vectors that capture semantic meaning, enabling similarity search, clustering, and the foundation for RAG systems.

An embedding is a way to represent text (or images, audio, etc.) as a list of numbers — a vector. The key insight is that similar meanings result in similar vectors.

For example:

• "dog" and "puppy" → vectors that are very close
• "dog" and "airplane" → vectors that are far apart

This enables semantic search: finding content by meaning, not just keyword matching.

Think of it like coordinates on a map. "Paris" and "Lyon" are close together on a map of France. Similarly, semantically related concepts are close together in embedding space.

How Embeddings Work

1. Input: A piece of text (sentence, paragraph, document)
2. Model: A specialized embedding model processes the text
3. Output: A fixed-size vector (e.g., 1536 dimensions for OpenAI's text-embedding-3-small)

These vectors can then be:

• Compared using cosine similarity
• Stored in vector databases (Pinecone, Weaviate, pgvector)
• Used for retrieval in RAG systems

embeddings_example.ts

import OpenAI from 'openai';

const openai = new OpenAI();

// Generate embeddings for text
async function getEmbedding(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: text
  });
  return response.data[0].embedding; // 1536-dimensional vector
}

// Calculate similarity between two texts
function cosineSimilarity(a: number[], b: number[]): number {
  const dotProduct = a.reduce((sum, val, i) => sum + val * b[i], 0);
  const magnitudeA = Math.sqrt(a.reduce((sum, val) => sum + val * val, 0));
  const magnitudeB = Math.sqrt(b.reduce((sum, val) => sum + val * val, 0));
  return dotProduct / (magnitudeA * magnitudeB);
}

// Example usage
const query = await getEmbedding('How do I train a dog?');
const doc1 = await getEmbedding('Puppy training tips');
const doc2 = await getEmbedding('Airplane maintenance');

console.log(cosineSimilarity(query, doc1)); // ~0.85 (high similarity)
console.log(cosineSimilarity(query, doc2)); // ~0.20 (low similarity)

L8: Embedding models are fast and cheap

L11: Each text becomes a 1536-dimensional vector

L15: Cosine similarity measures angle between vectors

Common Embedding Models (2025)

Model	Dimensions	Best For
text-embedding-3-small	1536	General purpose, good cost/performance
text-embedding-3-large	3072	Higher accuracy, more expensive
Cohere embed-v3	1024	Multilingual, good for search
BGE-large	1024	Open source alternative

The RAG Connection

Retrieval-Augmented Generation (RAG) is the killer app for embeddings:

1. Index: Embed all your documents, store vectors in a database
2. Query: Embed the user's question
3. Retrieve: Find the most similar document chunks
4. Generate: Pass retrieved chunks to the LLM with the question

This lets LLMs answer questions about your specific data without fine-tuning.

Embeddings capture semantic meaning but not exact facts. "The capital of France" and "Paris is the capital" will have similar embeddings, but the embedding alone doesn't tell you the answer is Paris.

Key Takeaways

• Embeddings convert text to numerical vectors capturing meaning
• Similar meanings → similar vectors (measured by cosine similarity)
• Essential for semantic search and RAG systems
• Embedding models are separate from chat/completion models
• Vector databases store and search embeddings efficiently

In This Platform

Embeddings power semantic search for sources. When a claim needs backing, we could search sources by meaning rather than keywords to find relevant citations.

Relevant Files:

• sources/research_papers.json
• sources/official_docs.json

source_search.ts (future)

// Future: semantic source search
async function findRelevantSources(claim: string) {
  const claimEmbedding = await getEmbedding(claim);
  return sources
    .map(s => ({ source: s, similarity: cosineSimilarity(claimEmbedding, s.embedding) }))
    .sort((a, b) => b.similarity - a.similarity)
    .slice(0, 5);
}

Related Concepts

Tokens & TokenizationRetrieval-Augmented Generation (RAG)Context Windows

Sources

✓ OpenAI: Embeddings measure relatedness of text strings and are used for search, clustering, and recommendations ⓘ

×

OpenAI Embeddings Documentation

OpenAI

Embeddings convert text into numerical vectors that measure semantic relatedness, enabling semantic search, clustering, recommendations, and anomaly detection. OpenAI's text-embedding-3 models are foundational for RAG (Retrieval-Augmented Generation) systems. Understanding embeddings is essential for AI development maturity - it's how AI systems 'understand' and retrieve relevant context.

Key Findings:

• Embeddings measure relatedness of text strings
• text-embedding-3-small and text-embedding-3-large models
• Used for search, clustering, recommendations, anomaly detection

Credibility: high

View Source ↗