Embeddings Guide

Embeddings turn text into vectors where similarity means relatedness. They're the foundation of semantic search, RAG retrieval, clustering, and recommendations. The skill is choosing the right model, using it correctly, and evaluating on your data.

Overview

An embedding model maps text to a dense vector; cosine similarity (usually) measures relatedness. Quality varies by domain, language, and text length — a model great at English Wikipedia may fail on your support tickets. Key choices: the model itself, how you handle long texts (chunk + pool), whether to use task-specific instructions (asymmetric search: query vs. document phrasing differs), and how you evaluate (on your data, not the leaderboard).

When to use

• Semantic search or RAG retrieval over your corpus.
• Clustering, deduplication, or topic discovery on text.
• Recommendation: "similar items" features.
• Anywhere you need "find text like this text."

Core concepts

• Model selection: size/quality/speed trade-offs; domain fit matters more than leaderboard rank. Evaluate candidates on your own query-document pairs.
• Asymmetric vs. symmetric: search queries differ from documents (short question vs. long passage). Instruction-tuned models handle this with task prefixes; test whether yours needs them.
• Pooling long texts: embed chunks and aggregate (mean, max) or retrieve at chunk level. Don't truncate long documents to the model's limit and hope.
• Similarity measures: cosine (standard, normalize vectors), dot product (with unnormalized, scale-sensitive), Euclidean (less common for text). Be consistent between indexing and querying.
• Normalization: L2-normalize vectors so cosine = dot product; many vector DBs assume this. Mismatched normalization silently breaks ranking.
• Evaluation: build a small set of query → known-relevant documents from your corpus. Measure recall@k per candidate model. This beats any public benchmark for your decision.

Practical workflow

1. Collect 30–50 representative queries with known-relevant documents from your corpus.
2. Shortlist 3–4 embedding models (varying size); embed your corpus with each.
3. Measure recall@k and MRR on your eval set; also measure embedding latency and cost.
4. Decide chunking: chunk documents first, embed chunks, retrieve chunks — tune chunk size on the eval set.
5. Normalize consistently; verify the similarity metric matches between index and query time.
6. Monitor: sample production queries periodically; re-evaluate when the corpus or model changes.

Embedding eval template:
QUERIES:  30-50 real queries + known-relevant docs
MODELS:   3-4 candidates (note dims, latency, cost)
METRICS:  recall@5, recall@20, MRR
DECIDE:   best quality within latency/cost budget

Common pitfalls

• Leaderboard shopping: picking the top benchmark model without testing on your data. Domain mismatch is common.
• Truncating long documents: embedding only the first 512 tokens of a 10-page doc. Chunk properly.
• Ignoring the query/document asymmetry: same embedding approach for both when the model expects task prefixes. Read the model's docs.
• Normalization mismatch: indexing normalized, querying unnormalized (or vice versa). Ranking breaks silently.
• No eval set: "embeddings feel good." Build the query-relevance set; it's a day's work that pays forever.
• Embedding everything at query time: precompute document embeddings; embed only the query live. Obvious but often missed under deadline.