Embeddings: From Theory to Production Choice

Problem Context

Embeddings turn text, images, and code into points in a high-dimensional vector space where geometric distance approximates semantic similarity. They are the unglamorous foundation under RAG, semantic cache, deduplication, classification, recommendation, and clustering. If you understand embeddings well, every "AI feature" on your roadmap becomes a vector-distance problem with a known shape.

What an Embedding Actually Is

An embedding is a dense float vector — typically 256–3072 dimensions — produced by a model trained so that semantically similar inputs map to nearby points. Modern text embedders (OpenAI text-embedding-3, Cohere embed-v3, Voyage voyage-3) are contrastively trained: pull similar pairs together, push unrelated pairs apart.

flowchart LR
    T["'Cancel my order'"] --> E[Embedding model]
    T2["'How do I refund?'"] --> E
    T3["'Best pizza in Berlin'"] --> E
    E --> V1["[0.12, -0.34, ...]<br/>1536-d"]
    E --> V2["[0.11, -0.31, ...]"]
    E --> V3["[-0.42, 0.88, ...]"]
    V1 -. cosine 0.92 .- V2
    V1 -. cosine 0.18 .- V3
      

The Three Numbers That Matter

• Dimensions — 1536 is the modern default. Higher = more capacity, more storage, more compute. Matryoshka models let you truncate (3072 → 256) with graceful degradation.
• Context length — how many tokens fit per embedding call. 8K for most, 32K for newer models. Long docs get chunked first.
• Distance metric — cosine (angle) for most text models, dot product when vectors are normalised, L2 for some image models. Pick the one the model card specifies; mixing is silently wrong.

Cosine Similarity Has No Universal Threshold

Every model has a different distribution. On text-embedding-3-small, random pairs hover around 0.1, on-topic pairs around 0.4, true paraphrases above 0.7. On embed-v3, random pairs sit near 0.3. The lesson: never hard-code a threshold without measuring your own corpus.

# Calibrate once per model + corpus
random_sims = [cosine(embed(a), embed(b)) for a, b in random_pairs(n=1000)]
related_sims = [cosine(embed(a), embed(b)) for a, b in known_related(n=1000)]
threshold = percentile(related_sims, 25)  # accept top 75% of true pairs

Chunking Strategies

• Fixed-token (e.g. 512 tokens, 64 overlap) — simple, predictable cost.
• Recursive — split on paragraphs, then sentences, then tokens. Better boundaries.
• Semantic — split where embedding similarity between adjacent sentences drops below a threshold. Best recall, highest compute.
• Document-aware — respect headings/sections. Always preserve title and section_path as metadata.

Hybrid Retrieval Beats Pure Vector

Pure vector search is bad at exact-match queries ("error code E1234") and proper nouns. Pure BM25 (keyword) is bad at synonyms. Production retrieval almost always combines:

score = w_bm25 * bm25(q, d) + w_vec * cosine(embed(q), embed(d))
       + w_rerank * cross_encoder(q, d)

A cross-encoder reranker on the top-50 candidates is the single highest-leverage retrieval improvement in 2026.

Re-Embedding Is Expensive — Plan For It

• Store the model name + version alongside every vector. Mixed vectors in one index = silent corruption.
• Re-embed in a shadow index; cut over with a single config flip; keep the old index for a week.
• Use Matryoshka models — same vector at 1536-d and 256-d — to ship faster/cheaper variants without re-embedding.
• Budget: 100M chunks × $0.02 / 1M tokens × 400 tokens/chunk ≈ $800 — surprising, but bearable. Storage is the real cost over time.

Failure Modes

• Mixing models in one index — silent garbage. Tag every vector with model name and version, refuse mixed queries.
• Embedding the wrong text — embedding the title only, missing the body. Test retrieval on real questions.
• Normalisation mismatch — embedder returns unit vectors, you store raw → cosine works, dot product silently rescales.
• Stale embeddings — source doc updated, vector not. Embed-on-write with a job queue.

Production Checklist

• Pick a model with stable versioning and good multilingual support if you have non-English users.
• Calibrate similarity thresholds against your own corpus, not the model card.
• Hybrid retrieval (BM25 + vector) + cross-encoder rerank on top-K.
• Store model_name, model_version, chunk_size, content_hash with every vector.

Key Takeaways

• Embeddings are points in a model-specific space — distances only mean something inside that space.
• Hybrid retrieval + reranking beats pure vector search in almost every production scenario.
• Versioning, chunking strategy, and threshold calibration are the parts you'll regret skipping.