Start With Keyword Search
Keyword search is the right first search. It's cheap, it's exact, and while people type the words your documents use, it finds what they want. It fails at one specific thing: people describing a problem in their own words.
A customer types "my package arrived crushed" into your help centre. You have an article that answers it exactly, titled "Returning items damaged in transit". Keyword search returns nothing, because the two texts do not share a single word. The customer opens a ticket, and someone on your support team replies with a link to the article they could not find.
Embeddings fix that particular failure. A model turns each piece of text into a list of numbers, placed so that texts with similar meaning end up close together, and search becomes "find the nearest vectors". The same idea sits under RAG, semantic caching, deduplication and recommendations. The maths is rarely the hard part. The hard parts are the decisions around it: which model, what exactly you embed, and what happens when you change either.
- You picked
text-embedding-3-smallbecause it was cheapest and now retrieval is bad - You can't explain why cosine similarity 0.62 is "good" or "bad"
- You re-embedded the entire corpus to upgrade models and your bill exploded
What an Embedding Actually Is
An embedding is a dense float vector — typically 256–3072 dimensions — produced by a model trained so that semantically similar inputs map to nearby points. Modern text embedders (OpenAI text-embedding-3, Cohere embed-v3, Voyage voyage-3) are contrastively trained: pull similar pairs together, push unrelated pairs apart.
flowchart LR
T["'Cancel my order'"] --> E[Embedding model]
T2["'How do I refund?'"] --> E
T3["'Best pizza in Berlin'"] --> E
E --> V1["[0.12, -0.34, ...]<br/>1536-d"]
E --> V2["[0.11, -0.31, ...]"]
E --> V3["[-0.42, 0.88, ...]"]
V1 -.->|cosine 0.92| V2
V1 -.->|cosine 0.18| V3
The Three Numbers That Matter
- Dimensions — 1536 is the modern default. Higher = more capacity, more storage, more compute. Matryoshka models let you truncate (3072 → 256) with graceful degradation.
- Context length — how many tokens fit per embedding call. 8K for most, 32K for newer models. Long docs get chunked first.
- Distance metric — cosine (angle) for most text models, dot product when vectors are normalised, L2 for some image models. Pick the one the model card specifies; mixing is silently wrong.
Cosine Similarity Has No Universal Threshold
Every model has a different distribution, so the same pair of texts can score very differently on two models, and a threshold that worked on one tells you nothing about the next. The lesson: never hard-code a threshold without measuring your own corpus.
# Calibrate once per model + corpus
random_sims = [cosine(embed(a), embed(b)) for a, b in random_pairs(n=1000)]
related_sims = [cosine(embed(a), embed(b)) for a, b in known_related(n=1000)]
threshold = percentile(related_sims, 25) # accept top 75% of true pairsCheck it: Take twenty pairs of texts you know are related and twenty you know are not. Embed them and look at the two sets of scores. Put your cut-off where they separate. If they overlap heavily, a cut-off won't work for this model, and you should show the top results instead.
Chunking Strategies
- Fixed-token (e.g. 512 tokens, 64 overlap) — simple, predictable cost.
- Recursive — split on paragraphs, then sentences, then tokens. Better boundaries.
- Semantic — split where embedding similarity between adjacent sentences drops below a threshold. Best recall, highest compute.
- Document-aware — respect headings/sections. Always preserve
titleandsection_pathas metadata.
Check it: Pick five real questions whose answers sit in the middle of long documents. Search for each one and check that the passage which answers it is in the top results, not just the right document.
Hybrid Retrieval Beats Pure Vector
Pure vector search is bad at exact-match queries ("error code E1234") and proper nouns. Pure BM25 (keyword) is bad at synonyms. Production retrieval almost always combines:
score = w_bm25 * bm25(q, d) + w_vec * cosine(embed(q), embed(d))
+ w_rerank * cross_encoder(q, d)Treat that formula as a sketch. Those scores sit on different scales, so normalise them, or merge the rankings with reciprocal rank fusion, and tune any weights on an evaluation set. A cross-encoder reranker on the top candidates is often the best next step, but confirm that on your own queries.
Check it: Search for an exact identifier from your own content, such as an error code. Vector search alone often misses it. With keyword search merged in, it should come back first.
Re-Embedding Is Expensive — Plan For It
- Store the model name + version alongside every vector. Mixed vectors in one index = silent corruption.
- Re-embed in a shadow index; cut over with a single config flip; keep the old index for a week.
- Use Matryoshka models — same vector at 1536-d and 256-d — to ship faster/cheaper variants without re-embedding.
- Budget: 100M chunks × $0.02 / 1M tokens × 400 tokens/chunk ≈ $800 — surprising, but bearable. Storage is the real cost over time.
Check it: Pick any vector in your index and try to say which model and version produced it. If you can't tell from what is stored, an upgrade will mix models without anyone noticing.
Failure Modes
- Mixing models in one index — silent garbage. Tag every vector with model name and version, refuse mixed queries.
- Embedding the wrong text — embedding the title only, missing the body. Test retrieval on real questions.
- Normalisation mismatch — dot product and cosine only agree on unit vectors. If your vectors aren't normalised, or you transform them, dot product also reflects their length. Check what your model returns.
- Stale embeddings — source doc updated, vector not. Embed-on-write with a job queue.
Production Checklist
- Pick a model with stable versioning and good multilingual support if you have non-English users.
- Calibrate similarity thresholds against your own corpus, not the model card.
- Hybrid retrieval (BM25 + vector) + cross-encoder rerank on top-K.
- Store
model_name,model_version,chunk_size,content_hashwith every vector.
Try It on Your Own System This Week
Three checks on a search you already run, about an hour in total:
- Collect twenty searches that returned nothing. Your search logs have them. For each one, check whether an article exists that answers it. Those are the cases embeddings fix.
- Read the exact text you embed. Open one stored record and look at what was sent to the model. If it's a title without its body, or a body without its title, retrieval is working with half the meaning.
- Change a document and search for the new wording. If it isn't found, your vectors are going stale, and you need to embed on every write.
What Changed in Embedding Models
Two developments changed how I choose embedding models.
Dimension control is now common. Several current models can return shorter vectors from the same model, which is called Matryoshka embedding. You can store 512 dimensions in place of 3,072 and keep most of the quality, at a fraction of the storage and with faster search. Test this on your own data before you commit, because the loss varies, but it's often the cheapest optimisation available. Binary quantisation goes further: storing one bit per dimension cuts memory by about 32 times, and many vector stores can search the compact form first and re-rank the top results with full vectors.
Multimodal embeddings are also practical now. Models that embed images and text into one space let you search a document's diagrams and tables as well as its prose, which helps with PDFs that plain text extraction ruins.
Choosing a model
Pick by running your own test. Take 50 to 100 real queries with known good documents, embed your corpus with two or three candidates, and compare recall@10. Public leaderboards use data that isn't yours, and differences between good models are often smaller than the gain from better chunking or adding a reranker. Domain matters as well. A model that does well on general web text can do badly on legal contracts or code, and that only shows up when you test on your own documents.
Think about cost in two parts. Embedding the corpus is usually a one-off, and it's cheap with a batch API. Query embedding is small but it's on your latency path, so a local model can make sense there even if the corpus was embedded elsewhere, provided it's the same model.
Check it: Run the same fifty queries against two candidate models and count how often the known good document is in the top ten. If the difference is a point or two, pick the cheaper model.
Multi-language search
If your users write in several languages, test across them. Many models handle same-language retrieval well and cross-language retrieval poorly. Ask in German against English documents and see what comes back. If results are poor, either choose a model built for multiple languages or translate the query as a separate step.
Metadata belongs in the chunk
An embedding only knows what's in the text it was given. A chunk that says "the limit is 500" means nothing without knowing which product it refers to. Prefixing each chunk with a short line of context, such as the document title and section, is cheap and usually improves retrieval noticeably. Anthropic described a stronger version as contextual retrieval, where a model writes that context for each chunk, and prompt caching makes it affordable.
Key Takeaways
- Embeddings are points in a model-specific space — distances only mean something inside that space.
- Hybrid retrieval + reranking beats pure vector search in almost every production scenario.
- Versioning, chunking strategy, and threshold calibration are the parts you'll regret skipping.