Every RAG article on this blog assumes you know what an embedding is and moves straight to chunking and retrieval. This one fills that gap, because the embedding model you choose quietly sets the ceiling on your whole RAG system's quality, and switching it later is a full re-embedding migration.

What an embedding actually is

An embedding is a vector, a list of numbers, that represents the meaning of a piece of text. A model with 1536 dimensions turns each input into 1536 floats. Texts with similar meaning land close together in that space; unrelated texts land far apart. Closeness is measured with cosine similarity or a dot product, and that is the whole trick behind semantic search: embed the query, find the nearest document vectors, and you have the most relevant text.

The reason this beats keyword search is that meaning is not words. "How do I cancel my plan" and "steps to end a subscription" share almost no vocabulary, but their embeddings sit right next to each other, so a vector search finds the right document with no keyword overlap. The same mechanism powers clustering, deduplication, recommendation and, of course, the retrieval half of RAG.

"cancel my subscription"  ->  [ 0.02, -0.41,  0.88, ... ]  -.
"end my plan"             ->  [ 0.05, -0.39,  0.85, ... ]  -+-- cosine ~ 0.95  (close)
"reset my password"       ->  [ 0.61,  0.12, -0.30, ... ]  ---- cosine ~ 0.20  (far)

Choosing a model: the real trade-offs

Start with an API model to validate the use case with no infrastructure. Move to a self-hosted open-source model only when per-call cost, latency or data residency actually forces the change, and budget the operational work that comes with it.

How to compare them: the MTEB benchmark

MTEB, the Massive Text Embedding Benchmark, ranks embedding models across retrieval, classification, clustering and reranking tasks in many languages. Look at the retrieval column, not the overall average, and filter the leaderboard to the languages your corpus uses. Then treat it as a shortlist: run your own small evaluation on your data, a dozen real queries with the documents they should return, because domain fit routinely beats a benchmark rank.

Dimensionality: bigger is not always better

A higher-dimensional embedding, 3072 versus 1536 for example, captures more nuance, but it costs more to store, is slower to search, and takes more index memory. Many modern models support Matryoshka-style truncation: you can cut a 3072-dimension vector down to 512 or 256 and keep most of the retrieval quality. The accuracy drop is usually small and the storage and speed win is large, but measure it on your own data before committing to a size.

Multimodal embeddings

Some models embed text and images into the same space, so a text query can retrieve a relevant image and an image can retrieve related text. CLIP-style models started this, and newer multimodal embedding models handle documents that mix text and figures, product catalogs with photos, or screenshots. Reach for one only when your corpus genuinely is not just text: a dedicated text model still outperforms a multimodal one on pure text retrieval, so cross-modal search has to be a real requirement, not a nice-to-have.

Generating an embedding

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Migrating a Laravel monolith to microservices",
    dimensions=512,  # Matryoshka truncation, down from the default
)

vector = response.data[0].embedding  # a list of 512 floats

When to fine-tune an embedding model

Fine-tuning an embedding model only pays off with a sizeable labelled dataset of your domain's query-document pairs, typically in the thousands. Before you reach that point, most teams get more value from better chunking, a reranker, or hybrid search. When fine-tuning does make sense, training a small adapter on top of a strong base model, with a library like sentence-transformers, is usually enough; you rarely train from scratch.

Operational details that bite later

• ✓Normalise consistently: if the model returns unnormalised vectors, normalise before storing so cosine similarity and dot product agree.
• ✓Batch your embedding calls: one request per document is slow and expensive at any real scale.
• ✓Record the model and its version in each vector's metadata, so you can tell which vectors came from which model during a migration.
• ✓Cache embeddings for unchanged documents and only re-embed what actually changed.
• ✓Chunk before embedding: every model has a token limit and truncates silently past it, so a long document embedded whole loses its tail.

FAQ

What is a text embedding?

A vector of numbers, produced by a model, that represents the meaning of a piece of text. Texts with similar meaning have vectors that are close together, which lets you search by meaning instead of by keyword.

Which embedding model should you use?

Start with an API model such as OpenAI text-embedding-3 or Cohere Embed to validate the use case with no infrastructure. Move to a self-hosted open-source model, BGE, E5 or GTE, when per-call cost, latency or data residency forces it. Check the retrieval column of the MTEB leaderboard for your languages, then run a small evaluation on your own data.

How many dimensions do you need?

Often fewer than the default. 512 to 1024 dimensions covers most retrieval use cases, and Matryoshka truncation lets you shorten a larger vector with a small quality loss. More dimensions mean more storage and slower search, so measure the accuracy drop on your data before paying for the full size.

Do embeddings work across languages?

With a multilingual model, yes. Cohere Embed and the multilingual BGE and E5 variants map different languages into a shared space, so a query in one language can retrieve documents in another. A monolingual model cannot, so match the model to the languages in your corpus.

Embeddings are the foundation every downstream retrieval decision depends on. Pick a model that fits your languages and domain, keep the dimensions no larger than you need, and treat a model change as a migration. Get this right early and the rest of the RAG stack has something solid to build on.