Embeddings are not understanding

There is a phrase that shows up in every embeddings tutorial, and it is a lie: "semantic understanding." Your embedding model does not understand anything. It produces a point in a high dimensional space, positioned by a training process that had no idea what your documents mean. The point encodes a statistical guess about which texts tend to appear in similar contexts. That is useful. It is not understanding.

I bring this up because I have seen the damage the word "semantic" does. A team builds a search system on embeddings, the demos look magical, then the first production query returns something confidently wrong, and nobody can explain why, because the mental model everyone had was wrong.

What an embedding actually is

An embedding model takes a piece of text and returns a list of numbers, say 768 of them or 1536. Those numbers are coordinates. The model has learned, from a huge amount of text, to place similar texts near each other in that space and dissimilar texts far apart. "Similar" here means something loose: texts that tend to be used in similar ways, appear near similar words, or answer similar questions.

The key property is that this is a geometric claim, not a logical one. Two texts are "similar" if they are close in the space. That turns search into a nearest neighbor query: embed the question, find the closest texts. It is a genuinely useful trick. It is also full of sharp edges that only show up when you rely on it for something that needs exactness.

The three things embeddings are bad at

Negation. "How to enable notifications" and "how to disable notifications" are nearly identical strings. They embed near each other. A pure vector search will happily return instructions for the opposite of what the user asked. Models have gotten better at this over time, but better is not reliable, and it is the exact class of mistake that is invisible in a demo and loud in production.

Exact identifiers. SKUs, license plate numbers, error codes, class names, phone numbers. "Find the order with reference TQ-8842" is not a semantic question. It is a lookup. An embedding of the query will be close to embeddings of other orders that happen to share words, and the actual TQ-8842 might be far away because the model does not treat the token as a unit. This is why every serious search system keeps an exact match path, not just a vector path.

Rare and specialized vocabulary. Embedding models are trained on general text. If your corpus is full of domain jargon, internal abbreviations, or product names that did not exist in training, the vectors for those terms will be mush. The model will place them by whatever weak context it has, and your search results will reflect that. Fine tuning or a small custom embedding model often fixes this, but nobody budgets for it at demo time.

The database part: pgvector and the recall trade

Once you have vectors, you need somewhere to store and search them. You can run exact nearest neighbor search, which scans everything and is always right, or you can build an approximate index and trade a little recall for speed.

Postgres with the pgvector extension is a reasonable default, and I like it for the boring reason: the vectors live next to the rest of your data, you can join them with your other tables, you get transactions and backups for free, and you do not add a second database to your infrastructure. The HNSW index is the usual choice. It builds a graph of vectors and searches a small part of it. Query time stays low even on big tables, at the cost of some recall: with default settings you might miss a few of the true nearest neighbors.

The setting that matters more than the index type is the candidate list, which pgvector calls ef_search. Higher values scan more candidates and improve recall at the cost of speed. The embarrassing failure mode is shipping the default, noticing that results are slightly off, and blaming the model when the real problem is that your approximate index skipped the right answer before the model ever saw it.

Two practical notes from real systems. First, build the index after loading data, and create it concurrently in production so you do not block writes. Second, check your recall. The honest way is to run the same query against exact search and the approximate index and compare. If recall is 92 percent, fine, that may be the right trade. If it is 60 percent, you have a settings problem, not a model problem.

If you want the index smaller, half precision vectors roughly halve the storage and often cost little in quality. Binary quantization goes further and is surprisingly good when combined with a rerank step over the original vectors. These matter when your index stops fitting in memory, which is when vector databases start to hurt.

Chunking is a retrieval decision, not a text decision

Embeddings have a length limit, and documents are longer than it, so you split documents into chunks and embed each chunk. The chunking choice is now the most important design decision in your search system, because it determines what a "nearest" result means.

Split too small and each chunk loses context. Split too big and you embed noise, and the answer is diluted across an entire section. The common compromise is a few hundred characters with overlap, but I have seen far better results from splitting on structure: headings, sections, paragraphs, list items. A chunk that is a coherent unit of meaning embeds better than a chunk that is a fixed number of characters.

The details that nobody remembers until it breaks: whether the document title gets prepended to every chunk, whether section headings are included in the chunk text, whether you filter by metadata before or after the similarity search. All of these move results more than the choice of model. Treat them as search parameters to be tuned, not implementation details.

When similarity is the wrong abstraction

The hardest lesson is knowing when not to use embeddings. If the query is a filter, do not embed it. "Show me invoices from March" is a date range, not a similarity problem. If the data is structured, query it structurally. If the identifier is exact, do an exact lookup.

The systems that work combine paths. Exact match for identifiers, keyword search for the words that matter, vector search for the paraphrase and concept cases, and filters for the structured dimensions. Embeddings are one leg of the stool. A stool with one leg does not stand, no matter how good the leg is.

The honest framing

Embeddings are a compression of text into geometry, learned statistically, useful for approximate matching of meaning. Say that sentence out loud when you are designing the system and you will make better decisions. You will add the exact match path. You will test negation. You will check your recall. You will treat chunking as retrieval engineering.

The moment you quietly replace "approximate" with "semantic" in your head is the moment the production incident starts writing itself.