Vector search

Why exact nearest-neighbor search does not scale

Semantic search turns your query into an embedding, then finds the stored vectors closest to it. Done exactly, that means comparing the query against every stored vector and keeping the top K. That is a full scan: O(N) distance computations per query. At 100k vectors it is fine; at 30M it is 30M dot products for every request, and latency falls apart.

In pgvector, ORDER BY embedding <-> :q LIMIT 10 with no vector index runs exactly and returns perfect results, just slowly. Adding an approximate index (HNSW or IVFFlat) changes the deal: the search becomes approximate, returning near neighbors in far less time and giving up a little recall, the fraction of true nearest neighbors it actually finds. That recall-for-speed trade is the whole reason ANN (approximate nearest neighbor) indexes exist.

Wrong "The index stores the vectors pre-sorted by distance, so lookups are instant." Distance is measured against each query vector, which is different every time, so there is no fixed order to sort by. Right an ANN index is a navigation structure you traverse per query, not a sorted list and not a result cache.

HNSW, the index most defaults reach for

HNSW (Hierarchical Navigable Small World) builds a layered graph. The bottom layer holds every vector with short links to close neighbors; each layer above keeps fewer nodes with longer links. A search starts at the top, takes coarse jumps toward the query, then drops down layer by layer, refining to the nearest neighbors at the bottom. That hierarchy (the paper compares it to a skip list) is why it finds neighbors in roughly logarithmic time instead of scanning everything.

Two build-time knobs and one query knob matter. In pgvector, m (default 16) is the number of links per node and ef_construction (default 64) is how hard the builder searches while wiring the graph; both raise recall and index size. At query time, hnsw.ef_search (default 40) is the size of the candidate list the search keeps. Raise it and the search explores more of the graph, so recall goes up and latency with it.

Wrong "Lower m makes the graph lighter, so search reaches more nodes and recall improves." Fewer links means a sparser, worse-connected graph, so recall drops. Bigger m and ef_construction cost memory and build time but improve recall; ef_search is the cheap dial you turn per query.

IVFFlat: clusters and probes

IVFFlat partitions the vectors into lists clusters, each with a centroid. A query compares itself to the centroids, then searches only inside the probes closest clusters instead of the whole set. In pgvector, probes defaults to 1, and raising it scans more clusters for higher recall at more latency. The lists count is fixed at build time (the docs suggest rows/1000 up to 1M rows, and sqrt(rows) beyond).

The trap is build order. IVFFlat learns its centroids from the data, so this:

CREATE INDEX ON items USING ivfflat (embedding vector_l2_ops) WITH (lists = 100);
-- then bulk-load the rows

builds clusters from an empty table, and recall is terrible no matter how you tune probes. Build the IVFFlat index after the table is populated. HNSW has no such requirement (it can be built on an empty table and filled incrementally), which is one reason it is the more common default.

Distance metrics have to match the model

pgvector exposes <-> (L2), <=> (cosine), and <#> (negative inner product). The metric is not a free choice: it has to match the distance the embedding model was trained to use, or "nearest" means something the model never optimized for. Inner-product and dot-product models usually need unit-length vectors; cosine handles magnitude for you. Get this wrong and the index works perfectly while returning subtly bad neighbors, which is harder to catch than an outright error.

Filtered vector search is where naive setups break

Real queries are "nearest neighbors that also match tenant_id = 42" or "in stock." How the filter combines with the ANN index decides whether you get usable results.

Post-filter runs the ANN search first, then drops candidates that fail the predicate. When the filter is selective, the graph hands back the globally nearest vectors, most of them fail the predicate, and you are left with far fewer than K, sometimes zero. pgvector applies filters after the index scan by default, which is exactly this behavior; its answer is iterative index scans (hnsw.iterative_scan = strict_order or relaxed_order) that keep scanning more of the index until enough rows pass.

Pre-filter, or filterable ANN, applies the predicate during traversal so the search only ever considers matching vectors. Elasticsearch does this and guarantees K matching results; when the filtered set is smaller than num_candidates it brute-forces the matches instead. The subtlety worth carrying in: a hard filter applied naively during a graph walk can disconnect the graph, leaving matching vectors unreachable through the links that survive. Qdrant's fix is a filterable HNSW that adds extra edges based on indexed payload values so the graph stays navigable under a filter, and it falls back to a full scan when the filtered set is below full_scan_threshold_kb.

Wrong "A WHERE clause on a vector query always returns K matches, the index just filters as it goes." Right only a pre-filtering index guarantees that. pgvector's default post-filter can starve the result set, and the fix is iterative scans, a higher ef_search or probes, a partial index per tenant, or a store with a real filterable index.

Quantization trades memory for recall

Full float32 vectors are large: 30M vectors at 768 dims is about 92 GB before the index. Quantization shrinks them, and each form sits at a different point on the memory-versus-recall curve.

• Scalar / int8: map each float32 component to an 8-bit integer. About 4x smaller, and in Qdrant's measurements under ~1% error, because SIMD makes int8 distance fast.
• Product quantization (PQ): split each vector into subvectors and replace each with the id of its nearest centroid in a learned codebook. Up to ~64x smaller, but the distance math is not SIMD-friendly, so it is slower to compute and loses more accuracy.
• Binary: reduce each component to a single bit. About 32x smaller and very fast, but only viable for high-dimensional vectors, and it drops real precision.

Wrong "Quantization is lossless, so it saves memory with the same results." Every form except plain float is lossy. Production setups recover recall by oversampling (pull more candidates from the quantized index than you need) and rescoring the top of that list against the full-precision vectors. pgvector spells this out with binary_quantize(...)::bit(n) plus a rerank step; Qdrant exposes an oversampling factor. If you cannot keep the original vectors around to rescore, expect the recall hit to stick.

The triangle you are actually tuning

Recall, latency, and memory are three corners, and every knob moves you along one edge. A bigger ef_search or more IVF probes buys recall and spends latency. Quantization buys memory and spends recall, some of it bought back by rescoring, which spends latency. Larger m and ef_construction buy recall and spend memory and build time. No single setting improves all three. Interviewers probe this because a candidate whose only lever is "make it faster" has not understood that faster usually means less recall.

When Postgres is enough

pgvector is the right call when the vectors live next to relational data you already query, the corpus is moderate, and you want one datastore with transactional writes and no sync pipeline. You get exact search for free (perfect recall, just slower) and HNSW or IVFFlat when you need speed. A dedicated store (Qdrant, Milvus, Elasticsearch kNN) earns its cost when you need filterable ANN at high selectivity, first-class quantization, horizontal sharding across nodes, or query volume you do not want competing with your OLTP workload. The honest interview answer is a workload test, not "always move" or "never move."

Building, deletes, and hybrid retrieval

An HNSW build is expensive and memory-hungry; give it enough maintenance_work_mem or the build spills and crawls. Deletes are the quiet maintenance cost: an ANN graph does not cleanly remove a node, so heavy churn degrades recall and eventually wants a rebuild (in Postgres, autovacuum reclaims dead tuples and the index repairs behind it).

Hybrid retrieval fuses this vector ranking with a lexical one (BM25) via Reciprocal Rank Fusion; the fusion strategy lives in the full-text-search and RAG topics.