Vector Index Design: HNSW, IVF, Flat, and the Cosine-vs-Dot Decision

There is no universally best vector index. The right choice is a function of corpus size, recall target, query latency budget, update frequency, filter selectivity, memory envelope, and the geometry the embedding model was trained for — not a ranking of algorithms. This article treats vector index design as a layered decision (similarity metric → index family → compression → dimensionality → storage engine), grounds each layer in primary sources, and addresses the practitioner question of when to leave Postgres+pgvector for a dedicated vector database.

Coverage note: verified through March 2026.

1. The actual question

Most articles on this topic answer "which index should I pick?" That is the wrong frame. The first decision is whether approximate nearest-neighbor (ANN) search is the bottleneck at all. FAISS itself frames similarity search as a precision–speed–memory tradeoff and includes exact, approximate, GPU, disk-resident, binary, and predicate-filtered indexes in one toolkit (FAISS docs). ANN-Benchmarks reports recall vs queries-per-second, index size, and build time across many datasets, and the original benchmark paper observes that very different algorithmic approaches yield comparable quality–performance frontiers (Aumüller et al., 2018; ANN-Benchmarks site). No single index dominates universally.

The decision tree underneath the prompt is roughly:

Geometry. What metric does the embedding model expect? Should vectors be L2-normalized?
Exact vs approximate. Is brute-force search adequate? What is the recall target?
Index family. Flat, IVF, HNSW, DiskANN, ScaNN, or a composition of these.
Compression. Scalar, FP16, PQ/OPQ, binary, or learned/non-uniform quantizers.
Dimensionality. Native dimension, truncated (Matryoshka), or projected (PCA / random projection).
System. Stay with Postgres+pgvector, move to a dedicated vector engine, or compose.

Each layer's correct answer is workload-dependent, and the layers interact: cosine vs dot changes how quantization error propagates; HNSW's graph topology interacts badly with selective metadata filters; DiskANN's SSD-resident graph design is motivated specifically by indexes that exceed RAM. Treating these as independent menu items produces brittle systems.

A second framing point worth surfacing: ANN recall is not the same as task quality. Recall@k against an exact baseline measures whether the approximate index reproduces the full-precision ranking; it does not measure whether the retrieved chunks answer the user's question, whether hybrid lexical search is needed, whether metadata filtering dominates the result set, or whether a reranker overrides ANN ordering anyway. A retrieval system can have 99% recall@10 against Flat and still produce poor RAG answers. Evaluation needs both ANN recall and end-to-end task metrics.

2. Similarity geometry: cosine, dot, L2, and normalization

The math

Given two vectors $a, b \in \mathbb{R}^d$:

Inner product (dot): $a \cdot b = \sum_i a_i b_i$. Higher is more similar. Sensitive to vector norm.
Cosine similarity: $\cos(a,b) = (a \cdot b) / (\lVert a \rVert \lVert b \rVert)$. Higher is more similar. Norm-invariant by construction.
Euclidean (L2) distance: $\lVert a - b \rVert_2$. Lower is more similar. For unit-norm vectors, $\lVert a - b \rVert_2^2 = 2 - 2(a \cdot b)$, so L2 ranking is monotonic with dot-product ranking.

The standard "trick" — that cosine equals inner product when both vectors are L2-normalized — is correct and is documented directly in the FAISS metric reference (FAISS MetricType and distances). For unit-norm vectors, dot, cosine, and L2 produce identical top-k rankings. OpenAI's embedding models are unit-normalized by default, and OpenAI explicitly states cosine can be computed with dot product and that cosine and Euclidean give identical rankings on those embeddings (OpenAI Embeddings FAQ).

When normalization is not free

This equivalence is mathematical hygiene. The harder question is whether you should normalize at all. The vector norm encodes signal in some embedding spaces and noise in others:

Models trained with cosine objectives (most modern sentence/text encoders, including normalized contrastive setups) produce embeddings where norm is approximately constant and uninformative. Normalizing is either a no-op or a small numerical correction.
Models trained with inner-product objectives (some retrieval-tuned encoders, recommendation embeddings, two-tower MIPS systems) may use norm to encode confidence, popularity, or salience. Normalizing discards that signal.
Heterogeneous concatenations (e.g., dense embedding concatenated with one-hot or count features) have norms that depend on the feature mix, not similarity, and almost always need normalization or per-block scaling.

Steck et al. argue against treating cosine similarity as semantically meaningful by default for embeddings; they construct cases where cosine becomes arbitrary or opaque depending on the embedding's training procedure (Steck, Ekanadham, & Kallus, 2024). The contribution is not "cosine is wrong" but "cosine inherits whatever geometric assumptions the model was trained under, and those assumptions are not always what the practitioner thinks they are." The operational rule is: use the metric the embedding model was trained and evaluated for, then validate empirically with task-level retrieval judgments before normalizing or denormalizing.

A compact decision matrix:

Embedding model contract	Default metric	Normalize?
Trained with cosine / unit-norm output (most text encoders)	Cosine ≡ dot ≡ L2 ranking	Already unit-norm; no-op
Trained for MIPS, norm encodes magnitude signal	Dot product	No — preserves signal
Concatenated heterogeneous features	Cosine after per-block scaling	Usually yes
Unsure / no documentation	Both, evaluated on labeled queries	Run both, diff the rankings

Implementation traps

The math is settled; implementation conventions are not. Three traps:

FAISS uses METRIC_INNER_PRODUCT for "higher is better" inner product but METRIC_L2 for "lower is better" L2 distance. Mixing thresholds across metric types silently breaks reranking (FAISS MetricType and distances).
pgvector exposes <#> as the negative inner product so its ascending-distance index ordering still puts the closest vectors first (pgvector README). Code that compares raw <#> values to a positive inner-product threshold inverts the ranking.
Many systems expose "cosine distance" as 1 - cos(a,b), which is not a true metric (no triangle inequality) but is monotonic with cosine similarity. It is not interchangeable with L2 distance even though both are "lower is better."

The fix is to separate the mathematical metric (a property of the embedding) from the library API (a property of the engine), and to keep that separation explicit in code.

3. The index family comparison

The five families that matter are Flat, IVF (with optional product quantization), HNSW, DiskANN, and ScaNN. They differ across more axes than recall and latency, and collapsing those axes into "fast vs slow" is where most articles on this topic mislead. The table below separates the axes that actually drive engineering decisions.

Family	Approach	Build cost	Query latency	Memory	Recall behavior	Update/delete	Filter interaction
Flat	Exhaustive scan	None	$O(N \cdot d)$ per query	$N \cdot d \cdot \text{bytes}$	Exact (recall = 1.0)	Cheap append; cheap delete	Trivial — apply filter pre- or post-scan
IVF / IVF-PQ	k-means partitioning ± product quantization	Moderate (requires training set)	$O((N/k) \cdot d)$ scaled by `nprobe`	Lower than Flat with PQ	Tunable via `nprobe`; depends on training quality	Append cheap; codebook drift on distribution shift requires retrain	Pre-filter (probe only matching cells) is possible but requires engine support
HNSW	Multi-layer navigable small-world graph	High (insert-time graph construction)	Sub-linear, sensitive to `efSearch`	$\approx d \cdot 4 + M \cdot 2 \cdot 4$ bytes/vector (FAISS guidance)	High recall at moderate `efSearch`; degrades sharply with selective post-filters	Insert is $O(\log N)$; deletes are usually tombstones, requiring periodic rebuild	Post-filter by default in many implementations — recall collapses on selective filters
DiskANN	Single-layer Vamana graph optimized for SSD	High; offline build	SSD-bounded; sub-millisecond on NVMe in published results	RAM holds compressed vectors + graph metadata; full graph on disk	High recall at billion scale	Update story varies by implementation; original work emphasized batch builds	Implementation-dependent; designed for read-heavy archival corpora
ScaNN	IVF-style partitioning + anisotropic vector quantization	Moderate; trains AVQ codebook	Fast on CPU with SIMD	Low (PQ-style codes)	High at top-k under MIPS objectives	Codebook retraining on drift	Implementation-dependent

A few notes the table can't carry:

Flat is routinely under-recommended. For corpora up to a few million vectors, batched queries, or GPU execution, exact matrix multiplication is simpler and more reliable than any ANN index. It avoids build time, index staleness, approximate misses, parameter tuning, and filtering surprises. FAISS's index-selection guidance explicitly recommends flat search when query volume will not amortize index build time, or when exactness is required (FAISS Guidelines to choose an index). Treating Flat as merely pedagogical pushes teams into premature infrastructure.
HNSW's memory overhead is real. FAISS documents that HNSW32 adds about 256 bytes per vector for the graph links alone, before the vector storage itself (FAISS index factory wiki). At $d = 1536$ and FP32 vectors, that is roughly a 4% overhead — but at $d = 384$ with FP16 or scalar quantization, the graph can be larger than the vectors.
HNSW under selective filters is a known failure mode. pgvector's documentation gives the canonical example: with default hnsw.ef_search = 40 and a metadata filter that matches 10% of rows, you get on average about four matching candidates after the index scan unless you raise ef_search or use iterative scans (pgvector README). The same pattern affects every implementation that filters after candidate generation.
DiskANN's claim was billion-scale on a single workstation. The original NeurIPS 2019 paper demonstrated billion-point search on a 64 GB workstation with SSD, motivated specifically by the regime where the graph cannot fit in RAM (Subramanya et al., 2019, NeurIPS; Microsoft Research project page). It is not a "general HNSW alternative" — it is an answer to a specific scaling regime.
ScaNN's contribution is not "use PQ." It is anisotropic vector quantization for maximum inner-product search, which penalizes error components differently based on their effect on inner-product ranking. The published version reported state-of-the-art ANN-Benchmarks numbers at the time (Guo et al., 2020, ICML; Google Research project page).

Index families in depth

Flat

A Flat index stores vectors in a contiguous array and computes the requested distance against every vector at query time. There is no graph, no partitioning, no codebook, and no training. FAISS exposes this as IndexFlatL2 and IndexFlatIP (FAISS Guidelines to choose an index).

Flat is the only index that guarantees exact nearest neighbors. That makes it the correctness baseline for every other index: if you do not measure recall against a Flat baseline on representative queries, you do not know how lossy your ANN setup is. This is true even when production runs on HNSW — the evaluation slice should run Flat for comparison.

Flat scales linearly. At $N \cdot d \cdot 4$ bytes (FP32) or $N \cdot d \cdot 2$ bytes (FP16), a 1M × 1536-dim FP32 corpus is ~6 GB, which fits on any developer laptop. At 10M × 1536, it is 60 GB, which already pushes most single-node deployments. The query is a matrix–vector multiply that vectorizes well; on modern CPUs with AVX-512 or on a single GPU, brute-force search through a few million vectors is comfortably sub-second per query and trivially batchable.

The point at which Flat stops working is workload-specific. The honest threshold is: when (corpus size × query rate × $d$) exceeds the memory bandwidth or latency budget the system can spend on retrieval. That is not a single number.

IVF and IVF-PQ

Inverted File (IVF) search partitions vectors into Voronoi cells using k-means: at index time, train $k$ centroids on a representative sample; at query time, identify the nprobe cells closest to the query and search only within those. Recall is controlled by nprobe: more probes, higher recall, more work.

IVF on its own (sometimes IVFFlat) reduces query work but does not reduce memory. IVF-PQ adds product quantization on top: split each vector into sub-vectors, learn a codebook per sub-vector, and store each vector as a tuple of codebook indices. PQ reduces memory by orders of magnitude (e.g., 1536-dim FP32 vector at 6144 bytes → 64-byte PQ code at 8 sub-vectors of 8 bits) at the cost of approximate distance computation.

Three IVF traps worth surfacing:

Training quality matters. The k-means centroids are trained on a sample. If the sample is not representative of the production distribution, recall degrades. Distribution drift over time is the same problem in slow motion.
Filter interaction is better than HNSW's if the engine supports per-cell filtering (you only probe cells that contain matching rows), but worse if it does not (post-filter behaves like HNSW under selective predicates).
PQ is not free. PQ codes give approximate distances. The standard pattern is candidate generation with PQ + exact reranking on the top-k candidates with full-precision vectors. If you do not store the full-precision vectors, you cannot rerank, and your reported recall is the PQ recall — which is often substantially lower than published numbers if the corpus has high intrinsic dimensionality.

pgvector's IVFFlat documentation gives a usable mental model: "fewer resources than HNSW, lower speed-recall performance," with lists controlling cell count and probes controlling search effort (pgvector README).

HNSW

Hierarchical Navigable Small World graphs (Malkov & Yashunin, 2016) are the algorithm most current vector systems treat as the default. The data structure is a multi-layer graph where each layer is a navigable small-world graph and higher layers are sparser. Search starts at the top layer with a single entry point, greedily descends to the closest neighbor, and uses that as the entry point for the next layer down. The lowest layer contains all vectors and is searched with a beam of size efSearch.

The empirical case for HNSW at moderate scale is strong: high recall, sub-millisecond latency, no training step, well-supported in FAISS, pgvector, hnswlib, and most managed engines. The original paper demonstrated state-of-the-art performance across the standard ANN benchmarks at publication time, and HNSW continues to dominate ANN-Benchmarks leaderboards for in-memory, read-heavy workloads (ANN-Benchmarks).

The case against treating HNSW as a universal default has four strands:

Memory overhead. Graph links cost roughly $M \cdot 2 \cdot 4$ bytes per vector in addition to the vectors themselves (FAISS Guidelines to choose an index). Default $M = 16$ adds ~128 bytes per vector; $M = 32$ adds ~256.
Build cost. Insert is $O(\log N)$ per vector, but the constant is large because each insert performs a beam search through the graph. Building a 100M-vector HNSW index takes hours to days depending on $M$, efConstruction, and CPU.
Update behavior. Most HNSW implementations cannot delete vectors in place; they tombstone the slot and require periodic rebuilds. Streaming-insert workloads degrade as the graph accumulates dead entries and the small-world property weakens. Recent work on dynamic large-scale vector systems (HAKES, LSM-VEC) explicitly identifies HNSW-style graphs as costly to build, contended under concurrent reads/writes, and difficult to scale across servers (HAKES, 2025; LSM-VEC, 2025).
Filter interaction. Standard HNSW search ignores predicates; predicates apply after the candidate set is returned. With a 10% selectivity filter, the practical recall on the filtered subset can be a fraction of the unfiltered recall unless efSearch is dramatically increased — at which point latency benefits disappear (pgvector README).

Tuning knobs: M (graph degree, build-time), efConstruction (build-time beam width, build-time only), efSearch (query-time beam width, the recall/latency dial). Default starting points in FAISS and pgvector are $M = 16$, efConstruction = 200, efSearch = 40–100, but production systems typically need efSearch raised when filters are present.

DiskANN

DiskANN is a graph index designed for the regime where the index does not fit in RAM. Its key construction, Vamana, is a single-layer graph that places vectors and graph adjacency on SSD and keeps a compressed (PQ-coded) summary in RAM. Search uses the in-memory summary to traverse the graph, fetching full-precision vectors from SSD only at the rerank step (Subramanya et al., 2019, NeurIPS; Microsoft Research project page).

The motivation is empirical. The original work demonstrated billion-point search on a 64 GB workstation with NVMe SSDs, with recall and latency competitive with in-memory HNSW on much smaller corpora. Big ANN 2021, the billion-scale benchmark explicitly created because million-scale benchmarks did not answer billion-scale deployment questions, used DiskANN and FAISS as baselines for the RAM-limited and SSD-assisted tracks (Simhadri et al., 2022).

DiskANN matters when the binding constraint is the cost of holding a graph index in RAM. It is less commonly used at sub-100M scale, where HNSW or IVF-PQ is simpler. Several managed vector DBs (Microsoft's various offerings, and recent Milvus releases) ship DiskANN-derived implementations.

ScaNN

ScaNN partitions the corpus IVF-style and uses anisotropic vector quantization to compress vectors. The compression objective is the contribution that distinguishes it from PQ: AVQ minimizes the expected error in inner-product ranking, not the expected reconstruction error of the vectors themselves. Components of the quantization error that move the inner-product estimate get penalized more heavily than components that do not (Guo et al., 2020, ICML; Google Research project page).

The published numbers are strong on MIPS workloads; the practical question is whether ScaNN is exposed in your stack. The reference implementation is Google's open-source library, which ships well-tuned CPU SIMD kernels but is less common as a managed offering. The core insight — that quantization error should be optimized for the ranking objective, not for L2 reconstruction — is now influencing other quantizers (see §4).

4. Quantization and the recent literature

Quantization compresses the per-vector storage. The four families that matter operationally in 2026 are scalar quantization, FP16/half-precision, product quantization (PQ/OPQ), and binary quantization. Recent research has added learned non-uniform quantizers, anisotropic methods, and rotation-based schemes (RaBitQ, SAQ, NVQ, TurboQuant, QINCo2) that change the achievable recall/memory frontier but vary widely in production maturity.

Established methods

FP16 / scalar quantization. Halves storage for negligible recall loss on most modern embeddings; widely supported in FAISS, pgvector, and most managed engines. The cheapest first move when memory is tight.
Product quantization (PQ) / Optimized PQ. Splits the vector into $m$ sub-vectors, learns a $k$-codeword codebook per sub-vector, and stores each vector as $m$ codebook indices. Memory shrinks by 8×–32× depending on configuration. Distance is computed approximately via lookup tables; exact reranking requires storing the full vectors separately. PQ's recall depends strongly on intrinsic dimensionality and on whether the data was pre-rotated (OPQ).
Binary quantization. Each component becomes a single bit (sign). Distance is Hamming or signed-popcount, which is extremely fast on modern CPUs (~1–2 ns per comparison with SIMD) but lossy. Production use almost always requires reranking with full-precision vectors. pgvector ships native binary quantization support (pgvector README).

Recent literature (2025+)

The literature is moving fast and does not converge to a single answer. The papers below optimize different objectives and measure different metrics, which is precisely the reason a wiki article should not collapse them into a "use the latest quantizer" recommendation.

QINCo2 (Huijben et al., 2025) — improved Quantized Implicit Neural Codecs with iterative encoding and pair-wise codeword search. Reports state-of-the-art on standard ANN datasets, with code released. Production maturity: research code.
TurboQuant (Zandieh et al., 2025) — argues that MSE-optimal quantizers can systematically bias inner-product estimation, and proposes a quantizer that targets unbiased inner-product recovery. Production maturity: paper-stage.
SAQ (Liu et al., 2025) — accepted at SIGMOD 2026; reports up to 80% lower quantization error and substantially faster encoding than Extended RaBitQ in its experiments. Production maturity: SIGMOD-accepted, reference implementation released.
NVQ (Wang et al., 2025) — learns per-vector non-uniform quantizers. Production maturity: research-stage.
Extended RaBitQ (Gao & Long, SIGMOD 2025) — practical, asymptotically optimal scalar quantization for high-dimensional vectors via random rotation; integrated into several open-source ANN systems.
SMEC (EMNLP 2025) — improved compressed retrieval embeddings on BEIR at 256 dimensions; the result is a model-side compression complementary to index-side quantization.

The honest summary: quantization quality at the recall-vs-memory frontier is improving year over year, and 2025 papers are making meaningful gains, but production-ready support in FAISS, pgvector, and managed engines lags by 12–24 months for non-trivial methods. RaBitQ-family and PQ-family methods are deployable today; QINCo2/SAQ/NVQ should be evaluated against your own corpus before adoption.

Why "compression knob" framing misleads

A common error is treating PQ, scalar, binary, and FP16 as interchangeable settings on the same dial. They are not:

They differ in recall behavior by orders of magnitude — binary without reranking can drop recall@10 by 30+ points on hard corpora; FP16 typically drops by less than 1 point.
They differ in hardware support — binary distance vectorizes to popcount; PQ relies on cache-resident lookup tables; FP16 needs hardware FMA support or runtime dequantization.
They differ in training requirements — PQ and learned quantizers need a training set representative of the production distribution; FP16 and binary do not.
They differ in interaction with reranking — PQ and binary almost always require an exact rerank step on top-k candidates with full vectors; FP16 typically does not.

The right way to use these is to pick one, set up a Flat baseline, measure recall@k against it on representative queries with realistic filters, and only then decide whether the storage savings justify the recall loss. Reconstruction error and benchmark recall on Sift1M are not substitutes for that measurement.

5. Dimensionality

Storage and distance computation scale roughly linearly with embedding dimension. Dimensionality is therefore a tempting cost lever — but the lever can damage retrieval quality in ways that storage math does not predict.

There are three distinct strategies, and conflating them produces bad outcomes:

Native dimension. Use whatever the embedding model outputs. The model was evaluated at this dimension; ranking quality is whatever the model card claims.
Trained-to-be-truncated (Matryoshka). Some recent embedding models are trained with Matryoshka Representation Learning, which makes prefix-truncated vectors usable on their own. The original MRL paper reports up to 14× smaller embeddings and 14× retrieval speedups in their experiments while preserving useful accuracy (Kusupati et al., 2022, NeurIPS). Truncation is the intended use for these models.
Post-hoc dimensionality reduction (PCA, random projection, learned autoencoders). Project the existing high-dimensional space into a lower-dimensional one and search there. This is not the same as Matryoshka truncation. PCA preserves variance, not retrieval quality; rare or domain-specific queries are exactly the ones that get damaged because their signal lives in low-variance directions.

The operational rules:

If the embedding model documents Matryoshka support, truncating to a smaller dimension is a safe optimization. Validate it on labeled queries anyway; published numbers are aggregate.
If the model does not document Matryoshka support, prefer storage-side compression (FP16, PQ, binary) over dimension-side reduction. Quantization preserves the embedding geometry and is reversible at evaluation time; dimensionality reduction discards information that cannot be recovered.
If you must reduce dimensions, train the projection on representative data (PCA on a sample of production embeddings is typically better than random projection), and re-evaluate retrieval quality against the original-dimension baseline on the queries that matter most — including rare queries, where reduction is most damaging.

A common mistake is reducing dimension to fit a memory budget and accepting the loss because aggregate benchmark recall is "only down 2%." That 2% can concentrate on the long tail of queries where retrieval quality matters most.

6. Filters, updates, and operational reality

The comparison table in §3 lists "filter interaction" and "update/delete" as columns because in production they are often the binding constraints, not raw recall or latency. Two failure modes deserve their own treatment.

Filter interaction

The naive flow is: ANN index returns top-k candidates → metadata filter is applied to those candidates → result is the filtered subset of top-k. With a selective filter (say, 1% of the corpus matches), this collapses recall: an HNSW search returning 40 candidates yields about 0.4 matching rows on average. pgvector documents this directly, recommending raising hnsw.ef_search, using iterative scans, or pre-filtering with an exact scan when the predicate is selective (pgvector README).

Three engineering responses:

Pre-filter (scan only matching rows) — works when the predicate is selective enough that exact scan over the matching subset is cheap. Postgres B-tree + Flat is often the right answer here.
Index-aware filtering — some engines (Vespa, Weaviate, Qdrant, Milvus) integrate predicates into the graph traversal or partition probing, so the filter biases candidate generation rather than truncating it after the fact. This is the main feature differentiator among managed vector DBs.
Candidate inflation — raise efSearch or nprobe until the filtered top-k is large enough. This is a recall band-aid, not a fix; latency rises faster than recall recovers.

A 2026 study of filtered vector search in PostgreSQL systems finds that filtered-search behavior is dominated by system-level overhead (planner choices, predicate pushdown, locking) rather than by the index distance computations themselves, which reinforces that this is an operational problem as much as an algorithmic one.

Updates, deletes, and rebuild cadence

The four index families differ sharply on mutability:

Flat: trivial. Append a row; delete a row; nothing to rebuild.
IVF / IVF-PQ: appending is cheap (assign to nearest centroid). Deletes are usually tombstones. Codebook drift on distribution shift requires retraining the centroids and PQ codebooks, which is a periodic offline operation.
HNSW: insert is online but cumulatively degrades the small-world property as the graph drifts from its construction-time state. Most implementations cannot delete in place; they tombstone slots, and a full rebuild is required when tombstones exceed some fraction. High-throughput streaming insert is a known weak point.
DiskANN: original work emphasized batch builds; streaming-insert variants exist but are implementation-specific.

For workloads with frequent inserts, deletes, or updates, HAKES and LSM-VEC explicitly identify the dynamic-large-scale regime as a place where HNSW-style graphs become the operational bottleneck and propose alternative architectures (HAKES, 2025; LSM-VEC, 2025). The right rule is: model the rebuild cadence at peak insert rate before committing to an HNSW-only design, and budget for it.

Memory accounting

Memory estimates that say "$N \cdot d \cdot 4$ bytes" are wrong by a factor of 2–4× in production. The full accounting includes:

The vectors themselves at the chosen precision.
Graph edges (HNSW: $\approx M \cdot 2 \cdot 4$ bytes/vector at default $M = 16$).
Vector IDs and metadata (typically 8–24 bytes/vector).
Tombstones for deleted rows (small but non-zero).
Allocator overhead (10–20% on glibc malloc; less on jemalloc).
Replicas, WAL, and on-disk page cache.
Reranking buffers (full-precision copies if the primary index is quantized).

For a 100M × 1536 corpus on HNSW with default $M$ at FP32, the naive estimate is 614 GB; the realistic estimate including graph, IDs, allocator overhead, and a single replica is closer to 1.4–1.6 TB. The authoritative number is measured RSS under load, not the formula.

7. When pgvector is enough; when to leave

The "leave Postgres at $N$ vectors" framing is the wrong shape. There is no clean threshold by corpus size. pgvector already supports exact search, HNSW, IVFFlat, cosine/IP/L2/L1 operators, half vectors, binary quantization, sparse vectors, and reasonable filtering patterns (pgvector README). At small-to-medium scale on adequate hardware, the operational simplicity of running one database — joins, ACID transactions, backups, tooling, observability, single deployment surface — is a real advantage that is structurally unavailable to a separate vector engine.

The right framing is that vector search becomes an independent scaling problem when one or more of the following becomes the binding constraint:

Pressure	Symptom	When pgvector usually still works	When it usually does not
Memory	HNSW index plus working set exceeds RAM	Index < 50% of RAM after fragmentation	Multi-billion-row corpora, or modest corpora at high replica counts
Filtered recall	Selective predicates collapse ANN recall	Filters are non-selective, or selective filters are exact-scan-cheap	Tenant isolation with selective per-tenant filters at high QPS
Latency SLO	p95/p99 misses target under realistic load	p95 < 50 ms is acceptable, batch sizes are small	Sub-10ms p99 with heavy filtering and large $k$
Update rate	HNSW rebuild window can't keep up	Daily/weekly rebuilds are tolerable	High-throughput streaming insert with strict freshness SLO
Horizontal scale	Single-node CPU/RAM/IOPS exhausted	Vertical scaling has headroom	True multi-node distributed search required
Operational	Backup, HA, tenancy, or observability becomes hard	Postgres tooling covers it	Vector-specific observability, sharding, or tenancy is needed

Two non-binary alternatives worth surfacing before migration:

Compose, don't replace. Keep Postgres as the source of truth and run FAISS or ScaNN in-process for read-mostly search. This trades replication complexity for not running a second database.
Split candidate generation and reranking. Approximate index (anywhere) for top-N candidate retrieval; exact rerank against the full-precision Postgres rows. This makes the vector engine's recall budget more forgiving.

Migration to a dedicated vector DB adds another distributed system, consistency boundary, backup story, deployment path, vendor (or open-source operational) story, and cost model. It is warranted when vector search has independent scale or SLO requirements that pgvector cannot meet — measured against a real workload — not merely when a table contains embeddings. See Postgres vs. Dedicated Vector Databases for the operational checklist.

A residual point that often gets ignored: vendor and managed-DB documentation tends to highlight the architecture's strengths and not its weaknesses. Pair vendor docs with primary papers and independent benchmarks (ANN-Benchmarks, Big ANN, your own measured workload) before committing to a migration.

8. A practical evaluation protocol

The cheapest decisive experiment for any vector index decision is the same shape:

Build a representative slice. A 1–5% sample of the production corpus, plus a frozen query set with relevance labels (human or production-derived).
Run Flat as ground truth. Exact nearest neighbors on the slice with the model's native metric. This is the recall denominator.
Compare candidates. Flat (baseline), HNSW with default parameters, IVFFlat, IVF-PQ or scalar/binary quantized + rerank, pgvector HNSW/IVFFlat if Postgres is on the table, and one frontier compressor (RaBitQ, SAQ, or QINCo2) if memory matters.
Measure honestly.
Recall@k against Flat (10, 50, 100).
End-to-end task quality (nDCG, MRR, or RAG answer quality if downstream).
Latency at p50, p95, p99 under target QPS.
RAM, index size on disk, build time.
Insert/update cost at production rate.
Filter behavior at the selectivities the production workload actually has.
Vary the right knobs. HNSW: M and efSearch. IVF: lists and nprobe. PQ: bits per sub-vector, sub-vector count, oversampling, rerank depth.

The decisive migration test for "leave Postgres" is: can pgvector meet p95/p99 latency and recall under realistic filters and concurrent writes with operational headroom? If yes, a dedicated vector DB is extra infrastructure without justifying evidence. If no, evaluate a candidate engine on the exact failure mode — not on vendor benchmark curves.

The protocol is unsexy and adds days to the timeline. It also tends to overturn the conclusion that came from reading vendor blogs. Most teams that "knew" they needed a vector DB and ran this protocol found that pgvector with appropriate tuning, or HNSW + reranking, was sufficient — and most teams that "knew" pgvector was fine and ran the protocol found a specific filter selectivity or update rate where it fell over. The point is not the destination; it is that the destination becomes legible.

9. Decision rules

Distilled to load-bearing rules, in priority order:

The metric is a property of the embedding model, not a hyperparameter. Use what the model was trained for. If the model produces unit-norm vectors, cosine, dot, and L2 give equivalent rankings. If it does not, the vector norm probably encodes signal — measure before normalizing.
Flat is the correctness baseline, always. Even if production runs on HNSW, the evaluation slice should run Flat. You cannot tune what you cannot compare to ground truth.
Start with Flat. Add ANN only when measurement requires it. "Measurement requires it" means a measured latency, memory, or cost target, not an intuition about scale.
HNSW is a strong default, not a universal one. Strong on read-heavy in-memory workloads with non-selective filters, modest update rates, and adequate RAM. Weak on selective filters, streaming updates, and billion-scale RAM-constrained corpora.
IVF-PQ when memory is the binding constraint at moderate-to-large scale. Always with reranking against full-precision vectors stored separately.
DiskANN when the index does not fit in RAM. Verify implementation availability in your stack before committing.
ScaNN when you have MIPS workloads and the runtime is available. The anisotropic quantization insight is real; the implementation surface is narrower than HNSW's.
Quantization is a recall problem, not a storage problem. Measure recall against Flat with realistic queries, including the long tail. Reconstruction error is not a substitute.
Dimensionality reduction is high-risk except for Matryoshka-trained models. Prefer storage-side compression (FP16, PQ, binary) over dimension-side reduction.
Stay with Postgres+pgvector while it meets recall, latency, filtering, update, and operational requirements. Leave when a measured bottleneck is vector-specific and a dedicated system fixes that bottleneck enough to justify another datastore. Migration by row count is a category error.

The article that this wiki entry would not write — the one with a clean "use HNSW" verdict and a Postgres-vs-vector-DB threshold by row count — is shorter and more confident. It is also wrong on enough workloads to be operationally dangerous. The taxonomy of indexes is real and useful; the right answer for a given system is a function of constraints that the taxonomy does not see.

Companion entries

Core theory: - Embedding Geometry and the Cosine-vs-Dot Decision - Approximate Nearest Neighbor Algorithms: A Family Comparison - Product Quantization and Anisotropic Quantization - Matryoshka Representation Learning

Practice: - Postgres vs. Dedicated Vector Databases - pgvector Tuning: HNSW, IVFFlat, and Filtered Search - Building a Retrieval Evaluation Harness with Flat Baselines - Reranking Architectures: Candidate Generation and Exact Rerank - Memory Accounting for Vector Indexes in Production

Adjacent systems: - Hybrid Lexical–Vector Search - Filtered ANN: Pre-filter, Post-filter, and Index-Aware Strategies - Streaming Insert and Rebuild Cadence in HNSW Systems - Billion-Scale ANN: DiskANN and Big ANN Benchmark Lessons

Counterarguments: - When Brute-Force Search Beats ANN - The Limits of ANN Recall as a Quality Metric - Vector Database Migration as a Source of Operational Debt

Frontier: - 2025–2026 Quantization Literature: RaBitQ, SAQ, NVQ, QINCo2, TurboQuant - Dynamic Large-Scale Vector Indexes: HAKES, LSM-VEC, and Successors