Vector search has a dimension problem nobody wants to say out loud: the embedding models with the best retrieval quality also produce the biggest vectors, and the biggest vectors are the most expensive thing to index. A 3072-dimensional embedding scores better on every retrieval benchmark than its 768-dimensional cousin — and it also takes four times the RAM to hold in an approximate-nearest-neighbor index, and four times the compute to search it at scale. Most teams pick a lane: fast-but-shallow, or accurate-but-expensive. There's a third option that isn't a compromise between the two — it's both, in sequence, using vectors you already computed.
The pattern is sometimes called matryoshka retrieval, after the nesting dolls, and it works because of a training technique called Matryoshka Representation Learning: an embedding model trained so that the first N dimensions of a longer vector are themselves a usable, if coarser, embedding. That property turns a single expensive embedding call into two retrieval stages — a cheap, indexed pass over the short prefix to find plausible candidates, and an exact, unindexed pass over the full vector to rank only those candidates correctly. No second embedding model, no second index, and — this is the part worth dwelling on — no second full-collection scan.
§ 01Why full-dimension ANN gets expensive before it gets slow
Approximate nearest neighbor search — HNSW is the default almost everywhere now — trades exactness for speed by building a navigable graph over your vectors instead of comparing a query against every point. That graph has to live somewhere fast: RAM, ideally, or you lose most of the benefit. And the graph's cost scales with vector width. A million 3072-dimensional float32 vectors is roughly 12GB just for the raw vectors, before you count HNSW's graph overhead, which typically runs another 1.5-2x on top. That's before quantization, before replication, before you've indexed a second field.
The instinctive fix is to shrink the vector — use a smaller embedding model, or quantize aggressively. Both work, and both cost you retrieval quality: a genuinely smaller model has less capacity to encode nuance, and aggressive quantization (scalar int8, or binary 1-bit-per-dimension) throws away precision uniformly across the vector, with no way to know in advance which dimensions mattered for a given query. What you actually want is a way to search cheaply most of the time, and pay for full precision only on the small set of candidates where it counts. That's a two-stage retrieval problem, and it's been solved for full-text search for years — coarse filter, then rerank. Matryoshka embeddings let you do the same thing to dense vectors, using one model and one collection.
§ 02What Matryoshka training actually changes
An embedding model trained normally optimizes a single loss at its full output width — say, 3072 dimensions — and has no reason to organize information within that vector in any particular order. Truncate it to the first 768 dimensions and you get something close to random noise for retrieval purposes, because nothing forced the model to front-load meaning into an arbitrary prefix.
Matryoshka Representation Learning changes the training objective, not the architecture: the same loss is computed and summed across several nested prefix lengths — commonly something like 64, 128, 256, 512, 768, and the full width — so the model is penalized if the early dimensions alone don't already carry a usable signal. The effect is a coarse-to-fine encoding, like a JPEG that renders blurry-but-recognizable before the rest of the file arrives. Published results on this training method show a striking retention curve: truncating to roughly 8% of a Matryoshka model's full width can retain well over 95% of full-dimension retrieval accuracy, a gap that would be catastrophic on a conventionally trained embedding.
Several production embedding APIs support this natively now — you request the model's largest output dimensionality once, and the prefix is a valid smaller embedding for free, no separate call, no separate model to keep in sync. That's the detail that makes the two-stage pattern nearly free to adopt: you're not choosing between a big model and a small model at ingestion time. You compute one vector per document and get two usable representations out of it.
// one ingested chunk, two retrieval-relevant slices of the same vector { id: "chunk_004f2a", vector: { dense_768: embedding.slice(0, 768), // HNSW-indexed, lives in RAM dense_3072: embedding, // full width, m=0, on_disk=true sparse: sparseEncoder(text), // optional — see §05 }, }
§ 03The two-stage query: prefetch cheap, rerank exact
Say you have 50,000 documents, chunked into maybe 400,000 passages, each with a 3072-dim embedding. The naive approach indexes all 400,000 at full width. The two-stage approach indexes only the 768-dim prefix for ANN search, and stores the full 3072-dim vector in a second named vector on the same point — configured with no HNSW graph (m=0) and marked on_disk=true. That configuration matters more than it looks: an m=0 vector isn't a smaller index, it's no index at all. Qdrant (and the same pattern exists in most modern vector databases via their multi-vector query APIs) will happily store it and fetch it by point ID, but it will never build a graph over it or let you ANN-search it directly. It's a cold, disk-resident lookup table keyed by the same IDs your prefetch stage already returns.
The query then runs in two passes against the one collection, in one round trip:
results = client.query_points( collection_name="docs", prefetch=Prefetch( query=query_vector[:768], // cheap ANN pass, ~50 candidates using="dense_768", limit=50, // oversample — see §04 ), query=query_vector, // full 3072-dim, exact score using="dense_3072", // ID lookup, no graph traversal limit=10, )
Stage one is a normal HNSW search over a small, RAM-resident index — fast, and cheap enough to run generously. Stage two takes the ~50 candidate IDs stage one returned, fetches their full-width vectors from disk, computes exact cosine similarity against the query, and re-sorts. Fifty disk reads and fifty dot products is nothing; it's the equivalent cost of reranking a single page of search results, because that's exactly what it is.
A 768-dimensional prefix isn't a worse embedding — it's the same embedding, asked a coarser question.
— on why truncation isn't degradation
§ 04Why not just build two collections?
The obvious alternative — a fast collection at 768 dims and a separate, fully-indexed collection at 3072 dims — works, but it duplicates storage, duplicates writes, and adds a synchronization problem: every ingest, update, and delete now has to hit two places atomically, or your two collections drift. The single-collection, dual-vector approach sidesteps that entirely, in the same spirit as reaching for the infrastructure you already have instead of standing up a second system to solve a problem the first one can handle with a config change. One write path, one point ID space, one place that can go stale.
The other lever worth naming is the prefetch limit — how many candidates the cheap stage hands to the exact stage. Too low, and you risk the coarse pass missing a genuinely relevant document that the full-dimension vector would have ranked highly; the rerank stage can only promote candidates it's given, it can't rescue ones that never made the shortlist. Too high, and you're paying disk reads and exact-score computation for candidates that were never going to make the final cut. A prefetch limit in the 25-100 range against a final limit of 10-20 is a reasonable starting point — oversample by roughly 3-5x, then measure. This is a recall/precision knob you tune empirically against your own corpus and query distribution, not a constant you copy from a blog post.
Reranking is cheap when you only rerank fifty candidates. It stops being cheap the moment you rerank the whole collection.
§ 05Adding a second signal: hybrid fusion with RRF
Dense vectors are good at "this passage is about the same thing," and bad at exact terms — model numbers, statute citations, product SKUs, anything where a near-miss is a wrong answer. The standard fix is to run a sparse, keyword-aware retriever alongside the dense one — classic BM25-style term weighting, or a learned sparse encoder — and fuse the two ranked lists. The fusion method that's become the default because it requires no tuning is Reciprocal Rank Fusion: score each document by 1 / (k + rank) in each list, summed across lists, where k is a small constant (60 is the common default) that keeps a single very-high rank in one list from steamrolling everything else. RRF only looks at rank position, never at raw similarity scores — which matters, because a cosine score from a dense model and a BM25 score from a sparse one aren't on comparable scales and shouldn't be added together directly.
results = client.query_points( collection_name="docs", prefetch=[ Prefetch(query=query_vector[:768], using="dense_768", limit=25), Prefetch(query=sparse_query, using="sparse", limit=25), ], query=FusionQuery(fusion="rrf"), limit=10, )
The two patterns compose cleanly — hybrid prefetch narrows the candidate set with dense-plus-sparse signal, and the matryoshka rerank still runs as a separate exact pass over whatever the fusion returns. Most vector databases don't ship that exact three-stage combination as a single built-in query yet, so in practice it's two sequential calls: fuse to get a shortlist, then rerank that shortlist against the full-width vectors. It's still one collection and one write path — the added cost is a second round trip, not a second index.
§ 06Where a cross-encoder still earns its cost, and where this pattern doesn't fit
Everything above reranks with the same representation family, just at full width instead of truncated — it's still a bi-encoder comparing a query vector to a document vector independently. A cross-encoder, which scores a query and a document together in a single forward pass, catches relevance signal neither embedding sees alone — but it's slow enough that it's only viable on a very short list, typically the top 10-20 candidates after the matryoshka rerank has already done the cheap filtering. Bolting a cross-encoder directly onto raw ANN output — thousands of candidates — is the mistake this whole staircase exists to avoid; bolting it onto the output of a two-stage retrieval that's already down to a handful of strong candidates is close to free, and often the single highest-leverage step in the whole pipeline for precision-sensitive use cases.
None of this is free of judgment calls, and it's worth being honest about where the pattern doesn't pay for itself. If your collection is small enough that a full-dimension HNSW index fits comfortably in memory — tens of thousands of vectors, not tens of millions — the two-stage complexity buys you very little; just index at full width and skip the prefetch entirely. And if your embedding model wasn't trained with a Matryoshka objective, truncating it isn't a shortcut, it's just throwing away information — you'd need to either switch models or fall back to a real dimensionality-reduction step (PCA, or a distilled smaller model) trained specifically to approximate the big one, which is a different and more expensive project. The pattern earns its keep specifically at the point where your corpus has outgrown comfortable full-dimension indexing, your embedding model already supports nested prefixes, and you'd rather tune one collection's query shape than operate two. As with any recall/precision system, the numbers that matter — hit rate at k, latency per stage, disk I/O on the rerank pool — are worth instrumenting from day one, not inferred from a benchmark someone else ran on a different corpus. And when you're defining the vector config itself — dimensions, distance metric, quantization, which fields get payload indexes — treat it with the same rigor you'd give any other schema that the rest of your system depends on: written down once, validated at write time, not re-guessed at every call site.
— End of essay. Designing a retrieval layer that has to scale past a demo? Start a project →