Multi-tenant RAG has a bug that never throws

Every open-source RAG starter kit ships the same shape: one password, one vector namespace, one corpus. You clone it, point it at a folder of PDFs, ask it a question, and it works — genuinely well, often on the first try. That demo is doing real work to convince you the hard part is retrieval quality. It isn't. The hard part shows up the day a second customer signs up and their documents land in the same index as the first one's.

Most teams catch the obvious failure mode immediately: without a filter, tenant A can read tenant B's chunks. They add a tenant_id column, wire it into the query filter, ship it, and call multi-tenancy solved. It isn't. There's a second failure mode that doesn't throw an error, doesn't show up in an access-control audit, and doesn't fail a single test — it just quietly changes which documents rank highest, for every tenant, forever. It lives in a place almost nobody thinks to look: the term-frequency statistics behind sparse retrieval.

§ 01What the single-tenant shell actually assumes

Strip a typical self-hosted RAG project down to its retrieval core and you find three assumptions baked in at every layer: there is one corpus, there is one set of users who can all see all of it, and search quality is measured against that one corpus as a whole. Brute-force cosine similarity over a flat list of vectors. One shared password gating the whole instance. One namespace, because there was never a reason to imagine a second one.

None of those assumptions are wrong for a single-tenant tool. They're wrong the moment you productize the same retrieval core as SaaS, and the failure isn't just "add auth." Access control, vector isolation, and term statistics are three separate problems that happen to look like one problem from the outside — because in the single-tenant shell, all three were trivially satisfied by having only one tenant. Solve access control and ship, and you've fixed the problem that would have gotten you sued. You haven't touched the one that will quietly degrade search for every customer you onboard after the first.

§ 02Where IDF actually lives

Dense retrieval — cosine similarity over embeddings — doesn't have a corpus-statistics problem in the same way. An embedding is computed from the text of one document; nothing about the rest of the collection feeds into it. Sparse and learned-sparse retrieval is different by design. BM25, the workhorse behind decades of keyword search, weighs a term by how rare it is across the whole corpus you're searching:

IDF(t) = log( N / df(t) ) — N is the total number of documents, df(t) is how many of them contain term t.

A term that appears in 3 documents out of 10,000 gets a high weight — it's discriminative, it's probably load-bearing for whatever query used it. A term that appears in 9,000 out of 10,000 gets pushed toward zero — it's everywhere, so it tells you almost nothing about which document the user actually wants. That's the entire mechanism BM25 uses to separate signal from noise, and it's a property of the collection, not of any single document in it.

Learned sparse retrievers built on the same idea — miniCOIL is one, and the class includes BM25-flavored successors more broadly — inherit this corpus-dependence directly: a sparse vector config with an IDF modifier tells the vector database to compute document frequency over whatever set of documents lives in that collection, then bake the resulting weight into every score. SPLADE-style models are a partial exception — they lean more on learned, contextual term weights than on a query-time IDF lookup — but even those are trained on a fixed corpus and carry that corpus's vocabulary bias with them; they just bury the dependency at training time instead of at query time. There is no version of corpus-derived term weighting that is tenant-agnostic by default. Someone has to decide what "the corpus" means.

§ 03The trap: partition by payload, dilute by design

Here's where it gets genuinely uncomfortable, because the trap isn't a rookie mistake — it's the recommended architecture. Vector database vendors, Qdrant included, actively steer you away from one collection per tenant: past a few hundred tenants it stops scaling, burns resources on redundant index structures, and hits hard collection-count ceilings on managed clusters. The documented fix is payload partitioning — one shared collection, a tenant field on every point, a payload index on that field, and a filter applied on every query. For dense HNSW search, that's simply correct, and there's a purpose-built indexing hint for exactly this shape.

Apply that same shared-collection pattern to a sparse vector with an IDF modifier and you've quietly changed what the statistics mean. The document-frequency term is computed over every point in the collection that has that sparse field — across every tenant, with no awareness that a tenant boundary exists. A term that's rare and highly diagnostic inside tenant A's corpus can be common and near-meaningless inside tenant B's — and the collection only ever sees one blended frequency for it. Onboard a tenant with a large, narrow-vocabulary corpus and you can measurably shift term weights for every other tenant sharing that collection, with zero errors logged, zero failed queries, and a support ticket three weeks later that just says "search got worse."

Concretely: imagine one tenant is a specialty medical-device manufacturer whose corpus is almost entirely regulatory filings, and another is a general customer-support knowledge base spanning dozens of unrelated product categories. The term "biocompatibility" is rare and load-bearing across the support tenant's documents — if it shows up at all, it's exactly the chunk a query should surface. Blend the two corpora into one collection and that same term's document frequency climbs, because the device maker's corpus is full of it. Its IDF weight drops. The support tenant's most discriminative term just got quietly demoted, and nothing in the system will ever tell you why.

§ 04What the shared-collection config actually looks like

The hazard is invisible in the code, which is exactly the problem — it reads like reasonable, idiomatic multi-tenant vector search:

# one shared collection, sparse IDF computed over ALL tenants
client.create_collection(
    collection_name="documents",
    vectors_config={"dense": VectorParams(size=768, distance="Cosine")},
    sparse_vectors_config={"sparse": SparseVectorParams(modifier="IDF")},
)

# tenant_id is enforced on every read...
hits = client.query_points(
    collection_name="documents",
    query=sparse_vec, using="sparse",
    query_filter=Filter(must=[FieldCondition(
        key="tenant_id", match=MatchValue(value=tid))]),
)
# ...but the df() behind "sparse" was never scoped to tenant_id at all
if hits[0].score < min_score: fall_back_to_dense()

The filter is real and it does its job — no tenant reads another tenant's rows. What it doesn't do, and can't do, is un-blend a document-frequency count that was computed before the filter was ever applied. The access-control layer and the statistics layer are enforced at completely different points in the pipeline, and only one of them respects the tenant boundary.

§ 05The isolation checklist

Multi-tenant RAG needs four separate kinds of isolation, and it's worth naming them separately because a team that solves one often assumes it solved all four:

• Statistics isolation. Anywhere a retrieval component derives a weight from "the corpus" — IDF, document frequency, any collection-level normalization — decide explicitly what corpus means per tenant. The honest options are: a separate collection (or shard) per tenant once one grows past the point where shared statistics stop being acceptable; a tiered approach where small tenants share a pool and large tenants graduate to their own index, mirroring how vector databases already recommend tiering hot tenants onto dedicated shards for performance; or dropping the IDF modifier for shared collections and leaning on dense retrieval plus rerank for cross-tenant fairness, accepting a real quality tradeoff instead of an invisible one.
• Vector isolation. Whatever topology you pick, every point needs an unambiguous tenant key, a payload index on that key, and — if the database supports it — a tenant-aware indexing hint so co-located tenant data doesn't force a full-collection scan on every query. This is table stakes and most teams get it right early.
• Access control at query time. The tenant filter belongs on the server side of the retrieval service, never trusted from a client-supplied parameter, and it should fail closed — a request with no resolvable tenant should return nothing, not "everything." Where you keep the source of truth for who owns what matters here too; a relational store with real row-level security, the kind of thing we've argued for keeping boring and centralized in Postgres is the answer, is still the right place to resolve identity before a query ever reaches the vector layer.
• Index sizing and cost. Per-tenant collections buy clean statistics and clean isolation, but they cost real memory and operational overhead per collection, and most vector databases have a hard ceiling on how many you can run per cluster. Shared collections buy efficiency at the cost of exactly the dilution described above. There's no topology that's free on every axis — pick deliberately, and revisit the choice as your largest tenant's corpus size stops looking like your median tenant's.

Vendor guidance on this is explicit: don't create one collection per tenant, partition by payload instead. That's correct for a dense HNSW index. It's also exactly the shape that lets one tenant's vocabulary quietly reweight another's — the two pieces of advice point in genuinely different directions once sparse retrieval enters the picture.

§ 06Dense retrieval isn't a free pass either

It's tempting to read all of this as an argument for skipping sparse retrieval and shipping dense-only. That trades one problem for a worse one on quality: dense embeddings alone are reliably weaker on exact terms — order IDs, product SKUs, model numbers, proper nouns — which is precisely the class of query where a support agent or a compliance reviewer needs the literal match, not the semantically-nearby one. We've written before about tuning the dense side of that tradeoff with two-stage matryoshka retrieval; none of that work makes the sparse-statistics question go away, because the two techniques solve different problems and a serious retrieval stack usually wants both.

The honest position is that hybrid retrieval is worth the complexity, and multi-tenancy is a design constraint you account for from the first collection you create, not a migration you run later. We see the same shape of decision on any multi-tenant SaaS surface with per-customer catalogs or content libraries — from course platforms to niche product catalogs like the recommendation and search work behind a multi-brand marketplace — anywhere search quality depends on a corpus boundary someone has to define on purpose. A single-tenant demo will never surface this bug, because a single-tenant demo only ever has one corpus. The moment you have two, "the corpus" stops being a given and starts being an architecture decision — and it's one worth making before the second tenant signs the contract, not after their search results start quietly getting worse.

— End of essay. Standing up multi-tenant RAG and want the isolation model reviewed before you scale past tenant one? Start a project →