When building a multi-tenant RAG system, isolating vector stores per tenant is necessary but not sufficient. Hybrid pipelines that combine dense vector search with sparse BM25 keyword search introduce a second isolation requirement that is easy to miss when self-hosting: per-tenant BM25 index isolation.
This is a lessons-learned account of how we hit this problem in production with ZettaBrain Teams — an open-source, self-hosted multi-tenant RAG server — and how we fixed it. Managed RAG platforms handle this at the platform level; if you are self-hosting, you own this problem yourself.
We were confident the document boundaries were solid. Each team gets its own ChromaDB collection in a separate directory. Vector search is cleanly scoped. We tested it, demos worked, everything looked good.
Then we added hybrid retrieval — BM25 (keyword search) alongside dense vector retrieval, reranked with FlashRank. The results were noticeably better for technical queries.
But when a user on the Finance team searched for "cardiac monitoring protocol", they got back a document from the Health team's corpus. The boundary had leaked.
BM25 is not just a keyword counter. It is a probabilistic ranking function whose scores depend on two corpus-level statistics:
Both statistics are computed over all documents in the index at build time. If you build a single BM25 index over every tenant's documents, IDF is contaminated: a medical term that appears rarely in Finance documents but frequently in Health documents will receive an artificially inflated IDF score when retrieved by a Finance query — because the index doesn't know that most of those occurrences "belong to someone else."
This is not a retrieval bug. It is a statistical property of BM25 itself. The only correct fix is a separate BM25 index per tenant.
Our initial implementation stored a single bm25_index.pkl in the shared ChromaDB directory and loaded it for every query regardless of which team was asking:
BM25_PATH = CHROMA_DIR / "bm25_index.pkl" # shared — wrong
def rebuild_bm25_index(vectorstore):
docs = vectorstore.get()["documents"] # all tenants' docs
tokenised = [d.lower().split() for d in docs]
BM25_PATH.write_bytes(pickle.dumps(BM25Okapi(tokenised)))When Finance team ingested its payroll documents, the BM25 index was rebuilt over Finance + Health documents combined. When a Finance query arrived, the BM25 scores were computed against this contaminated corpus.
Managed RAG platforms handle tenant isolation as a platform-level concern — if you use one of those services, BM25 scoping is likely taken care of for you.
If you are self-hosting — running LangChain, LlamaIndex, RAGFlow, or your own stack on your own infrastructure — you own this problem. The default BM25Retriever.from_documents() call in every major framework operates on whatever corpus you hand it. No framework will stop you from handing it all tenants' documents at once. That decision, and its consequences, are yours.
The solution is straightforward once you see it: store and build one BM25 index per team, in that team's own ChromaDB directory, using only that team's documents.
def _team_bm25_path(team_slug: str) -> Path:
return CHROMA_DIR / team_slug / "bm25_index.pkl"
def _rebuild_team_bm25(vectorstore, team_slug: str):
result = vectorstore.get()
docs = result.get("documents", [])
metas = result.get("metadatas", [])
if not docs:
_team_bm25_path(team_slug).unlink(missing_ok=True)
return
tokenised = [d.lower().split() for d in docs]
index = BM25Okapi(tokenised)
data = {"index": index, "docs": docs, "metas": metas}
_team_bm25_path(team_slug).write_bytes(pickle.dumps(data))
def _bm25_search_team(query: str, team_slug: str, k: int = 12):
p = _team_bm25_path(team_slug)
if not p.exists():
return []
data = pickle.loads(p.read_bytes())
index, docs, metas = data["index"], data["docs"], data["metas"]
tokens = query.lower().split()
scores = index.get_scores(tokens)
top_k = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]
return [(docs[i], metas[i], float(scores[i])) for i in top_k if scores[i] > 0]Each team's BM25 index is built exclusively from that team's documents, so IDF scores reflect only that team's corpus. The statistical boundary is now correct.
The full hybrid retrieval function merges dense (MMR) and sparse (BM25) results, deduplicates by content hash, and reranks with FlashRank:
def _hybrid_retrieve(question, vectorstore, team_slug, top_k=5):
# Dense: MMR over team's vector store
mmr_docs = vectorstore.max_marginal_relevance_search(
question, k=6, fetch_k=30, lambda_mult=0.82
)
# Sparse: per-team BM25
bm25_hits = _bm25_search_team(question, team_slug, k=10)
# Merge and deduplicate by MD5 hash of content
seen, candidates = set(), []
for doc in mmr_docs:
h = hashlib.md5(doc.page_content.encode()).hexdigest()
if h not in seen:
seen.add(h)
candidates.append(PassageObject(text=doc.page_content))
for text, meta, _ in bm25_hits:
h = hashlib.md5(text.encode()).hexdigest()
if h not in seen:
seen.add(h)
candidates.append(PassageObject(text=text))
# Rerank
ranked = flashrank_client.rerank(
RerankRequest(query=question, passages=candidates)
)
return [r["text"] for r in ranked[:top_k]]To verify isolation, we wrote a test that ingests two teams with completely separate documents and confirms each team only retrieves its own content:
def test_per_tenant_bm25_isolation():
# Setup: Finance team has payroll docs, Health team has clinical protocols
finance_vs = load_team_vectorstore("finance")
health_vs = load_team_vectorstore("health")
_rebuild_team_bm25(finance_vs, "finance")
_rebuild_team_bm25(health_vs, "health")
# Finance query — must not surface Health documents
finance_hits = _bm25_search_team("payroll processing schedule", "finance", k=5)
health_hits = _bm25_search_team("payroll processing schedule", "health", k=5)
finance_texts = [h[0] for h in finance_hits]
health_texts = [h[0] for h in health_hits]
# No overlap between the two result sets
assert not set(finance_texts) & set(health_texts), \
"BM25 isolation breach: shared content found across tenant indices"
def test_shared_index_would_fail():
# Demonstrate contamination with a single combined index
all_docs = finance_docs + health_docs
shared_bm25 = BM25Okapi([d.lower().split() for d in all_docs])
scores = shared_bm25.get_scores("cardiac monitoring protocol".lower().split())
top_idx = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:3]
# Top results include Health docs even when "queried" in Finance context
top_texts = [all_docs[i] for i in top_idx]
health_contamination = any(
"cardiac" in t or "monitoring" in t for t in top_texts[:3]
)
assert health_contamination, \
"Expected shared index to surface Health docs — proves isolation is needed"WRONG — single shared index
┌──────────────────────────────────────────┐
│ BM25 index (all tenants) │
│ finance_docs + health_docs combined │
│ IDF contaminated │
└──────────────────┬───────────────────────┘
│ query from Finance user
▼ returns Health documents ✗
RIGHT — per-team indices
┌──────────────────────┐ ┌──────────────────────┐
│ BM25: finance/ │ │ BM25: health/ │
│ bm25_index.pkl │ │ bm25_index.pkl │
│ (finance docs only) │ │ (health docs only) │
└──────────┬───────────┘ └──────────────────────┘
│ Finance query
▼ only Finance results ✓_rebuild_team_bm25 at the end of every ingestion run so the index stays in sync with the vector store.Multi-tenant, private-by-design document AI — self-hosted on your own infrastructure. No data leaves your environment.
Questions? Email us at hello@zettabrain.io