Every RAG demo looks the same: a clean question, a perfect chunk of context, a confident answer. Then it hits real documents — inconsistent formatting, contradicting sources, ambiguous queries — and accuracy collapses. The gap between "it works in the notebook" and "it works for 10,000 employees" is where most enterprise RAG budgets go to die.
Over the last two years building RAG systems for enterprise clients at Botmartz, we've seen the same four failure modes repeat across industries — support, legal, healthcare, logistics. This post walks through each one, the fix, and runnable code so you can check your own system against it.
1. Retrieval Quality, Not Model Quality, Is the Bottleneck
Teams spend weeks tuning prompts and swapping models when the real problem sits upstream: the retriever is returning the wrong chunks. A better LLM cannot reason over context it never received. Before touching the model, we measure retrieval in isolation — precision and recall against a hand-labeled eval set of 100–200 real queries.
def precision_at_k(retrieved, relevant, k=5):
top_k = retrieved[:k]
hits = sum(1 for doc in top_k if doc.id in relevant)
return hits / k
# Run against a hand-labeled eval set before touching the model
scores = [precision_at_k(r, rel) for r, rel in eval_pairs]
print(f"Mean Precision@5: {sum(scores) / len(scores):.2f}")2. Chunking Strategy Has to Match Document Structure
Fixed-size chunking is the default and, for most enterprise documents, the wrong choice. Contracts, policy manuals and technical specs carry structure — sections, clauses, tables — that naive splitting destroys. We chunk along document structure first, then apply size limits as a constraint, not a rule.
3. Hybrid Search Is Mandatory, Not Optional
Standard vector search excels at matching conceptual queries, but fails completely on specific terms, SKU codes, and version numbers. We implement a hybrid search setup, matching semantic vectors from dense models together with BM25 keyword indices, merged via Reciprocal Rank Fusion (RRF).