Most 'RAG demos' fall apart the moment they meet real documents. Here is the stack I've had success with at scale.
Chunk with intent
Semantic chunking beats fixed-window every time. Split on structure — headings, sections, list boundaries — then attach breadcrumbs as metadata.
Hybrid retrieval
Dense embeddings for meaning, BM25 for exact matches. Fuse with reciprocal rank fusion. Rerank the top 50 with a cross-encoder to get the top 5 that actually matter.
results = rrf([dense.search(q, k=50), bm25.search(q, k=50)])
top = reranker.rerank(q, results)[:5]Ground the prompt
Cite passages inline, force the model to answer 'I don't know' when confidence is low, and log every retrieval for evaluation.
A RAG system without evaluation is a chatbot with a database.