RAG

RAG#

Retrieval-Augmented Generation puts a retrieval step in front of the model: pull the relevant documents from a corpus, concatenate them into the prompt, then generate. RAG is the pragmatic answer to “the model does not know about my data” without the cost of fine-tuning.

The pipeline#

  1. Ingest. Source documents (logs, reports, code, scraped sites) become text. Strip boilerplate; preserve structure (headings, tables) the model can use.

  2. Chunk. Split into retrievable units, typically 200-1000 tokens with some overlap. The chunk is what gets retrieved and what gets shown to the model; size for both.

  3. Embed. Each chunk becomes a vector via an embedding model. The same model is used at query time so the spaces match.

  4. Index. Vectors land in a vector store (FAISS, Qdrant, Weaviate, pgvector, Pinecone). Often paired with a keyword index (BM25 over the original text) for hybrid search.

  5. Retrieve. At query time: embed the query, fetch top-k nearest chunks, optionally re-rank with a cross-encoder.

  6. Generate. Pass the retrieved chunks as context, alongside the user query, to the LLM.

Operational concerns#

  • Chunk for the answer, not the document. The chunk shown to the model should be self-contained enough to answer questions drawn from it. Heading-aware chunking beats fixed-size on prose; AST-aware beats both on code.

  • Hybrid search outperforms pure vector on factoid and identifier queries (CVE numbers, IPs, names). Pair BM25 + dense retrieval and merge.

  • Re-rank. A small cross-encoder (e.g. bge-reranker) over the top 50 candidates beats taking the raw top 5.

  • Cite. Return the chunk IDs with the answer; show the analyst which chunks the model used. No citation, no trust.

  • Update strategy. Incremental indexing for streams; full rebuild for periodic snapshots; stale chunks compete with fresh ones for retrieval slots.

  • Cache embeddings. Embedding cost dominates ingest; cache by content hash so re-runs are free.

When RAG is the wrong answer#

  • Small corpora (< 50 documents). Just put it all in the prompt. Modern context windows handle it; prompt caching makes it cheap.

  • Live data. A tool call to the source system is more accurate and more current than a vector index.

  • Reasoning over the whole corpus. RAG retrieves; it does not reason globally. For “what changed across all 1000 reports”, reach for a real query engine, not vectors.

Stack choices#

  • Vector store. pgvector for “we already run Postgres”; qdrant / weaviate for purpose-built; faiss for local / single-process. Hosted: Pinecone, Turbopuffer.

  • Embedding model. Vendor-managed (Anthropic, OpenAI, Cohere) for default quality; sentence-transformers / bge-* for self-hosted.

  • Framework. llama-index and langchain are common; both are thick. Prefer plain ingest scripts + a thin retriever for anything beyond a prototype.

References#

  • NLP, the broader text-processing stack.

  • LLM-Assisted Analysis, LLM-as-analyst patterns that ride on retrieval.

  • Agents, when retrieval becomes a tool an agent calls.

Contents