The data behind an agent that answers from our documents, in thirteen slides: retrieve then generate, ingestion and metadata, chunking by structure, fixed-size chunks with overlap, embedding, similarity search, a vector store in pgvector, the query path to a cited answer, hybrid search, re-indexing on every push, measuring recall@k, and the failure modes of a RAG pipeline.
The slides
The tool is lesson 4 of
LLMOps for AI Agents, in thirteen slides. The on-call
assistant answers questions from the team's runbooks, and these slides follow the runbooks from the git repository to
the model's context:
- What is RAG?: one question answered by the model alone, and by the model with the runbook's section in front of it.
- RAG: retrieve, then generate: the indexing pipeline and the query path, meeting in the vector store.
- Ingestion: loading and parsing the sources: four kinds of source, and a parsed document with its metadata.
- Chunking by structure: one chunk per section: a runbook cut at its headings.
- Fixed-size chunks with overlap: a 500-token window, and the trade-off of the chunk size.
- Embedding: a chunk becomes a vector: one model for the chunks and the questions.
- Similarity search: the nearest neighbours: the top 3 for a question, and approximate indexes.
- A vector store in Postgres: pgvector: the chunks table, its HNSW index and the filtered search.
- The query path: from question to cited answer: search, rerank, and the context with its sources.
- Hybrid search: keywords and vectors together: BM25 and vector rankings merged by reciprocal rank fusion.
- Keeping the index fresh: re-indexing on every push: a CI job that upserts by chunk id and deletes removed files.
- Measuring retrieval: recall@k: two index versions compared on six test questions.
- Failure modes of a RAG pipeline: a stale document, a wrong chunk, a permission leak.
The buttons on the left and right edges move between slides. So do the arrow keys and a swipe.
The site has more on the same subject:
The lab
The runbooks, the vectors, the positions of the chunks in the similarity drawing, the keyword and vector rankings,
the ranks in the recall table and the output of the indexing job are made up for the drawings. The fused scores of
the hybrid search and the recall figures are computed by the slides from those ranks.
Sources