LLMOps Lab 3: Answers from Our Runbooks
In this lab you give the model the team's runbooks before it answers.
- First you read a runbook, and find a fact in it that no model could know.
- Then you ask the model alone, and it answers with general advice.
- Last you ask with retrieval: the script finds the nearest runbook sections and sends them with the question.
It goes with the lesson RAG Data Pipelines. It needs the setup of Lab 1.
Read a runbook
Go into the lab's directory, and list the runbooks:
1$ cd lab3-runbooks 2$ ls runbooks/ 3certificate-expiry.md crashloop.md kafka-consumer-lag.md node-disk-full.md payments-api-high-latency.mdRead the one for a slow payments-api:
1$ cat runbooks/payments-api-high-latency.md- The first check is the team's own: the database connection pool, and the exact command to read it.
- Only this team knows it, so no model learned it in training.
Ask the model alone
Ask the question without the runbooks:
1$ python3 ask.py "payments-api is slow. What do I check first?" 2First, check the response times and error rates in your monitoring tools ... Then, review the logs ...- The answer is reasonable advice for any service.
- It is not the team's procedure, and it does not mention the connection pool.
Ask with the runbooks
Ask the same question with
--rag:1$ python3 ask.py --rag "payments-api is slow. What do I check first?" 2Retrieved sections: 3 0.812 payments-api-high-latency.md: payments-api: high latency > Check 4 0.774 payments-api-high-latency.md: payments-api: high latency > When this applies 5 6Based on the information provided in the `payments-api-high-latency.md` runbook, you should first check the 7database connection pool by running the command: 8kubectl -n payments exec deploy/payments-api -- curl -s localhost:8080/metrics | grep db_pool_in_useRead what the script did, in
ask.py:chunks()cuts every runbook at its##headings: each section is one chunk, with its runbook's title.search()asks the embedding model for a vector of each chunk and of the question.- The number beside each retrieved section is how close its vector is to the question's, from 0 to 1.
- The two closest sections are sent.
- The prompt tells the model to answer only from those sections, and to name the file of each step.
Ask about another runbook, and look at the sections the search picks:
1$ python3 ask.py --rag "The payments worker is far behind on Kafka. Should I add workers?"