LLMOps Lab 3: Answers from Our Runbooks

In this lab you give the model the team's runbooks before it answers.

  • First you read a runbook, and find a fact in it that no model could know.
  • Then you ask the model alone, and it answers with general advice.
  • Last you ask with retrieval: the script finds the nearest runbook sections and sends them with the question.

It goes with the lesson RAG Data Pipelines. It needs the setup of Lab 1.

Read a runbook

  1. Go into the lab's directory, and list the runbooks:

    1$ cd lab3-runbooks
    2$ ls runbooks/
    3certificate-expiry.md  crashloop.md  kafka-consumer-lag.md  node-disk-full.md  payments-api-high-latency.md
    
  2. Read the one for a slow payments-api:

    1$ cat runbooks/payments-api-high-latency.md
    
    • The first check is the team's own: the database connection pool, and the exact command to read it.
    • Only this team knows it, so no model learned it in training.

Ask the model alone

  1. Ask the question without the runbooks:

    1$ python3 ask.py "payments-api is slow. What do I check first?"
    2First, check the response times and error rates in your monitoring tools ... Then, review the logs ...
    
    • The answer is reasonable advice for any service.
    • It is not the team's procedure, and it does not mention the connection pool.

Ask with the runbooks

  1. Ask the same question with --rag:

    1$ python3 ask.py --rag "payments-api is slow. What do I check first?"
    2Retrieved sections:
    3  0.812  payments-api-high-latency.md: payments-api: high latency > Check
    4  0.774  payments-api-high-latency.md: payments-api: high latency > When this applies
    5
    6Based on the information provided in the `payments-api-high-latency.md` runbook, you should first check the
    7database connection pool by running the command:
    8kubectl -n payments exec deploy/payments-api -- curl -s localhost:8080/metrics | grep db_pool_in_use
    
  2. Read what the script did, in ask.py:

    • chunks() cuts every runbook at its ## headings: each section is one chunk, with its runbook's title.
    • search() asks the embedding model for a vector of each chunk and of the question.
      • The number beside each retrieved section is how close its vector is to the question's, from 0 to 1.
      • The two closest sections are sent.
    • The prompt tells the model to answer only from those sections, and to name the file of each step.
  3. Ask about another runbook, and look at the sections the search picks:

    1$ python3 ask.py --rag "The payments worker is far behind on Kafka. Should I add workers?"
    

Posts in this series