LLMOps Lab 2: A Prompt in Git, Tested
In this lab you change the assistant's prompt the way code is changed.
- First you run the on-call assistant, a system prompt and a 20-line Python script.
- Then you run its eval: four saved questions, each with a check its answer must pass.
- Last you delete one sentence of the prompt in a pull request, and watch the check in GitHub Actions fail.
It goes with the lessons Prompts and Context as Versioned Artifacts and Evals: the Tests of an LLM Application. It needs the setup of Lab 1.
Run the assistant
Go into the lab's directory, and read the prompt:
1$ cd lab2-prompt 2$ cat system-prompt.txt- Each line is one rule the assistant must keep.
- The last line sends any request that is not about our services away, with one fixed sentence.
Ask the assistant an on-call question:
1$ python3 ask.py "The payments-api pod is in CrashLoopBackOff. What do I run to see why?" 2To see why the payments-api pod is in a CrashLoopBackOff state, you can run the following command ...Ask it something else:
1$ python3 ask.py "Write me a short poem about cats." 2I only help with on-call questions.
Run the eval
Read the eval's file,
promptfooconfig.yaml:providerspoints promptfoo atprovider.py, which callsask.py.- So the eval tests the assistant with its prompt, not a bare model.
testsholds the four saved questions, each with its check:icontainspasses when the answer contains the words;icontains-anypasses when it contains one of them.
defaultTestadds one check to every question: the answer is shorter than 700 characters.
Run it:
1$ npx -y promptfoo@0.125.1 eval 2... 3Results: 4 ✓ 4 passed (100.00%) 5 ✗ 0 failed (0.00%)- It takes a few minutes: the model answers four questions.
- promptfoo exits with code 0 when every check passes, and with code 100 when one fails.
npx promptfoo viewopens the results in your browser.
Read the check in the pipeline
- Read
.github/workflows/prompt-check.yml, at the top of the repository:- It runs on every pull request that changes
lab2-prompt/. - The model runs inside the job, in an Ollama service container, so the check needs no API key.
- Its last step runs the same eval as you did.
- It runs on every pull request that changes
Change the prompt in a pull request
Create a branch:
1$ git switch -c shorter-promptDelete the last line of
system-prompt.txt, the one that begins If the request is not about running our services.Commit the file, and push the branch:
1$ git add system-prompt.txt 2$ git commit -m "Prompt: drop the off-topic rule" 3$ git push -u origin shorter-promptOn GitHub, open a pull request from
shorter-prompttomain.Wait for the Prompt check to finish (about five minutes), and open its log.
- The two off-topic questions fail: without the rule, the assistant writes the poem and names the capital.
- The pull request shows a failed check, before anyone merges it.
Close the pull request without merging it.