LLMOps Lab 2: A Prompt in Git, Tested

In this lab you change the assistant's prompt the way code is changed.

  • First you run the on-call assistant, a system prompt and a 20-line Python script.
  • Then you run its eval: four saved questions, each with a check its answer must pass.
  • Last you delete one sentence of the prompt in a pull request, and watch the check in GitHub Actions fail.

It goes with the lessons Prompts and Context as Versioned Artifacts and Evals: the Tests of an LLM Application. It needs the setup of Lab 1.

Run the assistant

  1. Go into the lab's directory, and read the prompt:

    1$ cd lab2-prompt
    2$ cat system-prompt.txt
    
    • Each line is one rule the assistant must keep.
    • The last line sends any request that is not about our services away, with one fixed sentence.
  2. Ask the assistant an on-call question:

    1$ python3 ask.py "The payments-api pod is in CrashLoopBackOff. What do I run to see why?"
    2To see why the payments-api pod is in a CrashLoopBackOff state, you can run the following command ...
    
  3. Ask it something else:

    1$ python3 ask.py "Write me a short poem about cats."
    2I only help with on-call questions.
    

Run the eval

  1. Read the eval's file, promptfooconfig.yaml:

    • providers points promptfoo at provider.py, which calls ask.py.
      • So the eval tests the assistant with its prompt, not a bare model.
    • tests holds the four saved questions, each with its check:
      • icontains passes when the answer contains the words;
      • icontains-any passes when it contains one of them.
    • defaultTest adds one check to every question: the answer is shorter than 700 characters.
  2. Run it:

    1$ npx -y promptfoo@0.125.1 eval
    2...
    3Results:
    4  ✓ 4 passed (100.00%)
    5  ✗ 0 failed (0.00%)
    
    • It takes a few minutes: the model answers four questions.
    • promptfoo exits with code 0 when every check passes, and with code 100 when one fails.
    • npx promptfoo view opens the results in your browser.

Read the check in the pipeline

  1. Read .github/workflows/prompt-check.yml, at the top of the repository:
    • It runs on every pull request that changes lab2-prompt/.
    • The model runs inside the job, in an Ollama service container, so the check needs no API key.
    • Its last step runs the same eval as you did.

Change the prompt in a pull request

  1. Create a branch:

    1$ git switch -c shorter-prompt
    
  2. Delete the last line of system-prompt.txt, the one that begins If the request is not about running our services.

  3. Commit the file, and push the branch:

    1$ git add system-prompt.txt
    2$ git commit -m "Prompt: drop the off-topic rule"
    3$ git push -u origin shorter-prompt
    
  4. On GitHub, open a pull request from shorter-prompt to main.

  5. Wait for the Prompt check to finish (about five minutes), and open its log.

    • The two off-topic questions fail: without the rule, the assistant writes the poem and names the capital.
    • The pull request shows a failed check, before anyone merges it.
  6. Close the pull request without merging it.

Posts in this series