Lesson 6 of LLMOps for AI Agents, in fifteen slides: grading an answer instead of comparing it, the parts of an eval, the dataset, three kinds of grader, LLM-as-judge and its calibration, the pass rate by category, RAG metrics, agent metrics, a promptfoo config, the pull request gate in CI, the regression set, online evals on live traffic, the release gate for a model upgrade, and eval tools.
The slides
The tool is lesson 6 of
LLMOps for AI Agents, in fifteen slides:
- What is an eval?: one prompt change shipped after three questions tried by hand, and the same change tested on fifty saved questions.
- Grading an answer instead of comparing it: one question, two right answers, and an exact-match assert failing one of them.
- An eval: cases, runs, graders and a threshold: the four parts, and the change under test.
- The eval dataset: cases and expected facts: four cases of the on-call assistant, one per category.
- Three kinds of grader: code, a model and a person: what each judges and what each costs.
- LLM-as-judge: a model grading against a rubric: its input, its output, its biases, and its agreement with engineers.
- The pass rate by category: one run's scores against each category's threshold.
- RAG metrics: retrieval, faithfulness and relevance: one metric on each side of the question, the chunks and the answer.
- Agent metrics: task, tool calls, turns and cost: one agent run, turn by turn, and the metrics read from it.
- An eval suite in promptfoo's config file: our assistant's endpoint, a case, and its assertions.
- Offline evals in CI: the pull request gate: a pull request that changes the model, compared with the baseline and blocked.
- The regression set: production failures as cases: a thumbs-down that becomes a case.
- Online evals: grading live traffic: sampled judges, user feedback, guardrail metrics, shadow runs and A/B tests.
- The eval as the release gate for a model upgrade: two models compared, and the stages of the release.
- Eval tools: libraries and platforms: promptfoo, DeepEval, Ragas, LangSmith, Braintrust and Langfuse.
The buttons on the left and right edges move between slides. So do the arrow keys and a swipe.
The lab
- The scores are made up: the pass rates per category, the judge's agreement with the engineers (46 of 50), the
faithfulness and latency of the two models.
- The prices are examples: $3 per million input tokens and $15 per million output tokens; the cost of a run and
of the suite are computed from them.
evals/run.py and the cases.yaml format are our own, written for the slides; promptfoo's config follows its
documentation.
Sources