LLMOps Top
LLMOps for AI Agents
LLMOps is the work of running applications built on large language models in production.
- It is the DevOps of an AI agent: versioning, testing, monitoring, deploying and securing it.
This chapter of the DevOps course is ten academic hours, in nine lessons and four labs.
- Each lesson is a slide deck.
The lessons follow one example: the on-call assistant, an AI agent for a DevOps team.
- It answers questions from the team's runbooks.
- It reads metrics and pod status with read-only tools.
- It opens a ticket when a person should step in.
The chapter has four parts: each is one or more lessons, then a lab.
- The labs run a small model on your own computer.
- Their code is in YuvalShaul/llmops-labs, which you copy as a template.
Part 1: The model
Lessons
- Neural Networks and Transformers
- A very short introduction to the AI behind agents: neurons, training, tokens, attention and generation
- From MLOps to LLMOps
- Operating a model we did not train: choosing it, pinning its version, and paying for it per token
Lab
- Lab 1: One Call to a Model
- The setup, one request, its tokens and cost, and two temperatures
Part 2: Prompts as code
Lesson
- Prompts and Context as Versioned Artifacts
- The system prompt, the tool definitions and the context assembly kept in git and released like code
Lab
- Lab 2: A Prompt in Git, Tested
- An eval of four questions, and a prompt change caught in a pull request
Part 3: Answers from our documents
Lesson
- RAG Data Pipelines
- Ingestion, chunking, embedding and vector stores, run as a data pipeline
Lab
- Lab 3: Answers from Our Runbooks
- The model alone, then with the runbook sections a search found
Part 4: Agents in production
Lessons
- Agentic Workflows
- Tool use, orchestration, and durable execution that survives a crash
- Evals: the Tests of an LLM Application
- Datasets, graders and thresholds, offline in CI and online in production
- Observability for Agents
- Tracing agent runs, and measuring their tokens, cost and latency
- Deployment Patterns for LLM Applications
- Gateways, model routing, fallbacks, caching and rollouts
- Security and Governance for Agents
- Prompt injection, data leakage and guardrails
Lab
- Lab 4: A Small Agent, and an Attack on It
- Tools, an approval before a write, and an instruction planted in a log
Interactive views on the same subjects
- Simple Loop AI Agent
- The agent loop, with the JSON of every request to the model
- RAG AI Agent
- The agent searching the notes before every prompt
- Workflow AI Agent
- The model reads the message, and our code applies the policy
- Agent Monitoring: a Crew Observed
- A crew of agents observed with logs, traces and metrics