The slides The tool is lesson 5 of LLMOps for AI Agents, in thirteen slides. The on-call assistant reads metrics, pods and logs with three read-only tools and opens a ticket with one write tool, and these slides follow how it calls them, how its runs are orchestrated, and how a run survives a crash: What is an AI …
Read MoreThe slides The tool is lesson 8 of LLMOps for AI Agents, in thirteen slides: What is an LLM gateway?: three failures of services that call a provider by themselves, and what a gateway does about each. The on-call assistant in production: the services in the cluster, the model providers outside it. The LLM gateway: one …
Read MoreThe slides The tool is lesson 6 of LLMOps for AI Agents, in fifteen slides: What is an eval?: one prompt change shipped after three questions tried by hand, and the same change tested on fifty saved questions. Grading an answer instead of comparing it: one question, two right answers, and an exact-match assert failing …
Read MoreThe slides The tool is lesson 2 of LLMOps for AI Agents, in twelve slides: What is LLMOps?: three surprises from a model another company runs, and what the lesson does about each. The MLOps loop: data, training, evaluation, deployment, monitoring and retraining, with the model ours. The levers of LLMOps: model choice, …
Read MoreIn this lab you send one request to a language model and read what comes back. First you set up a small model on your own computer, and your copy of the lab repository: the other labs use both. Then you send one request and read the answer, the tokens the model counted, and what the call would cost. Last you send the …
Read MoreIn this lab you change the assistant's prompt the way code is changed. First you run the on-call assistant, a system prompt and a 20-line Python script. Then you run its eval: four saved questions, each with a check its answer must pass. Last you delete one sentence of the prompt in a pull request, and watch the check …
Read MoreIn this lab you give the model the team's runbooks before it answers. First you read a runbook, and find a fact in it that no model could know. Then you ask the model alone, and it answers with general advice. Last you ask with retrieval: the script finds the nearest runbook sections and sends them with the question. …
Read MoreIn this lab you run an agent, and then attack it through the data it reads. First you read the agent's tools: two that read the cluster, and one that opens a ticket after you approve it. Then you run the agent, and approve the ticket it asks for. Last you plant an instruction in a pod's log, run the agent again, and …
Read MoreLLMOps for AI Agents LLMOps is the work of running applications built on large language models in production. It is the DevOps of an AI agent: versioning, testing, monitoring, deploying and securing it. This chapter of the DevOps course is ten academic hours, in nine lessons and four labs. Each lesson is a slide deck. …
Read MoreThe slides The tool is lesson 1 of LLMOps for AI Agents, in sixteen slides: What is an LLM?: what a team that runs an LLM meets in production, and the mechanism that causes it. Artificial intelligence, machine learning and language models: four fields, each inside the one before. A neuron: a weighted sum and an …
Read More