The slides The tool is lesson 5 of LLMOps for AI Agents, in thirteen slides. The on-call assistant reads metrics, pods and logs with three read-only tools and opens a ticket with one write tool, and these slides follow how it calls them, how its runs are orchestrated, and how a run survives a crash: What is an AI …
Read MoreThe slides The tool is lesson 8 of LLMOps for AI Agents, in thirteen slides: What is an LLM gateway?: three failures of services that call a provider by themselves, and what a gateway does about each. The on-call assistant in production: the services in the cluster, the model providers outside it. The LLM gateway: one …
Read MoreThe slides The tool is lesson 6 of LLMOps for AI Agents, in fifteen slides: What is an eval?: one prompt change shipped after three questions tried by hand, and the same change tested on fifty saved questions. Grading an answer instead of comparing it: one question, two right answers, and an exact-match assert failing …
Read MoreThe slides The tool is lesson 2 of LLMOps for AI Agents, in twelve slides: What is LLMOps?: three surprises from a model another company runs, and what the lesson does about each. The MLOps loop: data, training, evaluation, deployment, monitoring and retraining, with the model ours. The levers of LLMOps: model choice, …
Read MoreThe slides The tool is lesson 1 of LLMOps for AI Agents, in sixteen slides: What is an LLM?: what a team that runs an LLM meets in production, and the mechanism that causes it. Artificial intelligence, machine learning and language models: four fields, each inside the one before. A neuron: a weighted sum and an …
Read MoreThe slides The tool is lesson 7 of LLMOps for AI Agents, in thirteen slides: What is a trace?: one complaint about a slow, wrong answer, looked into with ordinary monitoring and then with a trace. Observing an agent: one request, many calls: a web service's one path beside the on-call assistant's many calls. A trace of …
Read MoreThe slides The tool is lesson 3 of LLMOps for AI Agents, in twelve slides: What is a prompt?: one sentence of a prompt changed in the running service, and the same change made like a code change. The context window of one call: the five parts our code assembles, and where each comes from. Prompt engineering and context …
Read MoreThe slides The tool is lesson 4 of LLMOps for AI Agents, in thirteen slides. The on-call assistant answers questions from the team's runbooks, and these slides follow the runbooks from the git repository to the model's context: What is RAG?: one question answered by the model alone, and by the model with the runbook's …
Read MoreThe slides The tool is lesson 9 of LLMOps for AI Agents, in thirteen slides: How do we secure an AI agent?: one instruction hidden in a log line, read by an agent with no limits and by a limited one. The LLM as an attack surface: one context window, where instructions and data are the same tokens. Direct and indirect …
Read More