LLMOps Top

LLMOps for AI Agents

  • LLMOps is the work of running applications built on large language models in production.

    • It is the DevOps of an AI agent: versioning, testing, monitoring, deploying and securing it.
  • This chapter of the DevOps course is ten academic hours, in nine lessons and four labs.

    • Each lesson is a slide deck.
  • The lessons follow one example: the on-call assistant, an AI agent for a DevOps team.

    • It answers questions from the team's runbooks.
    • It reads metrics and pod status with read-only tools.
    • It opens a ticket when a person should step in.
  • The chapter has four parts: each is one or more lessons, then a lab.

    • The labs run a small model on your own computer.
    • Their code is in YuvalShaul/llmops-labs, which you copy as a template.

Part 1: The model

Lessons

  1. Neural Networks and Transformers
    • A very short introduction to the AI behind agents: neurons, training, tokens, attention and generation
  2. From MLOps to LLMOps
    • Operating a model we did not train: choosing it, pinning its version, and paying for it per token

Lab


Part 2: Prompts as code

Lesson

  1. Prompts and Context as Versioned Artifacts
    • The system prompt, the tool definitions and the context assembly kept in git and released like code

Lab


Part 3: Answers from our documents

Lesson

  1. RAG Data Pipelines
    • Ingestion, chunking, embedding and vector stores, run as a data pipeline

Lab


Part 4: Agents in production

Lessons

  1. Agentic Workflows
    • Tool use, orchestration, and durable execution that survives a crash
  2. Evals: the Tests of an LLM Application
    • Datasets, graders and thresholds, offline in CI and online in production
  3. Observability for Agents
    • Tracing agent runs, and measuring their tokens, cost and latency
  4. Deployment Patterns for LLM Applications
    • Gateways, model routing, fallbacks, caching and rollouts
  5. Security and Governance for Agents
    • Prompt injection, data leakage and guardrails

Lab


Interactive views on the same subjects