Operating a model we did not train, in twelve slides: the MLOps loop, the levers of LLMOps, a hosted API and a self-hosted open-weight model, model versions and aliases, a model upgrade as a release, the request and the response of an LLM API call, the price of a call in tokens, rate limits, latency, and the LLMOps stack of the on-call assistant.
The slides
The tool is lesson 2 of
LLMOps for AI Agents, in twelve slides:
- What is LLMOps?: three surprises from a model another company runs, and what the lesson does about each.
- The MLOps loop: data, training, evaluation, deployment, monitoring and retraining, with the model ours.
- The levers of LLMOps: model choice, the prompt and context, retrieval, tools, and fine-tuning as the last resort.
- A hosted API and a self-hosted open-weight model: where the model runs, and what each choice costs operations.
- The model as a dependency: versions, aliases and retirement: why production pins a dated version.
- A model upgrade as a release: the eval run that decides it, and the rollback.
- An LLM API call: the request: the model, the settings and the context in one JSON body.
- An LLM API call: the response: the content, the stop reason and the tokens billed.
- The price of a call: input and output tokens: the on-call assistant's daily bill, worked through.
- Rate limits: requests and tokens per minute: one minute of calls and the 429 at the end of it.
- Latency: time to first token and output speed: one call, and one agent run of several calls.
- The LLMOps stack of the on-call assistant: the layers of a request and the lesson for each.
The buttons on the left and right edges move between slides. So do the arrow keys and a swipe.
The lab
The model ids and their dates, the eval scores, the prices, the rate limits and the speeds are examples made up for
the drawings; real ones differ by provider and change often. The day's cost, the minute's tokens and the run's
seconds are computed by the slides from those examples. The GPU memory of a 70B model is its parameters times two
bytes, the size of a 16-bit number, and leaves out the memory the model needs while it runs.
Sources