How an agent that calls hosted models is deployed, in thirteen slides: the on-call assistant in production, the LLM gateway and its configuration, model routing, fallbacks, timeouts and streaming, three caches, rate limits and quotas, rolling out a new model or prompt and how that differs from a code deploy, self-hosting an open-weight model, and the deployment as code.
The slides
The tool is lesson 8 of
LLMOps for AI Agents, in thirteen slides:
- What is an LLM gateway?: three failures of services that call a provider by themselves, and what a gateway does about each.
- The on-call assistant in production: the services in the cluster, the model providers outside it.
- The LLM gateway: one endpoint in front of many providers: what it does on every call.
- The gateway's configuration file: a LiteLLM proxy's
config.yaml, and a team with a budget. - Model routing: the model chosen per request: a small model sorts the questions.
- Fallbacks: a request that fails over: two retries with backoff, then another provider.
- Timeouts and streaming: time to first token, and a timeout on it.
- Three caches for LLM calls: exact-match, semantic, and provider prompt caching.
- Rate limits and quotas: tokens per minute against a limit, a queue and back-pressure.
- Rolling out a new model or prompt: shadow traffic, a growing canary, gates and rollback.
- A model change compared with a code deploy: the same steps, different signals.
- Self-hosting an open-weight model: vLLM on GPU nodes, scaled on its queue, and a cold start.
- The deployment as code: the repository and the chart's values.
The buttons on the left and right edges move between slides. So do the arrow keys and a swipe.
The timings of the failover, the stream and the cold start, the tokens per minute, the image tag and the prompt
version are made up for the drawings. The model ids in the gateway's configuration are placeholders.
Sources