Lesson 7 of LLMOps for AI Agents, in thirteen slides: what observing an agent adds, a trace of one agent run, OpenTelemetry's GenAI semantic conventions, one model call's span, the cost of a run, cost by team and feature, the latency of a model call, run durations with p50 and p95, metrics for dashboards and alerts, prompts and answers in traces, the telemetry pipeline, and a trace linked to evals and feedback.
The slides
The tool is lesson 7 of
LLMOps for AI Agents, in thirteen slides:
- What is a trace?: one complaint about a slow, wrong answer, looked into with ordinary monitoring and then with a trace.
- Observing an agent: one request, many calls: a web service's one path beside the on-call assistant's many calls.
- A trace of one agent run: the run's spans as a waterfall, with each model call's tokens.
- OpenTelemetry's GenAI semantic conventions: the span names and main attributes of an agent run.
- One model call's span and its attributes: the
gen_ai.* attributes, and our own beside them. - The cost of one run: tokens times the price: each model call's cost, and the run's.
- Cost by team and by feature: one day's cost grouped by the spans' attributes.
- The latency of one model call: time to first token, tokens per second, and the total.
- Run durations, p50 and p95: a day of runs as a histogram, and where the traced run's time went.
- Metrics for dashboards and alerts: the assistant's dashboard tiles and three Prometheus alert rules.
- Prompts and answers in traces: debugging and privacy: a redacted prompt, and the controls on captured content.
- The telemetry pipeline: SDK, collector and backends: OTLP to an OpenTelemetry Collector, and on to the backends.
- A trace linked to evals and feedback: the records that carry one run's trace id.
The buttons on the left and right edges move between slides. So do the arrow keys and a swipe.
- The trace is made up: its durations and token counts; the cost, the run's split and the tokens per second are
computed from it.
- The prices are examples: $3 per million input tokens and $15 per million output tokens for a large model.
- The day of runs is generated (400 runs around the traced one), and p50 and p95 are computed from it; the cost
per team and per feature, and the dashboard's error and 429 rates, are made up.
app.* attributes and oncall_cost_usd_total are our own names, not part of the conventions.
Related pages
Sources