Agent Monitoring: a Crew Observed

A CrewAI crew of four agents and the programs that observe it, run stage by stage from the agentmon repository. We add logs, traces, metrics, a Grafana dashboard and a watchdog, then break the crew five ways.

The crew and its repository

The viewer replays agentmon, a repository that runs on one machine with Docker Compose.

The crew: four CrewAI agents, one topic at a time ( crew/).
- Planner: plans a short report on the topic.
- Researcher: finds three facts with its search tool.
- Writer: writes one paragraph from the facts.
- Reviewer: approves or rejects the paragraph.
The world outside: a fake model API and a fake search API, each with a fault switch ( world/server.py).
The monitoring: crew/observe/, one file per signal, fed by CrewAI's events.
- The agents' code knows nothing about it.
The programs that observe the crew: listed under Monitoring programs.

Every log line, alert, span name and metric name in the viewer is the repository's own, from a run on this site's machine.

Monitoring programs

Docker:
A program that runs and manages containers.
- Every program here, the crew included, runs in a container of its own.
- It keeps everything the crew prints to stdout, for docker logs to show.
OpenTelemetry Collector:
A relay for monitoring data:
- It receives logs, metrics (but not in our case: we use Prometheus for that) and traces, and sends them on to the stores.
- Here it receives the crew's spans and log lines over OTLP, and passes the spans to Jaeger and the log lines to Loki.
Jaeger:
A database for traces, with its own UI for finding and reading them.
- Its UI is on port 16686.
- The repository pins Jaeger 2.20.0: it is the last release serving the HTTP API that Grafana's Jaeger source searches with (/api/services, /api/traces).
- From 2.21, Grafana's Jaeger search and health check fail with 404.
Loki:
A database for log lines only, from Grafana Labs.
- It indexes each line's labels (service_name="crew"), never its text, which keeps it cheap to run.
- It has no UI of its own: we read it through Grafana.
Prometheus:
A database for metrics, numbers over time.
- It fetches metrics itself from each program's /metrics page.
- Here it fetches the crew's page every 5 s.
- Its own UI, on port 9090, runs queries.
Grafana:
A web application that draws dashboards from the data in other programs' databases.
- Here it reads all three: Loki, Jaeger and Prometheus.
- We run it on port 3000.
watchdog:
It's our own demo program, which keeps the crew running.
- It checks the crew's health and heartbeats, and restarts it, stops it or raises an alert.

Running it yourself

Docker with Compose is all it needs: no API key, and no internet once the images are downloaded.

./operator/stage <1-5>: starts the programs of that stage, and stops the others.
- The crew's code is the same at every stage: a stage only starts the programs that collect what it already makes.
./operator/fault <target> <mode>: breaks something on purpose; ./operator/fault clear mends it.
docker compose --profile '*' down: stops everything.

operator/README.md says what to open at each stage.

Reading the drawing

Left: the programs of the compose file.
- A row per signal: logs, traces, metrics, and the watchdog's health checks.
- A column per role: create, collect, store, read and alert.
- A dashed program is not running at that stage.
- The arrows carry the signals, each named by what travels on it.
Right: the screen we open at the step: a terminal, Jaeger, the crew's /metrics page, Prometheus or Grafana.
- A click on a span opens its attributes; a click on a trace ID opens its trace in Jaeger.
Top: the scene's explanation, the same through all its steps, and what the step has just done.

The roles, the signals and the programs competing for each are the subject of Observability.

How spans reach Jaeger

A span is sent once, whole, after it ends: there is no "span started" message.

What a span carries: its name, start, duration, attributes and status, and three IDs.
- Its trace ID, its own span ID, and its parent's span ID.
The tree is built at the other end. Jaeger links each span to its parent by the parent's span ID.
- So spans may arrive in any order.
- A child ends before its parent, so it arrives first; the root, crew.kickoff, ends last and arrives last.
A queue, sent in batches. An ended span waits in the SDK's queue for the next batch, every 5 s.
- The crew sends only to the OpenTelemetry Collector; the Collector passes the spans on to Jaeger.
- Nothing in the crew knows Jaeger exists.
What is never sent.
- A span that never ends: a hung call's.
- Spans still in the queue when the process is killed.
Log lines go the same way, to Loki, every second.
- That is a second copy of each line: the first goes to stdout, where Docker keeps it.
- Loki never reads Docker's copy.
Reading is the other direction. Our browser and Grafana query Jaeger, on port 16686.

Agents, tasks and runs in CrewAI

An agent in CrewAI is a configuration, not code of its own; the work it does comes from the task it is given (CrewAI 1.15).

Agent: a role, a goal, a backstory, a model ( llm) and its tools.
- Every agent runs the same loop, CrewAI's agent executor: call the model, run the tool it asks for, call it again, until it answers.
- The tools are code; the agent is not.
Task: a description, an expected_output and the agent that does it.
Process: how the crew goes through its tasks.
- Process.sequential, our crew: the tasks run in order, each by its own agent.
- Process.hierarchical: a manager agent hands pieces of every task to the others.
Run: one call to crew.kickoff(), from the topic to the reviewed paragraph.
- CrewAI calls it a kickoff; OpenTelemetry and Jaeger a trace, named by its trace ID.
- In logging in general, an ID written on every line of one request is a correlation ID: the trace ID serves as one.
Events: CrewAI announces every step it takes, such as "agent started" or "tool finished".
- crew/observe/events.py listens to them, and hands each step to the signals.