LLMOps Lab 1: One Call to a Model
In this lab you send one request to a language model and read what comes back.
- First you set up a small model on your own computer, and your copy of the lab repository: the other labs use both.
- Then you send one request and read the answer, the tokens the model counted, and what the call would cost.
- Last you send the same request at two temperatures and compare the answers.
It goes with the lessons Neural Networks and Transformers and From MLOps to LLMOps.
Set up the model and the repository
The labs run a small open-weight model, qwen2.5:3b, on your own computer, through
Ollama: no API key, and nothing to pay. It needs about 3 GB of disk and 4 GB of free memory.
You also need git, Python 3.10 or newer, Node.js 20 or newer (for lab 2), and a GitHub account.
Install Ollama. Choose one way, and follow its steps only.
Option 1: install Ollama on your computer (recommended)
On macOS or Windows, download the app from ollama.com/download and start it.
On Linux:
1$ curl -fsSL https://ollama.com/install.sh | sh
Option 2: run Ollama in Docker
The image is large (about 8.5 GB), so use this way only if Docker already has the room.
1$ docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:0.40.16 2$ alias ollama='docker exec -it ollama ollama'Download the two models the labs use: the language model, and an embedding model for lab 3.
1$ ollama pull qwen2.5:3b 2$ ollama pull nomic-embed-textCheck that the model's API answers:
1$ curl -s http://localhost:11434/v1/models 2{"object":"list","data":[{"id":"nomic-embed-text:latest",...},{"id":"qwen2.5:3b",...}]}Make your own copy of the lab repository:
Open YuvalShaul/llmops-labs on GitHub.
Click Use this template → Create a new repository, and name it
llmops-labs.Clone your copy, and go into it:
1$ git clone git@github.com:<you>/llmops-labs.git 2$ cd llmops-labs
Send one request
Go into the lab's directory, and look at the request:
1$ cd lab1-first-call 2$ cat request.jsonmodelnames the model, as you named it when you downloaded it.temperaturesets how the next token is picked: at 0, the most probable one every time.messagesis everything the model reads: a system prompt with its instructions, and the question.
Send it to the model, and print the reply with
show.py:1$ curl -s http://localhost:11434/v1/chat/completions \ 2 -H "Content-Type: application/json" -d @request.json | python3 show.py 3A pod in CrashLoopBackOff indicates that it is continuously restarting due to configuration issues, ... 4 5input tokens: 46 6output tokens: 37 7cost at $3.00 / $15.00 per million: $0.000693 8the same call 10,000 times a day: $6.93 a day- The first call takes longer: Ollama loads the model into memory first.
- Your answer may be worded differently.
Read the response
Send the request again, and print the whole response instead:
1$ curl -s http://localhost:11434/v1/chat/completions \ 2 -H "Content-Type: application/json" -d @request.json | python3 -m json.toolFind these three fields in it:
choices[0].message.content: the answer.choices[0].finish_reason: why the model stopped;stopmeans it wrote its stop token.usage: the input tokens (prompt_tokens) and the output tokens (completion_tokens).- A hosted model bills these two numbers, each at its own price.
show.pymultiplies them by example prices of a large hosted model.
Count the input tokens: the system prompt and the question are a few dozen words, and they cost 46 tokens.
- The model keeps nothing between calls: a conversation sends all of its earlier turns again, every time.
Compare two temperatures
Set the temperature to 1 in
request.json:1"temperature": 1,Send the request three times, and compare the first lines of the answers:
1$ for i in 1 2 3; do curl -s http://localhost:11434/v1/chat/completions \ 2 -H "Content-Type: application/json" -d @request.json | python3 show.py | head -1; done 3A pod in CrashLoopBackOff indicates that the pod is continuously restarting due to instability in its configur... 4A pod in CrashLoopBackOff status indicates that it is continuously restarting due to failed container starts, ... 5A pod in a CrashLoopBackOff status indicates that the pod is continuously restarting due to failed initializat...- At temperature 1, a less probable token is sometimes picked, so the same request gets different words.
- That is why lab 2 tests answers with checks on what they contain, not by comparing them with one expected text.
Set the temperature back to 0, and send the request three times again: the answers are now the same, or nearly.