LLMOps Lab 1: One Call to a Model

In this lab you send one request to a language model and read what comes back.

  • First you set up a small model on your own computer, and your copy of the lab repository: the other labs use both.
  • Then you send one request and read the answer, the tokens the model counted, and what the call would cost.
  • Last you send the same request at two temperatures and compare the answers.

It goes with the lessons Neural Networks and Transformers and From MLOps to LLMOps.

Set up the model and the repository

The labs run a small open-weight model, qwen2.5:3b, on your own computer, through Ollama: no API key, and nothing to pay. It needs about 3 GB of disk and 4 GB of free memory.

You also need git, Python 3.10 or newer, Node.js 20 or newer (for lab 2), and a GitHub account.

  1. Install Ollama. Choose one way, and follow its steps only.

    Option 1: install Ollama on your computer (recommended)
    • On macOS or Windows, download the app from ollama.com/download and start it.

    • On Linux:

      1$ curl -fsSL https://ollama.com/install.sh | sh
      
    Option 2: run Ollama in Docker

    The image is large (about 8.5 GB), so use this way only if Docker already has the room.

    1$ docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:0.40.16
    2$ alias ollama='docker exec -it ollama ollama'
    
  2. Download the two models the labs use: the language model, and an embedding model for lab 3.

    1$ ollama pull qwen2.5:3b
    2$ ollama pull nomic-embed-text
    
  3. Check that the model's API answers:

    1$ curl -s http://localhost:11434/v1/models
    2{"object":"list","data":[{"id":"nomic-embed-text:latest",...},{"id":"qwen2.5:3b",...}]}
    
  4. Make your own copy of the lab repository:

    • Open YuvalShaul/llmops-labs on GitHub.

    • Click Use this template → Create a new repository, and name it llmops-labs.

    • Clone your copy, and go into it:

      1$ git clone git@github.com:<you>/llmops-labs.git
      2$ cd llmops-labs
      

Send one request

  1. Go into the lab's directory, and look at the request:

    1$ cd lab1-first-call
    2$ cat request.json
    
    • model names the model, as you named it when you downloaded it.
    • temperature sets how the next token is picked: at 0, the most probable one every time.
    • messages is everything the model reads: a system prompt with its instructions, and the question.
  2. Send it to the model, and print the reply with show.py:

    1$ curl -s http://localhost:11434/v1/chat/completions \
    2    -H "Content-Type: application/json" -d @request.json | python3 show.py
    3A pod in CrashLoopBackOff indicates that it is continuously restarting due to configuration issues, ...
    4
    5input tokens:  46
    6output tokens: 37
    7cost at $3.00 / $15.00 per million: $0.000693
    8the same call 10,000 times a day: $6.93 a day
    
    • The first call takes longer: Ollama loads the model into memory first.
    • Your answer may be worded differently.

Read the response

  1. Send the request again, and print the whole response instead:

    1$ curl -s http://localhost:11434/v1/chat/completions \
    2    -H "Content-Type: application/json" -d @request.json | python3 -m json.tool
    
  2. Find these three fields in it:

    • choices[0].message.content: the answer.
    • choices[0].finish_reason: why the model stopped; stop means it wrote its stop token.
    • usage: the input tokens (prompt_tokens) and the output tokens (completion_tokens).
      • A hosted model bills these two numbers, each at its own price.
      • show.py multiplies them by example prices of a large hosted model.
  3. Count the input tokens: the system prompt and the question are a few dozen words, and they cost 46 tokens.

    • The model keeps nothing between calls: a conversation sends all of its earlier turns again, every time.

Compare two temperatures

  1. Set the temperature to 1 in request.json:

    1"temperature": 1,
    
  2. Send the request three times, and compare the first lines of the answers:

    1$ for i in 1 2 3; do curl -s http://localhost:11434/v1/chat/completions \
    2    -H "Content-Type: application/json" -d @request.json | python3 show.py | head -1; done
    3A pod in CrashLoopBackOff indicates that the pod is continuously restarting due to instability in its configur...
    4A pod in CrashLoopBackOff status indicates that it is continuously restarting due to failed container starts, ...
    5A pod in a CrashLoopBackOff status indicates that the pod is continuously restarting due to failed initializat...
    
    • At temperature 1, a less probable token is sometimes picked, so the same request gets different words.
    • That is why lab 2 tests answers with checks on what they contain, not by comparing them with one expected text.
  3. Set the temperature back to 0, and send the request three times again: the answers are now the same, or nearly.

Posts in this series