Ollama: self-hosting an LLM locally – a Docker and REST API quickstart

Running a language model on your own hardware sounds like a GPU cluster and a lost weekend. To get started, it is one container and one curl call. Ollama bundles model downloads, memory management and an HTTP interface into a single binary: you pull a model the way you pull a container image, then talk to it over REST. This post does exactly that – start the container, pull a model, use two endpoints – and explains when this route holds up and when it does not. Not a deep dive, but the minimum that gets you a working local LLM API in fifteen minutes.

What Ollama does – and when to use it

Ollama is a local inference server. The core idea is packaging: model weights, prompt template and default parameters live together under a name in model:tag form, for example llama3.2:3b. If the tag is omitted, latest applies. A pull fetches the package into a local directory, a request loads it into memory, and after an idle period (keep_alive, five minutes by default) Ollama releases that memory again. You manage no Python environment, no tokenizer configuration and no quantisation by hand.

The boundary follows directly from this. Ollama fits local development, internal tools, prototypes and small teams – anywhere convenience and a low idle footprint matter more than peak throughput. As soon as many requests run in parallel and you need batching, tensor parallelism and squeezed-out GPU sharing, a serving stack such as vLLM is the better route. I described what that looks like in self-hosting Hermes with vLLM. Short version: Ollama to get started and for single-seat use, vLLM for production load.

Starting Ollama with Docker

A single compose.yaml is enough. Two things matter: a volume for the models, otherwise you re-download several gigabytes after every restart, and a bind to 127.0.0.1 – more on that in the pitfalls below.

services:
  ollama:
    image: ollama/ollama:0.32.6
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    environment:
      OLLAMA_CONTEXT_LENGTH: 8192
    restart: unless-stopped
    # For NVIDIA GPUs, add this (requires the NVIDIA Container Toolkit):
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

volumes:
  ollama_models:

Start it and check:

docker compose up -d
curl -s http://localhost:11434/api/version

On a development machine you can skip Docker entirely: on Linux, curl -fsSL https://ollama.com/install.sh | sh installs the service natively, and macOS and Windows have installers. The API is identical in every case.

Pulling a model

Models come from the Ollama library. For a first test llama3.2:3b is a reasonable compromise: a 2.0 GB download, a 128K context window per its model card, and it will run on CPU if it has to (slowly).

docker compose exec ollama ollama pull llama3.2:3b
docker compose exec ollama ollama list

The same works over the API, which is more practical for automation:

curl -s http://localhost:11434/api/pull -d '{ "model": "llama3.2:3b", "stream": false }'
curl -s http://localhost:11434/api/tags

The minimal example: generate and chat

Ollama offers two endpoints you need to know. /api/generate takes a single prompt – ideal for classification, extraction or summarisation with no conversation history:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Fasse in einem Satz zusammen: Self-Hosting von LLMs.",
  "stream": false,
  "options": { "temperature": 0.2 }
}'

The reply is a single JSON object; the text sits in response. Ollama also returns metrics, with all durations in nanoseconds: total_duration, load_duration, prompt_eval_count (tokens in the prompt) and eval_count (tokens generated). Compute tokens per second as eval_count / eval_duration * 10^9 – the simplest way to compare different models on your hardware.

/api/chat takes a messages array instead, with the roles system, user, assistant and tool. Your application owns the conversation history; Ollama is stateless and expects the full array on every call:

curl -s http://localhost:11434/api/chat -d '{
  "model": "llama3.2:3b",
  "messages": [
    { "role": "system", "content": "Du antwortest knapp und auf Deutsch." },
    { "role": "user", "content": "Nenne drei Vorteile von Self-Hosting." }
  ],
  "stream": false
}'

Here the text sits in message.content. That is the entire foundation: append another {"role": "assistant", ...} plus the next user question to the array and you have a chat.

The flow at a glance:

   +--------------+   ollama pull llama3.2:3b   +-------------------+
   |   Library    | --------------------------> |  Volume           |
   |  ollama.com  |      weights + template     |  /root/.ollama    |
   +--------------+                             +---------+---------+
                                                          | loads on demand
   +--------------+   POST /api/chat                      v
   |   Your app   | --------------------------> +-------------------+
   |  (curl, SDK, |                             |  ollama serve     |
   |   backend)   | <-------------------------- |  Port 11434       |
   +--------------+   JSON: message.content     +-------------------+
                                                  unloads after keep_alive (5m)

Three common pitfalls

1. The API has no authentication. Accessing http://localhost:11434 locally requires no token – that is deliberate, and the reason a native installation only listens on 127.0.0.1. The official container image, by contrast, sets OLLAMA_HOST=0.0.0.0:11434 so that port mapping works at all. Your only boundary is therefore the left-hand side of the mapping. Write "11434:11434" instead of "127.0.0.1:11434:11434" and your model sits open on the network – and Docker publishes ports past common host firewalls. If you genuinely need to expose the service, put a reverse proxy with authentication and TLS in front of it.

2. Streaming is the default. Without "stream": false, /api/generate and /api/chat reply with a sequence of JSON objects, one per line, each carrying a fragment of text. Feed that naively into a JSON parser and you get a syntax error, or just the first token. For scripts and batch jobs, set stream to false consistently; for a UI with a typing effect, keep the default and read line by line. The same applies to /api/pull, which streams download progress.

3. The context window is smaller than you think. Ollama works with 4096 tokens by default – regardless of the 128K the model card advertises. Longer prompts are silently truncated, which shows up as "the model ignores half of my document". Set OLLAMA_CONTEXT_LENGTH on the server (see the compose.yaml above) or num_ctx in options per request. Budget for the cost: memory usage scales with OLLAMA_NUM_PARALLEL times context length, and a generous window plus parallelism will blow past your available VRAM quickly.

Where to go next

You now have a local LLM service with persistent models and two dependable endpoints. The obvious next step is POST /api/embed with an embedding model such as nomic-embed-text: the endpoint takes model and input and returns an array of vectors under embeddings, one per input – check the length of the first vector (embeddings[0]), because that number is exactly the dimension your vector database needs. Where it goes from there is covered in self-hosting Qdrant for RAG. Existing applications often connect without any rewrite: Ollama additionally exposes an OpenAI-compatible layer at http://localhost:11434/v1/, including /v1/chat/completions.

One note on version pinning: at the time of writing, 0.32.6 is the newest versioned tag of the standard image on Docker Hub while the GitHub releases already stand at v0.32.9 – the image tags trail the release list by a few days. Pin a fixed version anyway instead of :latest, or a docker compose pull will change your setup unnoticed. Go deeper in the Ollama quickstart, the API reference and the FAQ, which explains the memory and concurrency parameters in detail.

Publication note: This article was scheduled for 16 July 2026. Because of a technical fault in our publishing automation, it did not go live until 11 August 2026. All information was re-checked for accuracy before publication.