Running a language model on your own hardware sounds like a GPU cluster and a lost weekend. To get started, it is one container and one curl call. Ollama bundles model downloads, memory management and an HTTP interface into a single binary: you pull a model the way you pull a container image, then talk to it over REST. This post does exactly that (start the container, pull a model, use two endpoints) and explains when this route holds up and when it does not. Not a deep dive, but the minimum that gets you a working local LLM API in fifteen minutes.
Updated 19 October 2026: This article now also covers the content of the former post "Self-Hosting Hermes by Nous Research: vLLM, Tool Calling, and Self-Hosting in Practice".
Updated 7 October 2026: Made the default context length precise: since Ollama 0.15.5 it depends on VRAM (4K below 24 GiB, 32K at 24 to 48 GiB, 256K from 48 GiB).
What Ollama does and when to use it
Ollama is a local inference server. The core idea is packaging: model weights, prompt template and default parameters live together under a name in model:tag form, for example llama3.2:3b. If the tag is omitted, latest applies. A pull fetches the package into a local directory, a request loads it into memory, and after an idle period (keep_alive, five minutes by default) Ollama releases that memory again. You manage no Python environment, no tokenizer configuration and no quantisation by hand.
The boundary follows directly from this. Ollama fits local development, internal tools, prototypes and small teams: anywhere convenience and a low idle footprint matter more than peak throughput. As soon as many requests run in parallel and you need batching, tensor parallelism and squeezed-out GPU sharing, a serving stack such as vLLM is the better route. The section on Hermes and tool calling below shows what that looks like. Short version: Ollama to get started and for single-seat use, vLLM for production load.
Starting Ollama with Docker
A single compose.yaml is enough. Two things matter: a volume for the models, otherwise you re-download several gigabytes after every restart, and a bind to 127.0.0.1 (more on that in the pitfalls below).
services:
ollama:
image: ollama/ollama:0.32.6
ports:
- "127.0.0.1:11434:11434"
volumes:
- ollama_models:/root/.ollama
environment:
OLLAMA_CONTEXT_LENGTH: 8192
restart: unless-stopped
# For NVIDIA GPUs, add this (requires the NVIDIA Container Toolkit):
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: all
# capabilities: [gpu]
volumes:
ollama_models:
Start it and check:
docker compose up -d
curl -s http://localhost:11434/api/version
On a development machine you can skip Docker entirely: on Linux, curl -fsSL https://ollama.com/install.sh | sh installs the service natively, and macOS and Windows have installers. The API is identical in every case.
Pulling a model
Models come from the Ollama library. For a first test llama3.2:3b is a reasonable compromise: a 2.0 GB download, a 128K context window per its model card, and it will run on CPU if it has to (slowly).
docker compose exec ollama ollama pull llama3.2:3b
docker compose exec ollama ollama list
The same works over the API, which is more practical for automation:
curl -s http://localhost:11434/api/pull -d '{ "model": "llama3.2:3b", "stream": false }'
curl -s http://localhost:11434/api/tags
The minimal example: generate and chat
Ollama offers two endpoints you need to know. /api/generate takes a single prompt and is ideal for classification, extraction or summarisation with no conversation history:
curl -s http://localhost:11434/api/generate -d '{
"model": "llama3.2:3b",
"prompt": "Fasse in einem Satz zusammen: Self-Hosting von LLMs.",
"stream": false,
"options": { "temperature": 0.2 }
}'
The reply is a single JSON object; the text sits in response. Ollama also returns metrics, with all durations in nanoseconds: total_duration, load_duration, prompt_eval_count (tokens in the prompt) and eval_count (tokens generated). Compute tokens per second as eval_count / eval_duration * 10^9. That is the simplest way to compare different models on your hardware.
/api/chat takes a messages array instead, with the roles system, user, assistant and tool. Your application owns the conversation history; Ollama is stateless and expects the full array on every call:
curl -s http://localhost:11434/api/chat -d '{
"model": "llama3.2:3b",
"messages": [
{ "role": "system", "content": "Du antwortest knapp und auf Deutsch." },
{ "role": "user", "content": "Nenne drei Vorteile von Self-Hosting." }
],
"stream": false
}'
Here the text sits in message.content. That is the entire foundation: append another {"role": "assistant", ...} plus the next user question to the array and you have a chat.
The flow at a glance:
+--------------+ ollama pull llama3.2:3b +-------------------+
| Library | --------------------------> | Volume |
| ollama.com | weights + template | /root/.ollama |
+--------------+ +---------+---------+
| loads on demand
+--------------+ POST /api/chat v
| Your app | --------------------------> +-------------------+
| (curl, SDK, | | ollama serve |
| backend) | <-------------------------- | Port 11434 |
+--------------+ JSON: message.content +-------------------+
unloads after keep_alive (5m)
Three common pitfalls
1. The API has no authentication. Accessing http://localhost:11434 locally requires no token. That is deliberate, and the reason a native installation only listens on 127.0.0.1. The official container image, by contrast, sets OLLAMA_HOST=0.0.0.0:11434 so that port mapping works at all. Your only boundary is therefore the left-hand side of the mapping. Write "11434:11434" instead of "127.0.0.1:11434:11434" and your model sits open on the network, with Docker publishing ports past common host firewalls. If you genuinely need to expose the service, put a reverse proxy with authentication and TLS in front of it.
2. Streaming is the default. Without "stream": false, /api/generate and /api/chat reply with a sequence of JSON objects, one per line, each carrying a fragment of text. Feed that naively into a JSON parser and you get a syntax error, or just the first token. For scripts and batch jobs, set stream to false consistently; for a UI with a typing effect, keep the default and read line by line. The same applies to /api/pull, which streams download progress.
3. The context window is smaller than you think. Without your own setting, Ollama picks the context length based on VRAM: 4K (4096 tokens) below 24 GiB, 32K at 24 to 48 GiB, 256K from 48 GiB. On machines with less than 24 GiB of VRAM, or without a GPU at all, Ollama therefore works with 4096 tokens, regardless of the 128K the model card advertises. Longer prompts are silently truncated, which shows up as "the model ignores half of my document". Set OLLAMA_CONTEXT_LENGTH on the server (see the compose.yaml above) or num_ctx in options per request. Budget for the cost: memory usage scales with OLLAMA_NUM_PARALLEL times context length, and a generous window plus parallelism will blow past your available VRAM quickly.
Hermes and other models with tool calling
As soon as the model should not just return text but call functions in your application, you need models trained for tool calling. One example is the Hermes series from Nous Research: open-weight models that Nous Research fine-tunes on top of existing foundation models (fine-tuning, not its own pre-training). Their focus is reliable function and tool calling, stable JSON output and consistent steering via the system prompt. The newest generation is Hermes 4 (14B based on Qwen 3, 70B and 405B based on Llama 3.1), joined by Hermes 4.3 (36B based on ByteDance Seed-OSS-36B). The examples here use Hermes 3, which builds on Meta Llama 3.1 and is available in sizes including 8B, 70B and 405B. The weights live in the HuggingFace namespace NousResearch. Hermes 3 falls under the Llama 3.1 Community License: commercial use is permitted, but with conditions (including a threshold for very large user counts as well as naming and attribution requirements). Review the Llama 3.1 license before production use.
For trying it out, Ollama is enough. Available tags vary; check the library before pulling:
# Beispielhaft eine Hermes-Variante ziehen und starten
ollama run hermes3
# oder explizit eine Groesse
ollama run hermes3:8b
For production load, vLLM is the way to go: high throughput through continuous batching and PagedAttention, plus an OpenAI-compatible API. After installation (pip install vllm), start the server with the dedicated tool parser hermes and --enable-auto-tool-choice, so that vLLM correctly converts the tool calls generated by the model into the OpenAI format:
vllm serve NousResearch/Hermes-3-Llama-3.1-8B \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--host 0.0.0.0 \
--port 8000 \
--max-model-len 8192
The server then provides the endpoints /chat/completions, /completions and /models under http://localhost:8000/v1 (vLLM documentation on tool calling). As a rough memory estimate in half precision (FP16/BF16), plan for about two GB of VRAM per billion parameters plus headroom for the KV cache; the 8B variant runs on a single GPU.
For tool calling, you pass a tools array. If the model requests a call, the response delivers tool_calls instead of content. Your application executes the function and returns the result as a message with role: "tool":
import json
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Liefert die aktuelle Temperatur fuer einen Ort.",
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string", "description": "Stadtname"}
},
"required": ["city"],
},
},
}]
messages = [{"role": "user", "content": "Wie warm ist es gerade in Hamburg?"}]
first = client.chat.completions.create(
model="NousResearch/Hermes-3-Llama-3.1-8B",
messages=messages,
tools=tools,
tool_choice="auto",
)
msg = first.choices[0].message
if msg.tool_calls:
call = msg.tool_calls[0]
args = json.loads(call.function.arguments)
# Funktion in Ihrer App ausfuehren (hier nur simuliert)
result = {"city": args["city"], "temp_c": 18}
messages.append(msg) # Assistenten-Nachricht mit tool_calls
messages.append({
"role": "tool",
"tool_call_id": call.id,
"content": json.dumps(result),
})
final = client.chat.completions.create(
model="NousResearch/Hermes-3-Llama-3.1-8B",
messages=messages,
tools=tools,
)
print(final.choices[0].message.content)
The tool_calls object contains an id, the function name, and the arguments as a JSON string. This exact id must be referenced in the role: "tool" response as tool_call_id, so that the model can match the call with the result. Verify the loop with a simulated function like this one before connecting real systems, and validate structured outputs server-side via a schema rather than blindly trusting the model.
Where to go next
You now have a local LLM service with persistent models and two dependable endpoints. The obvious next step is POST /api/embed with an embedding model such as nomic-embed-text: the endpoint takes model and input and returns an array of vectors under embeddings, one per input. Check the length of the first vector (embeddings[0]), because that number is exactly the dimension your vector database needs. Where it goes from there is covered in self-hosting Qdrant for RAG. Existing applications often connect without any rewrite: Ollama additionally exposes an OpenAI-compatible layer at http://localhost:11434/v1/, including /v1/chat/completions.
One note on version pinning: at the time of writing, 0.32.6 is the newest versioned tag of the standard image on Docker Hub while the GitHub releases already stand at v0.32.9: the image tags trail the release list by a few days. Pin a fixed version anyway instead of :latest, or a docker compose pull will change your setup unnoticed. Go deeper in the Ollama quickstart, the API reference and the FAQ, which explains the memory and concurrency parameters in detail.
Note: The articles on this blog are produced with the help of AI and are editorially reviewed before publication. Editorial responsibility lies with Emre Yurtbay (see the legal notice).