Ollama¶

Run LLMs on your own machine with zero config. Ollama packages open-weight models (Llama 3, Mistral, Qwen, Phi) into a single CLI command — ollama run llama3 — and exposes an OpenAI-compatible API at localhost:11434. Perfect for development, privacy-sensitive tasks, and offline work.

Table of Contents¶

Installation / Setup
Core Syntax
Common Operations
Practical Examples
Common Patterns
Best Practices
Common Mistakes
Troubleshooting
Related Cheat Sheets
Official Documentation

Installation / Setup¶

# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows: download from https://ollama.com/download
# Docker
docker run -d -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

Core Syntax¶

CLI commands¶

# pull a model (downloads ~4-40GB depending on size)
ollama pull llama3.1              # default 8B, ~4.7GB
ollama pull llama3.1:70b          # 70B, ~40GB
ollama pull mistral               # 7B
ollama pull qwen2.5:7b            # 7B
ollama pull phi3                  # 3.8B

# run interactively
ollama run llama3.1               # chat in terminal
ollama run llama3.1 "hello"       # one-shot prompt

# list downloaded models
ollama list

# remove a model
ollama rm llama3.1

# show model info
ollama show llama3.1

Popular models¶

Model	Size	RAM needed	Quality	Speed
`phi3:mini`	3.8B	4 GB	Good for small tasks	★★★★★
`mistral`	7B	8 GB	Good general	★★★★
`llama3.1`	8B	8 GB	Best at this size	★★★★
`qwen2.5:7b`	7B	8 GB	Strong coding	★★★★
`llama3.1:70b`	70B	48 GB	Excellent	★★
`qwen2.5:72b`	72B	48 GB	Excellent	★★

OpenAI-compatible API¶

from openai import OpenAI

# point to Ollama's local server
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",                   # Ollama doesn't need a real key
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain machine learning in 3 sentences."},
    ],
    temperature=0.7,
)
print(response.choices[0].message.content)

Ollama native API¶

import ollama

# simple generate
response = ollama.generate(model="llama3.1", prompt="What is Python?")
print(response["response"])

# chat
response = ollama.chat(
    model="llama3.1",
    messages=[
        {"role": "user", "content": "Hello!"},
    ],
)
print(response["message"]["content"])

# streaming
stream = ollama.generate(model="llama3.1", prompt="Write a poem", stream=True)
for chunk in stream:
    print(chunk["response"], end="", flush=True)

Ollama REST API¶

# generate
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "What is 2+2?",
  "stream": false
}'

# chat
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'

# list models
curl http://localhost:11434/api/tags

Common Operations¶

Function calling (with OpenAI compatibility)¶

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

response = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Weather in London?"}],
    tools=tools,
)

Note: function calling support varies by model. Llama 3.1+ and Mistral support it well.

Custom models (Modelfile)¶

# create a custom model from a base
cat <<'EOF' > Modelfile
FROM llama3.1
SYSTEM "You are a Python expert. Write clean, well-commented code."
PARAMETER temperature 0.3
PARAMETER top_p 0.9
EOF

ollama create python-expert -f Modelfile
ollama run python-expert

Embeddings¶

import ollama

response = ollama.embeddings(model="llama3.1", prompt="Hello world")
vector = response["embedding"]    # list of floats

Pull specific quantizations¶

# smaller = faster, less quality
ollama pull llama3.1:8b           # default Q4_K_M
ollama pull llama3.1:8b-instruct-q5_K_M   # higher quality quantization

Practical Examples¶

Example 1 — Local chatbot with OpenAI SDK¶

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

messages = [{"role": "system", "content": "You are a helpful data science tutor."}]

def chat(user_input):
    messages.append({"role": "user", "content": user_input})
    response = client.chat.completions.create(
        model="llama3.1", messages=messages, temperature=0.7)
    reply = response.choices[0].message.content
    messages.append({"role": "assistant", "content": reply})
    return reply

# usage
print(chat("What is overfitting?"))
print(chat("How do I prevent it?"))

Example 2 — Local batch classification¶

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

def classify(text):
    response = client.chat.completions.create(
        model="llama3.1",
        messages=[
            {"role": "system", "content": "Classify as positive/negative/neutral. Reply with one word."},
            {"role": "user", "content": text},
        ],
        temperature=0.0,
    )
    return response.choices[0].message.content.strip()

texts = ["Great product!", "Terrible service", "It's okay"]
labels = [classify(t) for t in texts]
print(labels)    # ['positive', 'negative', 'neutral']

Example 3 — Use Ollama as a fallback for cloud API¶

from openai import OpenAI

def call_with_fallback(messages, model="gpt-4o-mini", fallback_model="llama3.1"):
    try:
        # try cloud first
        client = OpenAI(api_key="YOUR_API_KEY")
        response = client.chat.completions.create(model=model, messages=messages)
        return response.choices[0].message.content
    except Exception:
        # fallback to local
        client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
        response = client.chat.completions.create(model=fallback_model, messages=messages)
        return response.choices[0].message.content

Common Patterns¶

Pattern: local development with Ollama + OpenAI SDK¶

# develop against Ollama locally, swap to OpenAI for production
import os

if os.getenv("USE_LOCAL_LLM"):
    client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    model = "llama3.1"
else:
    client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    model = "gpt-4o-mini"

Pattern: model comparison¶

models = ["llama3.1", "mistral", "qwen2.5:7b"]
for m in models:
    response = client.chat.completions.create(
        model=m,
        messages=[{"role": "user", "content": "What is 2+2?"}],
        temperature=0.0,
    )
    print(f"{m}: {response.choices[0].message.content}")

Pattern: streaming for real-time output¶

stream = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Best Practices¶

Start with llama3.1 (8B) — best quality at the 8B size.
Use phi3:mini (3.8B) for resource-constrained environments.
Use qwen2.5:7b for code-heavy tasks.
The OpenAI-compatible API (/v1) means most OpenAI SDK code works with Ollama unchanged.
Always set api_key="ollama" when using the OpenAI SDK with Ollama.
Pull models ahead of time — first run downloads the model.
Use Modelfile for custom system prompts and parameters.
Ollama runs models in memory — close the server to free RAM.

Common Mistakes¶

Mistake	Fix
Forgetting `base_url` when using OpenAI SDK	Set `base_url="http://localhost:11434/v1"`
Using `api_key="real-key"` with Ollama	Set `api_key="ollama"` — Ollama doesn't need a real key
Pulling a model that's too large for your RAM	Check model size vs available RAM — start with 8B
Not pulling the model before first use	`ollama pull llama3.1` before `ollama run`
Assuming Ollama models are as good as GPT-4o	Open-weight models are good but not GPT-4o quality — set expectations
Running too many models simultaneously	Each model uses RAM — only load one at a time on limited hardware

Troubleshooting¶

Symptom	Cause / Fix
`ollama: command not found`	Install Ollama: `curl -fsSL https://ollama.com/install.sh \\| sh`
Connection refused on `localhost:11434`	Ollama server not running — `ollama serve`
Model download is slow	Check internet; use a closer mirror if available
`out of memory`	Model too large — use a smaller model or quantization
Slow inference	Use a smaller model (phi3, mistral 7B); check GPU availability
Model returns garbled output	Try a different quantization or model
Function calling fails	Not all models support it well — use llama3.1 or mistral
`context length exceeded`	Ollama default context is 2048 — set `num_ctx` in Modelfile

llm-concepts — tokens, temperature, prompting
huggingface — Hugging Face for LLMs
litellm — unified multi-provider API
generative-ai.md — AI landscape
openai-api — OpenAI API (same SDK)
Docker — containerizing Ollama
Linux — system requirements
Hardware & GPU — GPU for local inference

Official Documentation¶

Ollama docs — https://github.com/ollama/ollama
Ollama library — https://ollama.com/library
Ollama API — <https://github.com/ollama/ollama/blob/main/docs/api.md

Last Updated¶

2026-09-06 — Initial version.

Ollama¶

Table of Contents¶

Installation / Setup¶

Core Syntax¶

CLI commands¶

Popular models¶

OpenAI-compatible API¶

Ollama native API¶

Ollama REST API¶

Common Operations¶

Function calling (with OpenAI compatibility)¶

Custom models (Modelfile)¶

Embeddings¶

Pull specific quantizations¶

Practical Examples¶

Example 1 — Local chatbot with OpenAI SDK¶

Example 2 — Local batch classification¶

Example 3 — Use Ollama as a fallback for cloud API¶

Common Patterns¶

Pattern: local development with Ollama + OpenAI SDK¶

Pattern: model comparison¶

Pattern: streaming for real-time output¶

Best Practices¶

Common Mistakes¶

Troubleshooting¶

Related Cheat Sheets¶

Official Documentation¶

Last Updated¶