LiteLLM¶

Write once, call any LLM. LiteLLM gives you a single Python SDK and OpenAI-compatible proxy server that works with 100+ providers — OpenAI, Anthropic, Gemini, Ollama, Azure, AWS Bedrock, and more. Swap models by changing a string, not your code.

Table of Contents¶

Installation / Setup
Core Syntax
Common Operations
Practical Examples
Common Patterns
Best Practices
Common Mistakes
Troubleshooting
Related Cheat Sheets
Official Documentation

Installation / Setup¶

pip install litellm                       # or: uv add litellm

import litellm

# set API keys (litellm reads from env by default)
# OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, etc.

Core Syntax¶

The unified completion API¶

import litellm

# OpenAI
response = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Anthropic
response = litellm.completion(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Gemini
response = litellm.completion(
    model="gemini/gemini-2.0-flash",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Ollama (local)
response = litellm.completion(
    model="ollama/llama3.1",
    messages=[{"role": "user", "content": "Hello!"}],
)

# same response format for all — standard OpenAI-style
print(response.choices[0].message.content)

Model naming convention¶

provider/model-name

openai/gpt-4o
openai/gpt-4o-mini
anthropic/claude-sonnet-4-20250514
anthropic/claude-haiku-4-20250414
gemini/gemini-2.0-flash
gemini/gemini-2.5-pro
ollama/llama3.1
bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
azure/gpt-4o

Streaming¶

response = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True,
)
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Function calling¶

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

response = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Weather in London?"}],
    tools=tools,
)

Cost tracking¶

import litellm

response = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

# litellm automatically tracks cost
print(response._hidden_params.get("cost"))
print(f"Tokens: {response.usage.prompt_tokens} in, {response.usage.completion_tokens} out")

Common Operations¶

Available models¶

Provider	Models	Prefix
OpenAI	GPT-4o, GPT-4o-mini, o3	`openai/`
Anthropic	Claude 3.5, Claude 4	`anthropic/`
Google	Gemini 2.0, 2.5	`gemini/`
Ollama	Llama, Mistral, Qwen	`ollama/`
Azure	OpenAI models via Azure	`azure/`
AWS Bedrock	Claude, Llama, Mistral	`bedrock/`
Groq	Fast inference	`groq/`
Together	Open models	`together/`

Model fallbacks¶

import litellm

# if primary fails, try fallbacks
response = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    fallbacks=["anthropic/claude-sonnet-4-20250514", "gemini/gemini-2.0-flash"],
)

Router — load balancing across models¶

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "gpt", "litellm_params": {"model": "openai/gpt-4o", "api_key": "YOUR_API_KEY"}},
        {"model_name": "gpt", "litellm_params": {"model": "openai/gpt-4o-mini", "api_key": "YOUR_API_KEY"}},
    ],
    routing_strategy="simple-shuffle",    # or "least-busy", "latency-based"
)

response = router.completion(
    model="gpt",
    messages=[{"role": "user", "content": "Hello!"}],
)

LiteLLM Proxy Server¶

# start a proxy server (OpenAI-compatible)
litellm --model openai/gpt-4o --port 4000

# use it like OpenAI
curl http://localhost:4000/v1/chat/completions -d '{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Hello!"}]
}'

# connect to the proxy with any OpenAI SDK
from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000/v1", api_key="any-key")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

YAML config for the proxy¶

# config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  drop_params: true                    # drop unsupported params
  num_retries: 3
  fallbacks: [{"gpt-4o": ["claude"]}]  # fallback chain

litellm --config config.yaml --port 4000

Practical Examples¶

Example 1 — Multi-provider with fallback¶

import litellm

def ask(question, preferred="openai/gpt-4o"):
    try:
        response = litellm.completion(
            model=preferred,
            messages=[
                {"role": "system", "content": "Be concise."},
                {"role": "user", "content": question},
            ],
            temperature=0.7,
            max_tokens=500,
        )
        return response.choices[0].message.content
    except Exception:
        # fallback to cheaper model
        response = litellm.completion(
            model="openai/gpt-4o-mini",
            messages=[{"role": "user", "content": question}],
            temperature=0.7,
        )
        return response.choices[0].message.content

Example 2 — Compare models side-by-side¶

import litellm

models = [
    "openai/gpt-4o-mini",
    "anthropic/claude-haiku-4-20250414",
    "gemini/gemini-2.0-flash",
    "ollama/llama3.1",
]

question = "What is 2+2? Explain in one sentence."
for model in models:
    try:
        response = litellm.completion(
            model=model,
            messages=[{"role": "user", "content": question}],
            temperature=0.0,
        )
        cost = getattr(response, "_hidden_params", {}).get("cost", "N/A")
        print(f"{model:40s} | {response.choices[0].message.content[:60]}")
    except Exception as e:
        print(f"{model:40s} | ERROR: {e}")

Example 3 — Proxy server with spending limits¶

# config.yaml with budget controls
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: "sk-your-master-key"     # proxy auth

litellm_settings:
  num_retries: 3
  drop_params: true
  success_callback: ["langfuse"]        # log to Langfuse

litellm --config config.yaml --port 4000 --detailed_debug

Common Patterns¶

Pattern: provider-agnostic code¶

import litellm
import os

# define model by env var — swap providers by changing env
MODEL = os.getenv("LLM_MODEL", "openai/gpt-4o-mini")

def generate(prompt, system="You are a helpful assistant."):
    response = litellm.completion(
        model=MODEL,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": prompt},
        ],
        temperature=0.7,
    )
    return response.choices[0].message.content

Pattern: batch with cost tracking¶

import litellm

total_cost = 0
results = []
for text in texts:
    response = litellm.completion(
        model="openai/gpt-4o-mini",
        messages=[{"role": "user", "content": f"Classify: {text}"}],
        temperature=0.0,
    )
    cost = getattr(response, "_hidden_params", {}).get("cost", 0)
    total_cost += cost
    results.append(response.choices[0].message.content)

print(f"Total cost: ${total_cost:.4f}")

Pattern: A/B testing models¶

import litellm, random

def ab_test(question, model_a="openai/gpt-4o", model_b="anthropic/claude-sonnet-4-20250514"):
    model = random.choice([model_a, model_b])
    response = litellm.completion(
        model=model,
        messages=[{"role": "user", "content": question}],
    )
    return {"model": model, "response": response.choices[0].message.content}

Best Practices¶

Use litellm.completion() instead of provider-specific SDKs — same code, any provider.
Always set API keys via environment variables — never hard-code.
Use fallbacks for reliability — if one provider is down, try another.
The proxy server is great for teams — one endpoint, many providers.
cost tracking is automatic — use it to monitor spending.
drop_params: true in config prevents errors from unsupported parameters.
Use Router for load balancing across multiple instances of the same model.
Set max_tokens explicitly — different providers have different defaults.

Common Mistakes¶

Mistake	Fix
Not setting API keys for each provider	Each provider needs its own env var (`OPENAI_API_KEY`, etc.)
Ignoring cost tracking	Use `response._hidden_params.get("cost")` to monitor
Not using fallbacks	Add fallbacks for production reliability
Hard-coding model names	Use env vars or config files to swap models
Forgetting `max_tokens`	Set it explicitly to avoid surprise truncation
Using the wrong model prefix	Always prefix with provider: `openai/gpt-4o`
Proxy server has no auth	Set `master_key` in config for production

Troubleshooting¶

Symptom	Cause / Fix
`AuthenticationError`	Missing API key for that provider — set the env var
`Model not found`	Check model name and prefix (e.g. `openai/gpt-4o`)
Unsupported parameter error	Add `drop_params: true` or remove the parameter
High latency	Use a faster model or check provider status
Cost unexpectedly high	Check token usage; switch to cheaper model
Proxy returns 500	Check proxy logs; usually an auth or model issue
Fallback chain not working	Ensure fallback models are valid and keys are set

generative-ai.md — AI landscape overview
openai-api — OpenAI API details
anthropic-api — Anthropic API details
google-gemini — Gemini API details
ollama — local LLMs
llm-concepts — tokens, temperature, prompting
FastAPI — serving LLM apps
Docker — containerizing the proxy

Official Documentation¶

LiteLLM docs — https://docs.litellm.ai/
LiteLLM proxy — https://docs.litellm.ai/docs/simple_proxy
Supported providers — https://docs.litellm.ai/docs/providers

Last Updated¶

2026-09-06 — Initial version.

LiteLLM¶

Table of Contents¶

Installation / Setup¶

Core Syntax¶

The unified completion API¶

Model naming convention¶

Streaming¶

Function calling¶

Cost tracking¶

Common Operations¶

Available models¶

Model fallbacks¶

Router — load balancing across models¶

LiteLLM Proxy Server¶

YAML config for the proxy¶

Practical Examples¶

Example 1 — Multi-provider with fallback¶

Example 2 — Compare models side-by-side¶

Example 3 — Proxy server with spending limits¶

Common Patterns¶

Pattern: provider-agnostic code¶

Pattern: batch with cost tracking¶

Pattern: A/B testing models¶

Best Practices¶

Common Mistakes¶

Troubleshooting¶

Related Cheat Sheets¶

Official Documentation¶

Last Updated¶