LLM Concepts

The reference sheet for how LLMs work under the hood — tokens, context windows, sampling parameters, prompting strategies, and deployment choices. Everything here applies regardless of whether you're using OpenAI, Anthropic, Gemini, or a local model via Ollama.

Table of Contents

• Installation / Setup
• Core Syntax
• Common Operations
• Practical Examples
• Common Patterns
• Best Practices
• Common Mistakes
• Troubleshooting
• Related Cheat Sheets
• Official Documentation

Installation / Setup

pip install tiktoken                      # OpenAI tokenizer
pip install transformers                   # Hugging Face tokenizer (most models)

Core Syntax

Tokens — the atomic unit

1 token ≈ 0.75 English words ≈ 4 characters

"Hello world"           → 2 tokens
"machine learning"      → 2 tokens
"The cat sat on the mat" → 6 tokens

Code is less efficient:
"def foo(x): return x"  → ~9 tokens

Tokenizers differ by model:

Tokenizer	Used by	Encoding
cl100k_base	GPT-4, GPT-4o, GPT-4o-mini	BPE
o200k_base	GPT-4o, o3	BPE (larger vocab)
claude	Claude (Anthropic)	BPE (internal)
SentencePiece	Llama, Gemini, Mistral	BPE / Unigram

Counting tokens:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, world!")
print(len(tokens))   # ~3

Context windows — what fits

context window = system + messages (input) + response (output)

GPT-4o:       128K tokens total
Claude 3.5:   200K tokens total
Gemini 1.5:   1M+ tokens total
Llama 3:      8K–128K (model-dependent)

Important: tokens are not words. A 128K context fits roughly 96K words (a 200-page book).

Temperature and sampling — controlling randomness

Parameter	Range	Effect
temperature	0.0–2.0	0 = deterministic, 1 = standard, >1 = more random
top_p	0.0–1.0	Nucleus sampling — 0.1 = only top 10% probability mass
top_k	1–∞	Only consider top K tokens
frequency_penalty	-2.0–2.0	Penalize repeated tokens
presence_penalty	-2.0–2.0	Penalize tokens that appeared before

When to use what:

Task	temperature	top_p	Why
Code generation	0.0	1.0	Deterministic, correct output
Classification	0.0	1.0	Deterministic labels
Factual Q&A	0.0–0.3	1.0	Minimize hallucination
Summarization	0.3–0.5	0.9	Slight variation OK
Brainstorming	0.7–0.9	0.95	Creative variety
Creative writing	0.9–1.2	0.95	High variety

Prompting strategies

Strategy	Description	When to use
Zero-shot	No examples — just the task	Simple, well-defined tasks
Few-shot	Include 2-5 examples in the prompt	Complex or domain-specific tasks
Chain-of-thought	"Think step by step"	Reasoning, math, logic
Self-consistency	Multiple samples → majority vote	High-stakes decisions
System message	Set role, constraints, format	Always — it's free
Structured output	Request JSON / specific schema	When output goes to code

Fine-tuning vs RAG vs prompting

Approach	What it does	When to use
Prompting	No training — behavior via instructions	Most tasks, rapid iteration
RAG	Retrieve context → include in prompt	When you need specific/updated data
Fine-tuning	Train the model on your data	Custom style, domain jargon, format

Decision rule: start with prompting → add RAG → fine-tune only if needed.

Common Operations

Token math for cost estimation

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

input_text = "Your long document here..."
input_tokens = len(enc.encode(input_text))
output_tokens = 500  # estimate

# GPT-4o pricing (check current prices)
cost_per_1k_input = 0.0025    # $2.50 / 1M tokens
cost_per_1k_output = 0.01     # $10.00 / 1M tokens
total_cost = (input_tokens / 1000 * cost_per_1k_input +
              output_tokens / 1000 * cost_per_1k_output)
print(f"Estimated cost: ${total_cost:.4f}")

Quantization — running models locally

Type	Bits	Size reduction	Quality loss
FP32	32	1× (baseline)	None
FP16	16	2×	Negligible
INT8	8	4×	Small
INT4	4	8×	Moderate
GGUF (Q4_K_M)	4 (mixed)	~8×	Good for most tasks

LLM sizes (approximate):

Model	FP16 size	Q4 size	Runs on
Llama 3 8B	16 GB	4.5 GB	Laptop with 8GB RAM
Llama 3 70B	140 GB	40 GB	Server with 48GB+ VRAM
Mistral 7B	14 GB	4 GB	Laptop with 8GB RAM

Model selection matrix

Need	Model	Why
Fast + cheap	GPT-4o-mini, Claude Haiku, Gemini Flash	Low cost, high speed
Balanced	GPT-4o, Claude Sonnet, Gemini Pro	Good quality/speed
Best reasoning	Claude Opus, o3, Gemini 2.5 Pro	Complex tasks
Self-hosted	Llama 3, Mistral, Qwen	Privacy, no API costs
Long context	Gemini (1M), Claude (200K)	Large documents
Multimodal	GPT-4o, Gemini, Claude	Vision + text

Evaluation

Method	What it measures	Tool
Human eval	Quality, correctness	Manual review
BLEU / ROUGE	Text similarity to reference	nltk, rouge-score
LLM-as-judge	Model evaluates output	GPT-4 / Claude as judge
Domain benchmarks	Specific capability	MMLU, HumanEval, GSM8K
A/B testing	Real user preference	Production deployment

Practical Examples

Example 1 — Prompt engineering: from vague to precise

# Bad prompt
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Tell me about data science"}],
)

# Good prompt
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a data science instructor. Respond in bullet points. Keep each point under 20 words."},
        {"role": "user", "content": "What are the 5 most important skills for a data scientist?"},
    ],
    temperature=0.3,
)

Example 2 — Token counting before API call

import tiktoken

def estimate_cost(text, model="gpt-4o", output_tokens=500):
    enc = tiktoken.encoding_for_model(model)
    input_tokens = len(enc.encode(text))
    # pricing per 1M tokens (check current rates)
    prices = {"gpt-4o": (2.50, 10.00), "gpt-4o-mini": (0.15, 0.60)}
    input_price, output_price = prices.get(model, (2.50, 10.00))
    cost = (input_tokens / 1e6 * input_price) + (output_tokens / 1e6 * output_price)
    return {"input_tokens": input_tokens, "estimated_cost_usd": round(cost, 4)}

Example 3 — Chain-of-thought prompting

messages = [
    {"role": "system", "content": "You are a math tutor. Think step by step before answering."},
    {"role": "user", "content": """Solve: A store has 3 boxes. Each box has 4 bags. Each bag has 5 apples. How many apples total?
Show your reasoning step by step."""},
]

Common Patterns

Pattern: task decomposition

system = """You are a helpful assistant. For complex tasks:
1. Break the task into smaller steps
2. Address each step explicitly
3. Verify your answer before responding"""

Pattern: few-shot with consistent format

messages = [
    {"role": "system", "content": "Classify the sentiment. Respond with exactly one word: positive, negative, or neutral."},
    {"role": "user", "content": "I love this product!"},
    {"role": "assistant", "content": "positive"},
    {"role": "user", "content": "Terrible experience, waste of money."},
    {"role": "assistant", "content": "negative"},
    {"role": "user", "content": "It's okay, nothing special."},
    {"role": "assistant", "content": "neutral"},
    {"role": "user", "content": "Absolutely fantastic!"},
]

Pattern: output validation

import json

def call_with_validation(messages, expected_keys, model="gpt-4o"):
    for attempt in range(3):
        response = client.chat.completions.create(
            model=model,
            messages=messages + [
                {"role": "user", "content": f"Return JSON with keys: {expected_keys}"}
            ],
            response_format={"type": "json_object"},
            temperature=0.0,
        )
        data = json.loads(response.choices[0].message.content)
        if all(k in data for k in expected_keys):
            return data
    raise ValueError("Model could not produce valid output after 3 attempts")

Best Practices

• Start with the cheapest model — upgrade only if quality is insufficient.
• Temperature 0 for anything going to code; temperature 0.7+ for creative tasks.
• Always use a system message — it's free and shapes behavior.
• Count tokens before sending — avoid surprises and control costs.
• Use few-shot examples when the task is ambiguous.
• Chain-of-thought ("think step by step") improves reasoning accuracy.
• Validate output format — don't trust the model to always return JSON.
• For factual tasks: add "If unsure, say you don't know" to reduce hallucination.
• Fine-tune only after you've exhausted prompting and RAG.

Common Mistakes

Mistake	Fix
Using the most expensive model for everything	Start with mini/flash; upgrade only if needed
No system message	Always set one — it shapes behavior for free
Trusting LLM output without validation	Parse, validate, and retry if format is wrong
Ignoring token count	Use tiktoken to estimate; avoid context length errors
High temperature for factual tasks	Use 0.0–0.3 for facts, extraction, classification
Over-relying on fine-tuning	Prompt engineering + RAG solve most problems first
Not testing edge cases	LLMs behave differently on short vs long inputs
Assuming models remember across API calls	Each call is stateless — you must send history

Troubleshooting

Symptom	Cause / Fix
Model gives different answers each time	Temperature > 0 — lower it for consistency
Context length exceeded	Input too long — truncate or use longer-context model
Model makes up facts (hallucination)	Lower temperature; add context via RAG; say "if unsure, say so"
Output format inconsistent	Use structured output mode; validate in code
Slow responses	Use a faster model (mini/flash); reduce max_tokens
High API costs	Switch to cheaper model; cache responses; reduce tokens
Model ignores instructions	Make system message more explicit; put critical instructions last
Token count seems wrong	Different tokenizers — check with the model's own tokenizer

Related Cheat Sheets

• generative-ai.md — generative AI landscape
• openai-api — OpenAI API
• anthropic-api — Anthropic API
• google-gemini — Gemini API
• ollama — local LLMs
• huggingface — Hugging Face for LLMs
• litellm — unified multi-provider API
• tokenizers — subword tokenization details
• RAG — retrieval-augmented generation
• statistics — probability behind sampling

Official Documentation

• OpenAI prompting guide — https://platform.openai.com/docs/guides/prompt-engineering
• Anthropic prompting — https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering
• Google prompting guide — https://ai.google.dev/gemini-api/docs/prompting-intro

Last Updated

2026-09-06 — Initial version.