Hugging Face for LLMs¶

Hugging Face is the hub for open-weight LLMs — thousands of models (Llama, Mistral, Qwen, Phi) with hosted inference, local deployment via vLLM, and a simple pipeline API. This sheet covers the practical path from "load a model" to "serve in production."

Table of Contents¶

Installation / Setup
Core Syntax
Common Operations
Practical Examples
Common Patterns
Best Practices
Common Mistakes
Troubleshooting
Related Cheat Sheets
Official Documentation

Installation / Setup¶

pip install transformers accelerate torch   # core
pip install bitsandbytes                    # INT4/INT8 quantization
pip install vllm                            # fast serving

Core Syntax¶

Text generation pipeline (simplest)¶

from transformers import pipeline

generator = pipeline("text-generation", model="meta-llama/Llama-3.1-8B-Instruct")
response = generator("What is machine learning?", max_new_tokens=200)
print(response[0]["generated_text"])

Chat with Llama 3 / Mistral / Qwen¶

from transformers import pipeline

generator = pipeline("text-generation", model="meta-llama/Llama-3.1-8B-Instruct",
                     device_map="auto")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain RAG in simple terms."},
]

response = generator(messages, max_new_tokens=512, temperature=0.7)
print(response[0]["generated_text"][-1]["content"])

Using the tokenizer's chat template¶

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(prompt)   # formatted prompt with special tokens

Load model directly (more control)¶

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", torch_dtype=torch.float16)

messages = [{"role": "user", "content": "Hello!"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Quantized models (INT4 / INT8)¶

from transformers import BitsAndBytesConfig

# INT4 quantization (4GB for 7B model)
bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct", quantization_config=bnb_config, device_map="auto")

# INT8 quantization
bnb_config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct", quantization_config=bnb_config, device_map="auto")

Hugging Face Inference API (hosted)¶

from huggingface_hub import InferenceClient

client = InferenceClient(model="meta-llama/Llama-3.1-8B-Instruct")

# text generation
response = client.text_generation("What is Python?", max_new_tokens=200)

# chat
response = client.chat_completion(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=512,
)
print(response.choices[0].message.content)

Common Operations¶

Popular LLM models on HF¶

Model	Size	Best for	License
`meta-llama/Llama-3.1-8B-Instruct`	8B	General, balanced	Llama 3.1
`meta-llama/Llama-3.1-70B-Instruct`	70B	Best open-weight	Llama 3.1
`mistralai/Mistral-7B-Instruct-v0.3`	7B	Fast, good quality	Apache 2.0
`Qwen/Qwen2.5-7B-Instruct`	7B	Coding, reasoning	Apache 2.0
`microsoft/Phi-3.5-mini-instruct`	3.8B	Tiny, fast	MIT
`google/gemma-2-9b-it`	9B	Google quality	Gemma
`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`	7B	Reasoning	MIT

Model formats¶

Format	Extension	Best for
safetensors	`.safetensors`	Standard HF format
GGUF	`.gguf`	Ollama, llama.cpp, CPU inference
GPTQ	(HF hosted)	INT4 quantized, GPU
AWQ	(HF hosted)	INT4 quantized, faster than GPTQ

vLLM — fast serving¶

pip install vllm

# serve a model
vllm serve meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000

# OpenAI-compatible API at http://localhost:8000/v1

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="vllm")
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Practical Examples¶

Example 1 — Local LLM with quantization¶

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
model_id = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, quantization_config=bnb_config, device_map="auto")

messages = [
    {"role": "system", "content": "You are a Python expert."},
    {"role": "user", "content": "Write a function to check if a number is prime."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=512, temperature=0.3)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Example 2 — Batch inference¶

from transformers import pipeline

generator = pipeline("text-generation", model="meta-llama/Llama-3.1-8B-Instruct",
                     device_map="auto", batch_size=4)

prompts = [
    [{"role": "user", "content": "What is Python?"}],
    [{"role": "user", "content": "What is JavaScript?"}],
    [{"role": "user", "content": "What is Rust?"}],
]
responses = generator(prompts, max_new_tokens=100)
for r in responses:
    print(r[0]["generated_text"][-1]["content"][:80])

Example 3 — HF Inference API (no GPU needed)¶

from huggingface_hub import InferenceClient

client = InferenceClient()  # uses HF token from env

response = client.chat_completion(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Explain transformers in 2 sentences."},
    ],
    max_tokens=200,
)
print(response.choices[0].message.content)

Common Patterns¶

Pattern: model switching with pipeline¶

from transformers import pipeline

def create_generator(model_name="meta-llama/Llama-3.1-8B-Instruct"):
    return pipeline("text-generation", model=model_name, device_map="auto")

# switch models easily
gen = create_generator("mistralai/Mistral-7B-Instruct-v0.3")

Pattern: streaming for real-time¶

from transformers import pipeline

generator = pipeline("text-generation", model="meta-llama/Llama-3.1-8B-Instruct",
                     device_map="auto")

stream = generator("Write a story", max_new_tokens=200, do_sample=True,
                   temperature=0.8)
# pipeline doesn't stream by default — use vLLM or the model directly for streaming

Pattern: guardrails via system message¶

messages = [
    {"role": "system", "content": """Rules:
- Never provide medical advice
- Always cite when uncertain
- Respond in bullet points"""},
    {"role": "user", "content": user_input},
]

Best Practices¶

Use pipeline("text-generation") for the simplest API — handles tokenization + generation.
Use apply_chat_template for model-specific prompt formatting — different models expect different formats.
INT4 quantization (BitsAndBytesConfig(load_in_4bit=True)) cuts memory 4× with minimal quality loss.
device_map="auto" automatically distributes across available GPUs.
Use vLLM for production serving — much faster than pipeline for concurrent requests.
Hugging Face Inference API is free for small usage — good for prototyping.
Always check the model card for license, quantization, and usage instructions.

Common Mistakes¶

Mistake	Fix
`CUDA out of memory`	Use INT4 quantization or a smaller model
Garbled output	Wrong chat template — use `apply_chat_template`
Forgetting `add_generation_prompt=True`	Required for chat formatting
Using base model for chat	Use the `Instruct` variant for conversational use
Ignoring the model card	Check license, required tokens, and usage instructions
Pipeline returns raw text	Access `response[0]["generated_text"][-1]["content"]`
Model too slow on CPU	Use quantization or the Inference API

Troubleshooting¶

Symptom	Cause / Fix
`CUDA out of memory`	Load with `load_in_4bit=True`; use smaller model
Model returns empty/garbled	Wrong chat template — use `apply_chat_template`
`trust_remote_code` error	Add `trust_remote_code=True` for some models
Slow first inference	Model loading is slow — subsequent calls are fast
`safetensors` not found	Model not fully downloaded — re-run `from_pretrained`
Inference API rate limit	Upgrade to HF Pro or use local inference

llm-concepts — tokens, temperature, prompting
ollama — simpler local LLM runner
litellm — unified multi-provider API
huggingface-datasets — data processing
huggingface-transformers — model loading details
sentence-transformers — embeddings
Docker — containerizing LLM serving
Hardware & GPU — GPU for inference

Official Documentation¶

Hugging Face docs — https://huggingface.co/docs
Text generation pipeline — https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline.task_coordinates.task
vLLM — https://docs.vllm.ai/
Inference API — https://huggingface.co/docs/api-inference/

Last Updated¶

2026-09-06 — Initial version.

Hugging Face for LLMs¶

Table of Contents¶

Installation / Setup¶

Core Syntax¶

Text generation pipeline (simplest)¶

Chat with Llama 3 / Mistral / Qwen¶

Using the tokenizer's chat template¶

Load model directly (more control)¶

Quantized models (INT4 / INT8)¶

Hugging Face Inference API (hosted)¶

Common Operations¶

Popular LLM models on HF¶

Model formats¶

vLLM — fast serving¶

Practical Examples¶

Example 1 — Local LLM with quantization¶

Example 2 — Batch inference¶

Example 3 — HF Inference API (no GPU needed)¶

Common Patterns¶

Pattern: model switching with pipeline¶

Pattern: streaming for real-time¶

Pattern: guardrails via system message¶

Best Practices¶

Common Mistakes¶

Troubleshooting¶

Related Cheat Sheets¶

Official Documentation¶

Last Updated¶