LlamaIndex¶

LlamaIndex (formerly GPT Index) is the data framework purpose-built for RAG (Retrieval-Augmented Generation). While LangChain is a general LLM toolkit, LlamaIndex focuses on connecting LLMs to your data — indexing documents, building vector stores, and querying with retrieval. If your primary need is "ask questions over my documents," LlamaIndex is the specialized tool.

Installation / Setup¶

pip install llama-index llama-index-core
pip install llama-index-llms-openai llama-index-embeddings-openai
# or: uv add llama-index llama-index-llms-openai

Key packages:

Package	Purpose
`llama-index-core`	Core abstractions: indices, retrievers, query engines
`llama-index-llms-openai`	OpenAI LLM integration
`llama-index-embeddings-openai`	OpenAI embeddings
`llama-index-vector-stores-faiss`	FAISS vector store
`llama-index-vector-stores-chroma`	ChromaDB vector store
`llama-index-readers-file`	PDF, CSV, etc. readers

Core Syntax¶

The two core abstractions¶

Abstraction	What it does
Index	Organizes documents for efficient retrieval
Query Engine	Takes a question → retrieves relevant docs → sends to LLM → returns answer

Ingest documents → build index → query¶

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# 1) Load documents
documents = SimpleDirectoryReader("./data").load_data()

# 2) Build index (creates embeddings automatically)
index = VectorStoreIndex.from_documents(documents)

# 3) Query
query_engine = index.as_query_engine()
response = query_engine.invoke("What is the main topic?")
print(response)

That's the simplest RAG pipeline — 3 lines.

Documents¶

from llama_index.core import Document

# from text
doc = Document(text="LangChain is a framework for building LLM apps...")

# from files
from llama_index.readers.file import PDFReader
docs = PDFReader().load_data("report.pdf")

# from directory
from llama_index.core import SimpleDirectoryReader
docs = SimpleDirectoryReader("./data", recursive=True).load_data()

Query engine¶

# default: retrieve top-k → synthesize answer
query_engine = index.as_query_engine(
    similarity_top_k=3,                 # number of documents to retrieve
    response_mode="compact",            # compact, refine, tree_summarize
)

response = query_engine.invoke("What are the key findings?")
print(response)

response_mode	Behavior
`compact`	Concatenate context, one LLM call (default)
`refine`	Iteratively refine answer with each doc
`tree_summarize`	Build a summary tree (good for many docs)
`simple`	Just the raw context, no synthesis

Retriever (lower-level)¶

# retriever returns documents, no LLM call
retriever = index.as_retriever(similarity_top_k=3)
docs = retriever.invoke("What is RAG?")
for doc in docs:
    print(doc.text[:200])

Embedding models¶

from llama_index.embeddings.openai import OpenAIEmbedding

# default: text-embedding-3-small
embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# use with index
index = VectorStoreIndex.from_documents(documents, embed_model=embed_model)

LLM configuration¶

from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o", temperature=0.3)
query_engine = index.as_query_engine(llm=llm)

Common Operations¶

Vector store integration (FAISS)¶

import faiss
from llama_index.vector_stores.faiss import FaissVectorStore
from llama_index.core import VectorStoreIndex, StorageContext, SimpleDirectoryReader

# create FAISS index
vector_store = FaissVectorStore(faiss_index=faiss.IndexFlatL2(1536))
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# build index with FAISS
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context)

# save to disk
index.storage_context.persist("./index_storage")

# load from disk
from llama_index.core import load_index_from_storage
storage_context = StorageContext.from_defaults(persist_dir="./index_storage")
index = load_index_from_storage(storage_context)

ChromaDB integration¶

import chromadb
from llama_index.vector_stores.chroma import ChromaVectorStore
from llama_index.core import VectorStoreIndex, StorageContext

chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("my_docs")

vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)

Custom node parser (chunking)¶

from llama_index.core.node_parser import SentenceSplitter, SentenceWindowNodeParser

# sentence-level chunking (default: 1024 tokens per chunk)
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
nodes = splitter.get_nodes_from_documents(documents)

# sentence window (for better context)
window_parser = SentenceWindowNodeParser(window_size=3)

# build index from nodes
index = VectorStoreIndex(nodes)

Metadata filtering¶

from llama_index.core.vector_stores import ExactMatchFilter, MetadataFilters

# filter by metadata
filters = MetadataFilters(filters=[
    ExactMatchFilter(key="category", value="technical"),
])

query_engine = index.as_query_engine(similarity_top_k=5, filters=filters)
response = query_engine.invoke("What are the best practices?")

Practical Examples¶

Example 1 — Simple Q&A over PDFs¶

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# load all PDFs from a directory
documents = SimpleDirectoryReader(
    "./reports",
    recursive=True,
    required_exts=[".pdf"],
).load_data()

# build index (uses OpenAI embeddings by default)
index = VectorStoreIndex.from_documents(documents)

# query
engine = index.as_query_engine(similarity_top_k=4)
response = engine.invoke("What are the main findings in the Q3 report?")
print(response)

Example 2 — RAG with custom chunking¶

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI

# custom splitter for better chunks
splitter = SentenceSplitter(
    chunk_size=256,
    chunk_overlap=20,
    paragraph_separator="\n\n",
)

documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents, transformations=[splitter])

llm = OpenAI(model="gpt-4o", temperature=0)
engine = index.as_query_engine(llm=llm, similarity_top_k=3)
response = engine.invoke("Summarize the key points about deployment.")

Example 3 — Multi-document synthesis¶

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./research_papers").load_data()
index = VectorStoreIndex.from_documents(documents)

# use tree_summarize for synthesis across many documents
engine = index.as_query_engine(
    response_mode="tree_summarize",
    similarity_top_k=10,
)
response = engine.invoke(
    "Compare and contrast the approaches described in these papers."
)

Example 4 — Chat engine (multi-turn)¶

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# chat engine maintains conversation history
chat_engine = index.as_chat_engine(
    chat_mode="condense_plus_context",    # condenses history + retrieves
    similarity_top_k=3,
)

response = chat_engine.invoke("What is RAG?")
print(response)
response = chat_engine.invoke("How does it differ from fine-tuning?")  # has context
print(response)

Common Patterns¶

Pattern: hybrid search (keyword + semantic)¶

from llama_index.core import VectorStoreIndex
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.postprocessor import SimilarityPostprocessor

retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=5,
    similarity_postprocessor=SimilarityPostprocessor(similarity_cutoff=0.7),
)

Pattern: custom query engine¶

from llama_index.core.query_engine import CustomQueryEngine
from llama_index.core import Response

class MyQueryEngine(CustomQueryEngine):
    def custom_query(self, query_str: str):
        # custom retrieval logic
        nodes = self.retriever.retrieve(query_str)
        # custom synthesis
        context = "\n".join([n.node.text for n in nodes])
        response = self.llm.complete(f"Answer based on:\n{context}\n\nQ: {query_str}")
        return Response(response=str(response), source_nodes=nodes)

Pattern: save and load index¶

# save
index.storage_context.persist("./my_index")

# load
from llama_index.core import load_index_from_storage, StorageContext
storage_context = StorageContext.from_defaults(persist_dir="./my_index")
index = load_index_from_storage(storage_context)

Pattern: document metadata for filtering¶

from llama_index.core import Document

doc = Document(
    text="Quarterly revenue report...",
    metadata={
        "source": "finance",
        "date": "2025-Q3",
        "author": "CFO",
    },
)
# later filter by metadata

Best Practices¶

LlamaIndex is specialized for RAG — use it when document Q&A is the primary goal.
Use SentenceSplitter with appropriate chunk_size — too large loses precision, too small loses context.
Default response_mode="compact" is usually best — use tree_summarize for synthesis.
Always set similarity_top_k — default may retrieve too few or too many documents.
Save your index to disk after building — don't rebuild every time.
Use metadata filtering to narrow searches by source, date, category, etc.
chat_engine is better than query_engine for multi-turn conversations.
Embedding model choice matters — text-embedding-3-small is fast/cheap; 3-large is better quality.

Common Mistakes¶

Mistake	Fix
Rebuilding index every time	Save with `persist()` and load from disk
chunk_size too large (>1000 tokens)	Use 256–512 for most use cases
No chunk overlap	Add `chunk_overlap=20–50` to avoid losing context at boundaries
Ignoring metadata	Add source/date/category for filtering
Using default similarity_top_k for everything	Tune per use case: 3–5 for Q&A, 10+ for synthesis
Not checking response quality	Use `response_mode="refine"` for higher quality
Importing everything from top-level	Use specific packages: `llama_index.llms.openai`

Troubleshooting¶

Symptom	Cause / Fix
`ModuleNotFoundError`	Install specific integration: `pip install llama-index-llms-openai`
Poor retrieval quality	Adjust `chunk_size`, try `SentenceSplitter`, check embedding model
Slow indexing	Use `IngestionPipeline` with caching; fewer documents per batch
Empty responses	`similarity_top_k` too low or similarity cutoff too high
`context_length_exceeded`	Too many retrieved docs — reduce `similarity_top_k`
Index too large on disk	Use quantized embeddings or reduce document count
Chat engine forgets context	Use `condense_plus_context` mode, not `simple`

langchain — general LLM framework
langgraph — complex workflows
generative-ai.md — AI landscape
openai-api — direct API for embeddings
sentence-transformers — embedding models
RAG — retrieval concepts
huggingface-datasets — data processing
FastAPI — serving RAG apps

Official Documentation¶

LlamaIndex docs — https://docs.llamaindex.ai/
LlamaIndex tutorials — https://docs.llamaindex.ai/en/stable/getting_started/starter_example/
LlamaIndex examples — <https://github.com/run-llama/llama_index/tree/main/docs/examples

Last Updated¶

2026-09-06 — Initial version.

LlamaIndex¶

Table of Contents¶

Installation / Setup¶

Core Syntax¶

The two core abstractions¶

Ingest documents → build index → query¶

Documents¶

Query engine¶

Retriever (lower-level)¶

Embedding models¶

LLM configuration¶

Common Operations¶

Vector store integration (FAISS)¶

ChromaDB integration¶

Custom node parser (chunking)¶

Metadata filtering¶

Practical Examples¶

Example 1 — Simple Q&A over PDFs¶

Example 2 — RAG with custom chunking¶

Example 3 — Multi-document synthesis¶

Example 4 — Chat engine (multi-turn)¶

Common Patterns¶

Pattern: hybrid search (keyword + semantic)¶

Pattern: custom query engine¶

Pattern: save and load index¶

Pattern: document metadata for filtering¶

Best Practices¶

Common Mistakes¶

Troubleshooting¶

Related Cheat Sheets¶

Official Documentation¶

Last Updated¶