Reranking

Reranking reorders retrieved chunks by true relevance to the query. While initial retrieval (similarity search) is fast but approximate, reranking uses a more powerful model to find the best matches — improving answer quality at the cost of latency.

Table of Contents

• Installation / Setup
• Core Syntax
• Common Operations
• Practical Examples
• Common Patterns
• Best Practices
• Common Mistakes
• Troubleshooting
• Related Cheat Sheets
• Official Documentation

Installation / Setup

pip install cohere sentence-transformers langchain
# or: uv add cohere sentence-transformers

Core Syntax

Reranking methods

Method	Type	Speed	Quality	Cost
Cohere Rerank	API	★★★★	★★★★★	$1/1000 queries
Cross-encoder	Local	★★★	★★★★★	Free
ColBERT	Local	★★★★	★★★★	Free
LLM reranking	API	★	★★★★★	Expensive

Cohere Rerank (API)

import cohere

co = cohere.Client("YOUR_API_KEY")

results = co.rerank(
    model="rerank-v3.5",
    query="What is machine learning?",
    documents=[
        "Machine learning is a subset of AI that learns from data.",
        "Python is a programming language used for web development.",
        "Deep learning uses neural networks with many layers.",
    ],
    top_n=3,
)
for r in results.results:
    print(f"Index {r.index}: {r.relevance_score:.3f} — {r.index}")

Cross-encoder (local, free)

from sentence_transformers import CrossEncoder

# load a pretrained cross-encoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

query = "What is machine learning?"
documents = [
    "Machine learning is a subset of AI that learns from data.",
    "Python is a programming language used for web development.",
    "Deep learning uses neural networks with many layers.",
]

# score each query-document pair
pairs = [(query, doc) for doc in documents]
scores = reranker.predict(pairs)

# sort by score
ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
for doc, score in ranked:
    print(f"{score:.3f}: {doc[:80]}")

LangChain reranking

from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank
from langchain_openai import ChatOpenAI

# Cohere reranker
reranker = CohereRerank(model="rerank-v3.5", top_n=3)

# wrap existing retriever
compression_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 10}),
)

# retrieve + rerank in one call
docs = compression_retriever.invoke("What is RAG?")

Local reranker (cross-encoder via LangChain)

from langchain.retrievers import ContextualCompressionRetriever
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

reranker = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-6-v2")

compression_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 10}),
)

Common Operations

Retrieval → reranking pipeline

# step 1: retrieve many candidates
candidates = vectorstore.similarity_search(query, k=20)

# step 2: rerank top candidates
pairs = [(query, doc.page_content) for doc in candidates]
scores = reranker.predict(pairs)

# step 3: take top K after reranking
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
final_docs = [doc for doc, score in ranked[:5]]

Cohere rerank with metadata

results = co.rerank(
    model="rerank-v3.5",
    query=query,
    documents=[doc.page_content for doc in candidates],
    top_n=5,
    # Cohere can use document metadata for better reranking
)

Practical Examples

Example 1 — End-to-end RAG with reranking

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

# retriever + reranker
retriever = vectorstore.as_retriever(search_kwargs={"k": 15})
reranker = CohereRerank(model="rerank-v3.5", top_n=5)
compression = ContextualCompressionRetriever(
    base_compressor=reranker, base_retriever=retriever)

# RAG chain
rag = (
    {"context": compression | (lambda d: "\n\n".join(x.page_content for x in d)),
     "question": RunnablePassthrough()}
    | ChatPromptTemplate.from_template(
        "Answer based on context:\n{context}\n\nQ: {question}")
    | ChatOpenAI(model="gpt-4o")
    | StrOutputParser()
)

Example 2 — Compare reranking impact

# without reranking
docs_no_rerank = vectorstore.similarity_search(query, k=5)

# with reranking
candidates = vectorstore.similarity_search(query, k=20)
pairs = [(query, doc.page_content) for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
docs_reranked = [doc for doc, _ in ranked[:5]]

# compare — reranked should have more relevant docs in top positions

Common Patterns

Pattern: two-stage retrieval

# stage 1: fast retrieval (high recall)
candidates = vectorstore.similarity_search(query, k=20)

# stage 2: precise reranking (high precision)
reranked = reranker.rerank(query, [d.page_content for d in candidates], top_n=5)
final_docs = [candidates[r.index] for r in reranked.results]

Pattern: reranking with score threshold

results = co.rerank(model="rerank-v3.5", query=query, documents=docs, top_n=10)
# filter by relevance score
good_results = [r for r in results.results if r.relevance_score > 0.5]

Best Practices

• Always use reranking in production — it significantly improves answer quality.
• Retrieve more, rerank to fewer — K=15–20 candidates, rerank to top 5.
• Cohere Rerank v3.5 is the best API option — fast and high quality.
• Cross-encoders are free and nearly as good — use for offline/low-budget.
• Reranking adds 100–500ms latency — acceptable for most applications.
• Test the impact — measure answer quality with and without reranking.

Common Mistakes

Mistake	Fix
Not using reranking	Add it — the quality improvement is worth the latency
Retrieving too few candidates	Retrieve 15–20, then rerank to top 5
Reranking everything	Only rerank retrieved candidates, not the full corpus
Ignoring reranking scores	Use scores to filter low-relevance results

Troubleshooting

Symptom	Cause / Fix
Reranking is slow	Use a lighter model or reduce candidates
Quality didn't improve	Check embedding model and chunking first
API rate limit (Cohere)	Implement backoff; batch requests
Cross-encoder OOM	Use a smaller model or batch processing

Related Cheat Sheets

• retrieval.md — initial retrieval methods
• rag.md — the complete pipeline
• embeddings.md — vector representations
• rag-evaluation.md — measuring quality

Official Documentation

• Cohere Rerank — https://docs.cohere.com/docs/reranking
• Sentence Transformers Cross-Encoders — <https://www.sbert.net/docs/cross_encoder/overview.html

Last Updated

2026-09-06 — Initial version.