Agent Evaluation¶

Agents are non-deterministic and complex — you can't test them like regular code. Evaluation means measuring success rate, tracking failure modes, and iteratively improving. This covers metrics, benchmarking, and building evaluation pipelines.

Table of Contents¶

Installation / Setup
Core Syntax
Common Operations
Practical Examples
Common Patterns
Best Practices
Common Mistakes
Troubleshooting
Related Cheat Sheets
Official Documentation

Installation / Setup¶

pip install langchain-openai

Core Syntax¶

Agent evaluation dimensions¶

Dimension	What to measure	How
Task completion	Did the agent finish the task?	Binary pass/fail
Correctness	Is the answer correct?	Compare to ground truth
Efficiency	How many steps/tools used?	Count steps, API calls
Safety	Did it avoid harmful actions?	Check guardrails held
Cost	Token usage, API calls	Track usage per run
Latency	Time to complete	Wall-clock measurement

Failure modes¶

Failure mode	Description	Detection
Hallucination	Agent makes up facts	Compare to sources
Tool misuse	Wrong tool or wrong args	Log tool calls
Infinite loop	Agent never finishes	Step count limit
Wrong plan	Steps don't lead to solution	Check intermediate results
Context overflow	Too many tokens	Token count check
Permission error	Tries disallowed actions	Guardrail logs
Slow response	Takes too long	Timeout check

Basic evaluation¶

def evaluate_agent(agent, test_cases):
    results = {"passed": 0, "failed": 0, "errors": []}

    for case in test_cases:
        try:
            response = agent.invoke({"messages": [("human", case["input"])]})
            answer = response["messages"][-1].content

            # check correctness
            if case.get("expected"):
                correct = case["expected"].lower() in answer.lower()
            else:
                correct = len(answer) > 0   # basic sanity check

            if correct:
                results["passed"] += 1
            else:
                results["failed"] += 1
                results["errors"].append({
                    "input": case["input"],
                    "expected": case.get("expected"),
                    "got": answer[:200],
                })
        except Exception as e:
            results["failed"] += 1
            results["errors"].append({"input": case["input"], "error": str(e)})

    results["success_rate"] = results["passed"] / len(test_cases)
    return results

Common Operations¶

Track efficiency metrics¶

def evaluate_efficiency(agent, test_cases):
    metrics = []
    for case in test_cases:
        response = agent.invoke({"messages": [("human", case["input"])]})
        messages = response["messages"]

        tool_calls = sum(1 for m in messages if hasattr(m, "tool_calls") and m.tool_calls)
        total_tokens = sum(m.usage_metadata.get("total_tokens", 0)
                          for m in messages if hasattr(m, "usage_metadata"))

        metrics.append({
            "steps": len(messages),
            "tool_calls": tool_calls,
            "tokens": total_tokens,
        })

    return {
        "avg_steps": sum(m["steps"] for m in metrics) / len(metrics),
        "avg_tool_calls": sum(m["tool_calls"] for m in metrics) / len(metrics),
        "avg_tokens": sum(m["tokens"] for m in metrics) / len(metrics),
    }

LLM-as-judge evaluation¶

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY")

def judge_response(query, response, reference=None):
    prompt = f"""Rate this agent response on a scale of 1-5:
Query: {query}
Response: {response}
{"Reference: " + reference if reference else ""}

Score (1-5) and brief explanation:"""

    result = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
    )
    return result.choices[0].message.content

Track failure modes¶

def categorize_failures(errors):
    categories = {
        "hallucination": [], "tool_misuse": [], "wrong_plan": [],
        "slow": [], "error": [],
    }
    for err in errors:
        if "hallucin" in str(err).lower():
            categories["hallucination"].append(err)
        elif "tool" in str(err).lower():
            categories["tool_misuse"].append(err)
        elif "timeout" in str(err).lower():
            categories["slow"].append(err)
        else:
            categories["error"].append(err)
    return categories

Practical Examples¶

Example 1 — End-to-end agent evaluation¶

import time

def full_evaluation(agent, test_cases):
    results = {
        "total": len(test_cases),
        "passed": 0, "failed": 0,
        "times": [], "tokens": [], "scores": [],
        "failures": [],
    }

    for case in test_cases:
        start = time.time()
        try:
            response = agent.invoke({"messages": [("human", case["input"])]})
            elapsed = time.time() - start
            answer = response["messages"][-1].content

            score = judge_response(case["input"], answer, case.get("expected"))

            results["times"].append(elapsed)
            results["scores"].append(score)
            if "4" in score or "5" in score:
                results["passed"] += 1
            else:
                results["failed"] += 1
                results["failures"].append({"input": case["input"], "score": score})
        except Exception as e:
            results["failed"] += 1
            results["failures"].append({"input": case["input"], "error": str(e)})

    results["avg_time"] = sum(results["times"]) / len(results["times"])
    results["success_rate"] = results["passed"] / results["total"]
    return results

Example 2 — Regression testing¶

def regression_test(agent, test_cases, threshold=0.8):
    """Ensure quality doesn't drop below threshold."""
    results = full_evaluation(agent, test_cases)

    assert results["success_rate"] >= threshold, (
        f"Regression detected: {results['success_rate']:.2%} < {threshold:.2%}"
    )
    print(f"✓ Pass rate: {results['success_rate']:.2%} (threshold: {threshold:.2%})")
    return results

Example 3 — A/B testing agents¶

def ab_test(agent_a, agent_b, test_cases, n_runs=5):
    """Compare two agent versions."""
    scores_a, scores_b = [], []

    for _ in range(n_runs):
        for case in test_cases:
            r_a = agent_a.invoke({"messages": [("human", case["input"])]})
            r_b = agent_b.invoke({"messages": [("human", case["input"])]})

            s_a = float(judge_response(case["input"], r_a["messages"][-1].content).split()[0])
            s_b = float(judge_response(case["input"], r_b["messages"][-1].content).split()[0])

            scores_a.append(s_a)
            scores_b.append(s_b)

    avg_a = sum(scores_a) / len(scores_a)
    avg_b = sum(scores_b) / len(scores_b)
    print(f"Agent A: {avg_a:.2f}, Agent B: {avg_b:.2f}")
    return "A" if avg_a > avg_b else "B"

Common Patterns¶

Pattern: golden dataset¶

# maintain a set of known-good test cases
golden_dataset = [
    {"input": "What is Python?", "expected": "programming language", "category": "factual"},
    {"input": "Sort this list", "expected": "sorted list", "category": "code"},
    {"input": "Summarize this document", "expected": "summary", "category": "analysis"},
]

Pattern: monitor in production¶

def production_monitor(query, response, metadata):
    """Log production metrics for monitoring."""
    log_entry = {
        "query": query,
        "response_length": len(response),
        "tools_used": metadata.get("tool_calls", 0),
        "latency_ms": metadata.get("latency", 0),
        "tokens": metadata.get("tokens", 0),
    }
    # send to monitoring system
    save_to_log(log_entry)

Pattern: iterative improvement¶

def improve_from_failures(agent, failures, llm):
    """Analyze failures and suggest improvements."""
    failure_summary = "\n".join(f"- {f['input']}: {f.get('error', 'wrong answer')}"
                                for f in failures[:10])

    suggestions = llm.invoke(
        f"""Based on these agent failures, suggest improvements:
{failure_summary}

Suggest specific changes to tools, prompts, or routing."""
    ).content
    return suggestions

Best Practices¶

Build a golden test set — 20–50 cases covering common scenarios.
Track success rate over time — catch regressions.
Log everything — queries, responses, tool calls, latencies.
Use LLM-as-judge for subjective quality; exact match for factual tasks.
Test edge cases — empty inputs, adversarial queries, long inputs.
Monitor in production — real-world data reveals unexpected failures.
Iterate from failures — analyze and fix the most common failure modes first.

Common Mistakes¶

Mistake	Fix
No evaluation at all	Build a test set; measure before shipping
Testing only happy paths	Add edge cases and failure scenarios
Ignoring efficiency	Track steps, tokens, latency
Manual evaluation only	Automate with LLM-as-judge
Not monitoring production	Log metrics continuously

Troubleshooting¶

Symptom	Cause / Fix
Low success rate	Check tool descriptions, prompt quality
Agent too slow	Reduce steps, use faster model
High cost	Track token usage; optimize prompts
Inconsistent results	Fix temperature; add deterministic fallback

ai-agents.md — agent loop overview
tools.md — tool design
rag-evaluation — RAG evaluation
ml-evaluation — general evaluation concepts
langgraph — agent framework

Official Documentation¶

LangGraph evaluation — https://langchain-ai.github.io/langgraph/
LangSmith — https://docs.smith.langchain.com/

Last Updated¶

2026-09-06 — Initial version.

Agent Evaluation¶

Table of Contents¶

Installation / Setup¶

Core Syntax¶

Agent evaluation dimensions¶

Failure modes¶

Basic evaluation¶

Common Operations¶

Track efficiency metrics¶

LLM-as-judge evaluation¶

Track failure modes¶

Practical Examples¶

Example 1 — End-to-end agent evaluation¶

Example 2 — Regression testing¶

Example 3 — A/B testing agents¶

Common Patterns¶

Pattern: golden dataset¶

Pattern: monitor in production¶

Pattern: iterative improvement¶

Best Practices¶

Common Mistakes¶

Troubleshooting¶

Related Cheat Sheets¶

Official Documentation¶

Last Updated¶