Testing an LLM-driven agent is fundamentally harder than testing an ordinary service. The model is non-deterministic, tools call live external systems, and "quality" is partly subjective. Every framework has to give you some answer for three things: how to write a unit test that doesn't hit a real model, how to score quality across a golden-set of test cases, and how to reproduce a bad production run locally. This chapter walks through each in plain language and compares the surfaces across the ten frameworks.
You can't run your unit-test suite against OpenAI in CI — it's expensive, flaky, and non-deterministic. So you need a fake model: something with the same interface as the real chat client but returning canned responses you control. Three of the ten frameworks ship a mock LLM in their test toolkit; the rest expect you to reach for the host language's general-purpose mocking library.
You're testing an agent that classifies customer questions. The unit test wants to
check that when the model returns "billing", the agent routes to the billing tool.
With a built-in fake, the test reads:
agent = make_agent(model=FakeListChatModel(responses=["billing"])) and
runs offline in milliseconds. With no built-in fake, you reach for
unittest.mock / Moq / Mockito and patch the
chat-client interface yourself — which works but every team rebuilds the same
mock-builder.
Tool mocking is a separate matter. None of the ten frameworks ships a mock-tool primitive — every team uses its host language's mock library when a tool depends on a slow or external service. The pattern is uniform enough that the absence isn't a problem in practice; it's just one more piece of test scaffolding you write yourself.
| Framework | Built-in mock LLM? | What it does | Source |
|---|---|---|---|
| LangChainPython | Yes — five variants. | FakeListChatModel cycles a list of string responses; FakeMessagesListChatModel cycles full BaseMessage objects; GenericFakeChatModel supports streaming tokens; ParrotFakeChatModel echoes input. |
core/langchain_core/language_models/fake_chat_models.py |
| LangGraphPython | Yes — extends LangChain's primitives. | FakeChatModel in the test suite extends GenericFakeChatModel; supports bind_tools() for tool-use scenarios. |
libs/langgraph/tests/fake_chat.py |
| CrewAIPython | No. | Tests use unittest.mock to patch the LLM client directly. |
— |
| Google ADKPython | No. | Tests use fixtures and unittest.mock; the eval framework is the canonical answer for behavioural tests. |
— |
| Semantic KernelPython | No. | Tests use unittest.mock; tests/integration/fakes/ contains plugin stubs but no LLM doubles. |
— |
| MastraTypeScript | No. | Tests use sinon/vitest.mock to stub the chat client. |
— |
| MS Agent Framework.NET | No. | Tests use Moq to stub IChatClient. FakeAuthenticationTokenProvider exists for auth, but no fake chat client. |
— |
| LangChain4jJava | Yes — marked @Experimental. |
ChatModelMock implements the ChatModel interface and returns static responses; StreamingChatModelMock for the streaming variant; both record all invocations for assertion. |
langchain4j-core/src/test/java/dev/langchain4j/model/chat/mock/ChatModelMock.java |
| GenkitGo | No dedicated mock. | Tests define a model with a programmable callback (func(ctx, *ModelRequest, ModelStreamCallback) (*ModelResponse, error)), giving you a one-off fake per test. |
— |
| RigRust | No. | Tests implement the CompletionModel trait yourself for each test case — Rust's standard pattern for trait-based fakes. |
— |
Three frameworks (LangChain, LangGraph, LangChain4j) save you the small but repeat
work of building a fake chat client. LangChain ships the most variants — list-cycling,
message-cycling, streaming, parrot-style echo — covering most of the test scenarios
you'll write. The other seven expect you to reach for the host language's mocking
library (unittest.mock, Moq, sinon,
vitest, Mockito, custom Go callbacks, custom Rust trait
impls). Either approach works; only the convenience differs.
You changed a prompt. Did the agent get better or worse on the dozen scenarios that matter? An eval framework gives you a way to express "run the agent on these test cases and score each one" without rebuilding the runner, the metric, and the comparison view yourself. Six of the ten frameworks ship a meaningful eval framework — five mature, plus CrewAI's experimental module. The others leave you to wire something like Promptfoo, DeepEval, or your own scaffolding.
You're about to ship a prompt change. Before merging the PR, you want to know: did
this change improve quality on the 50 test scenarios you've curated, or regress? With
a built-in eval framework, you have a list of EvalCase objects (input,
expected output, expected tool calls), pick scorers (correctness, faithfulness,
tool-call accuracy), run the suite, and get a pass/fail report comparing this run to
the baseline. Without one, you build the runner, the scoring logic (often calling
another LLM as a judge), and the comparison report from scratch — which most teams do
in a Jupyter notebook that becomes load-bearing infrastructure.
Two important sub-features: LLM-as-judge (using a model to score whether an answer is good) and golden-set replay (re-running a curated set of test cases against the agent). Most production-grade eval frameworks bundle both.
| Framework | Built-in eval framework? | What ships | Source |
|---|---|---|---|
| LangChainPython | Yes — comprehensive. | Sub-packages for comparison, criteria-based scoring, embedding distance, exact-match, regex-match, QA, parsing, agent-trajectory; LLM-as-judge built in. LangSmith adds hosted experiment tracking. | libs/langchain/langchain_classic/evaluation/ |
| LangGraphPython | No native — uses LangChain's evaluation module. | Imports LangChain's scorers; LangGraph-specific evals for trajectory replay typically run via LangSmith. | — |
| CrewAIPython | Yes — marked experimental. | AgentEvaluator with six concrete evaluators (GoalAlignmentEvaluator, ParameterExtractionEvaluator, ReasoningEfficiencyEvaluator, SemanticQualityEvaluator, ToolInvocationEvaluator, ToolSelectionEvaluator); ExperimentRunner; LLM-as-judge via the agent's own LLM. |
src/crewai/experimental/evaluation/ |
| Google ADKPython | Yes — large dedicated module. | EvalCase / EvalSet definitions, AgentEvaluator, trajectory evaluator, rubric-based evaluator, hallucination detector, conversation-scenarios generator, simulation submodule, LLM-as-judge implementation. |
src/google/adk/evaluation/ |
| Semantic KernelPython | No. | Tests use pytest; the .NET SK has separate evaluation tooling. |
— |
| MastraTypeScript | Yes — dedicated package. | Twelve built-in scorers: answer-relevancy, answer-similarity, bias, context-precision, context-relevance, faithfulness, hallucination, noise-sensitivity, prompt-alignment, tool-call-accuracy, toxicity, trajectory. Split into llm/ (judge-driven) and code/ (deterministic) trees. |
packages/evals/src/scorers/ |
| MS Agent Framework.NET | Yes — integrated with Microsoft.Extensions.AI.Evaluation. | IAgentEvaluator, EvalCheck, EvalItem, FunctionEvaluator, ExpectedToolCall, IConversationSplitter; numRepetitions parameter for consistency runs. |
dotnet/src/Microsoft.Agents.AI/Evaluation/ |
| LangChain4jJava | No. | Tests use JUnit + Mockito; quality regression is not framework-managed. | — |
| GenkitGo | Yes — plugin-based. | Built-in evaluators: EvaluatorDeepEqual, EvaluatorRegex, EvaluatorJsonata; custom metrics via MetricConfig; LLM-as-judge configurable. |
go/plugins/evaluators/evaluators.go |
| RigRust | No. | Tests use Rust's built-in test harness; quality regression is not framework-managed. | — |
Five frameworks ship mature eval frameworks: LangChain, ADK, Mastra, MS Agent
Framework, and Genkit Go. The breadth varies — LangChain has the largest scorer
catalogue and the longest-running ecosystem; ADK has the most production-grade
artifacts (named EvalCase / EvalSet types, scenario generator,
simulation submodule); Mastra has the broadest scorer count (twelve including
toxicity, bias, hallucination, trajectory); MS Agent Framework integrates with
Microsoft.Extensions.AI.Evaluation; Genkit Go is the smallest but covers the basics.
CrewAI ships a sixth eval framework under experimental/evaluation/ —
comparable in shape (multiple metric evaluators, an experiment runner) but flagged
experimental.
The other four (LangGraph leans on LangChain's; SK Python / LangChain4j / Rig ship nothing) expect you to wire an external eval tool — Promptfoo, DeepEval, Ragas, or a Jupyter notebook with custom scoring. That works, but you lose the framework's awareness of how to run agents reproducibly under test conditions.
A user reports a bad answer in production. You want to reproduce it locally — same prompts, same tool responses, same model state — so you can put a debugger inside the agent and figure out what went wrong. That's replay: re-executing a recorded run offline. Two of the ten frameworks ship something close to it; the others ask you to reconstruct from logs.
At 9 AM, a customer escalates: the bot recommended an expired promo code. With replay, you pull the trace for that conversation, load the stored state into your local agent, and step through what happened — same model responses, same tool outputs, same intermediate state. You can see that the model misread a context field on step 3 and confidently propagated the wrong value to step 4. Without replay, you eyeball the trace, hand-craft a Python script that matches the prompt, and hope you reproduce the failure.
Replay also matters for regression. Before merging a prompt change, you want to re-run it against the same scenarios that have failed before — not just synthetic evals, but real captured production runs. The frameworks that ship replay are the ones that make this realistic.
| Framework | Replay support? | How it works | Source |
|---|---|---|---|
| LangChainPython | Indirect — through LangSmith / LangGraph checkpointers. | LangSmith captures runs and supports re-execution from the dashboard; pure LangChain has no native replay primitive. | — |
| LangGraphPython | Yes — first-class via checkpointers. | Every super-step writes to a checkpointer (memory / SQLite / Postgres). graph.get_state(config) retrieves any snapshot; you can resume execution from that point or replay the whole thread. |
libs/checkpoint/langgraph/checkpoint/base/__init__.py |
| CrewAIPython | No. | State lives in-memory in the crew object; nothing is persisted in a way that supports replay. | — |
| Google ADKPython | Yes — via eval cases. | EvalCase in eval_case.py defines a replayable case (input + expected trajectory). The eval suite re-runs cases against an agent. |
src/google/adk/evaluation/eval_case.py |
| Semantic KernelPython | No. | No native replay; ChatHistory can be passed in but reconstruction is manual. |
— |
| MastraTypeScript | Indirect — through stored thread history. | Memory thread state is persisted (working + observational memory); you can hydrate an agent with a thread and resume, but replay-as-debug isn't a first-class workflow. | — |
| MS Agent Framework.NET | No native replay. | Eval framework supports numRepetitions for consistency runs but there's no record/replay surface. |
— |
| LangChain4jJava | No. | State is held by host application; replay is on you. | — |
| GenkitGo | Indirect — via tracing. | The trace store captures execution history; reconstruction from traces is manual. | — |
| RigRust | No. | Stateless library; state and replay belong to the host application. | — |
First-class replay belongs to two frameworks: LangGraph (via the checkpointer model
— every super-step is recorded and any snapshot is replayable) and ADK (via the eval
framework's EvalCase abstraction, which defines a replayable scenario
as the unit of testing). LangChain inherits LangGraph's path through LangSmith. The
other six frameworks have no first-class replay — production debugging falls back to
reading traces and reconstructing what happened by hand.
On snapshot testing: only LangChain and LangGraph use snapshot testing in
their own test suites (via the syrupy library; LangGraph stores
.ambr files at tests/__snapshots__/). It's used for
framework-internal schema validation, not for agent-output regression — but it's the
only place where snapshot testing as a pattern is visible across the catalog.
Source-grounded observations on testing surfaces across the ten frameworks.
LangChain (five fake-model variants in fake_chat_models.py), LangGraph
(extends LangChain's via tests/fake_chat.py), and LangChain4j
(ChatModelMock + StreamingChatModelMock, marked
@Experimental). The other seven expect you to reach for the host
language's mocking library.
None of the ten frameworks ships a mock-tool primitive. Every team uses
unittest.mock / Moq / Mockito /
sinon and stubs the tool function directly. Uniformly absent across
the catalog — and uniformly easy to work around.
LangChain (longest-running scorer catalog), ADK (largest production-grade module
with EvalCase/EvalSet, scenario generator, simulation),
Mastra (twelve scorers including toxicity, bias, hallucination, trajectory),
MS Agent Framework (integrated with Microsoft.Extensions.AI.Evaluation), Genkit Go
(plugin-based). CrewAI's experimental module adds a sixth at
experimental/evaluation/.
All six frameworks with eval support include an LLM-as-judge implementation — using a separate model call to score whether an answer is correct, faithful, or satisfies a rubric. The pattern has converged: it's now what production-grade eval frameworks look like.
LangGraph's checkpointer model writes every super-step and lets
get_state() rehydrate any snapshot. ADK's EvalCase
abstraction defines replayable scenarios as the unit of testing. Everywhere else,
reproducing a production run means reading traces and hand-crafting the inputs.
LangChain and LangGraph use the syrupy library for snapshot tests in
their own test suites; LangGraph stores .ambr files at
tests/__snapshots__/. The pattern is for framework-internal schema
validation rather than agent-output regression — but it's the only place where
snapshot testing shows up as a discipline across the catalog.