These are the failures that happen at the data boundary — malformed payloads coming in, personal data leaking out into prompts and logs, the model producing answers that violate your organisation's policy, and the inability to reconstruct what the agent saw and said when a regulator asks. None of these are about whether the LLM is "smart enough"; they're about what the system around the LLM is allowed to do with data. This chapter walks through each failure in plain language, then catalogs the validation, redaction, filtering, and tracing surfaces the five frameworks expose for these conditions.
Whatever calls your agent — a web frontend, a backend service, a queue worker — eventually hands it a chunk of text and says "do something with this." If that text is malformed, empty, or maliciously crafted, the agent doesn't know. To the framework, all input is just a string. Validation has to happen before the agent runtime sees the request, because once the LLM is involved, every wasted tool call costs tokens and time.
A calling service has a bug and starts sending payloads where a customer ID field is the
literal string "undefined". The agent dutifully calls a "lookup customer"
tool with that value, gets back an empty record, decides to "search by phone number"
instead, calls a different tool, gets nothing, asks the user for clarification — except
there is no user, this is a backend job. Forty token-burning hops in, the job times out
and the upstream service gets a vague error. With validation at the API boundary, the
bad payload is rejected in microseconds before any LLM tokens are spent.
None of the frameworks treat the user-facing message as anything but opaque text. The only validation the frameworks do ship is at the tool boundary, where Pydantic models or annotated function signatures define the shape of arguments the LLM is allowed to pass. That catches "model called the tool with the wrong arg types" — it does not catch "upstream sent us garbage."
| Framework | Inbound message validation | Tool-arg validation |
|---|---|---|
| LangChain | String or BaseMessage sequence; no semantic checks. |
StructuredTool uses Pydantic args_schema; validation errors surface as ToolException handled by handle_tool_error. |
| LangGraph | Whatever the state schema accepts; the developer-defined State TypedDict is the contract. |
Tools defined via @tool reuse LangChain's Pydantic-based validation. |
| CrewAI | Task.description is a string template; no schema validation on inputs interpolated into it. |
BaseTool.args_schema Pydantic class enforces tool-arg shape. |
| Google ADK | Free-form text in Content; no inbound validator. |
Function tools infer schemas from Python type hints; mismatches raise at the runtime boundary. |
| Semantic Kernel | Free-form text on ChatHistory; no validator. |
@kernel_function with annotated parameters drives JSON-schema generation for tool calls. |
Personally identifiable information — emails, phone numbers, credit-card numbers, SSNs — slips into agent runs in three ways. Users paste it into questions. Tools return more of it than the agent asked for. The model itself repeats it (or hallucinates near-PII that looks real) in its output. Once it's in the prompt, it's in your trace store, your provider's logs, and possibly your provider's training-data retention. Removing PII before it gets there means scanning text at every boundary.
A user submits a customer-support question and pastes their account number, email, and last four credit-card digits as "context." Your agent forwards the whole message to the model. The provider logs the request — depending on the data-retention policy, that PII may sit in the provider's storage for 30 days. Your trace system also stores it, in plaintext. A few weeks later, compliance reviews trace logs and finds a treasure trove of customer PAN numbers. Now you owe the security team a write-up.
Catching this means scanning text at three points: incoming user input (before the model
sees it), outgoing tool results (before they're added to the prompt), and outgoing model
responses (before they reach the user or the trace store). LangChain v1's
PIIMiddleware is the only built-in PII implementation in the catalog — it
ships built-in detectors for email, credit-card, IP, MAC, and URL with four redaction
strategies ("block", "redact", "mask",
"hash"). Every other framework provides generic interception hooks
(callbacks, filters, guardrails) where a developer-supplied detector — typically a
third-party service like Microsoft Presidio, AWS Comprehend, or Google DLP — does the
actual work.
| Framework | PII surface | Source |
|---|---|---|
| LangChain v1 | PIIMiddleware with built-in detectors for email, credit_card, ip, mac_address, url; redaction strategies "block", "redact", "mask", "hash"; flags apply_to_input, apply_to_output, apply_to_tool_results. |
libs/langchain_v1/langchain/agents/middleware/pii.py:32 |
| LangGraph | No native PII surface. Detection runs in a node placed before or after the agent node, mutating messages. |
— |
| CrewAI | No native PII surface. Task.guardrail: GuardrailType | None is the validation hook on tasks; the detector implementation is supplied by the developer. |
src/crewai/task.py:244 |
| Google ADK | before_model_callback (line 376), after_model_callback (line 391), before_tool_callback (line 420), after_tool_callback (line 435) on LlmAgent are the interception points. |
src/google/adk/agents/llm_agent.py |
| Semantic Kernel | Three filter pipelines registered via kernel_filters_extension: function_invocation_filters, prompt_rendering_filters, auto_function_invocation_filters. Each filter receives (context, next) middleware-style. |
filters/kernel_filters_extension.py:33 |
Provider safety filters block the obvious — explicit content, weapons instructions, hate speech. They're generic and they don't know your business. A healthcare agent can't give diagnosis-shaped advice; a fintech agent can't recommend specific investments; a brand agent can't sound off-tone or mention competitors. These are domain-specific policies that have to be enforced after the model has spoken but before the answer leaves your service. Every framework ships a hook for this; the work of writing the policy is yours.
Your fintech support agent gets asked, "Should I move my savings into bitcoin?" The model — being helpful — produces a thoughtful pros-and-cons analysis that ends with "for someone with your risk tolerance, a small allocation may make sense." That's investment advice. Your legal team will not be amused. With a post-output policy hook, a classifier (or a smaller LLM acting as a judge) inspects the response, recognises it as advice, and substitutes a neutral message: "I can share educational resources but cannot make investment recommendations."
The hook shape varies: LangChain v1 wraps the call middleware-style and lets you mutate
the response on the way out; ADK's after_model_callback takes a
return-the-modified-response shape; CrewAI's guardrail uses a
validate-or-retry loop bounded by guardrail_max_retries; SK filters and
LangGraph nodes sit in the same call chain. The placement (last step before the answer
leaves the agent) is the same; the API ergonomics differ.
| Framework | Post-model interception | Source |
|---|---|---|
| LangChain v1 | Middleware wrap_model_call wraps the model invocation; the same
PIIMiddleware mechanism applies arbitrary policy via apply_to_output. |
libs/langchain_v1/langchain/agents/middleware/ |
| LangChain (classic) | BaseCallbackHandler exposes on_llm_end, on_chain_end, on_agent_finish; output mutation requires custom subclassing. |
libs/core/langchain_core/callbacks/base.py |
| LangGraph | Insert a node between the agent node and END; the node reads the latest
message from state and either passes it through or rewrites it. |
— |
| CrewAI | Task.guardrail (single) and Task.guardrails (list, line 255)
run after task execution; guardrail_max_retries (line 271) bounds retry
loops on guardrail failure. |
src/crewai/task.py:244 |
| Google ADK | after_model_callback on LlmAgent (line 391) returns the
modified or original response; plugin-level after_model_callback mirrors it
at the runtime layer. |
src/google/adk/agents/llm_agent.py:391 · plugins/base_plugin.py:220 |
| Semantic Kernel | function_invocation_filter wraps the call in middleware fashion: the
filter awaits next(context) and may mutate context.result on
the way out. |
filters/kernel_filters_extension.py |
A regulator asks: "Did your bot ever give medical advice to this customer?" "Did this user receive a refund recommendation they shouldn't have?" If the answer is "I don't know," that's a compliance failure regardless of what the bot actually did. Audit isn't optional in regulated industries — it's the ability to reconstruct, after the fact, every prompt, every tool call, and every response for any run. All five frameworks expose hooks rich enough to capture this; the shape of the hook determines what gets captured and where it lands.
Six months after launch, a customer disputes a charge and claims your support bot told them they'd be refunded. Without an audit trail, your team has no way to verify or refute the claim — the conversation is gone, the LLM responses unrecorded. With a full audit trail, you can pull the exact run by correlation ID and see every message: timestamps, the prompts that went to the model, the model's responses with token counts, every tool call with its arguments and results, and any retries or fallbacks. The dispute resolves quickly because you have evidence either way.
The hooks for capturing this are different in shape across the catalog. LangChain's
BaseCallbackHandler exposes the broadest lifecycle (17 methods covering
chat models, tools, chains, agents, retrievers, retry). ADK ships a 12-method
BasePlugin with the same coverage spread across plugin events. SK runs
three filter pipelines (function-invocation, prompt-rendering, auto-function-invocation).
CrewAI provides step_callback, task_callback, and
kickoff-level callbacks. LangGraph is unique: the same checkpointer that powers
durable resume (Chapter 4) also serves as the audit log — every super-step is recorded
and replayable via BaseCheckpointSaver.list().
| Framework | Trace surface | Source |
|---|---|---|
| LangChain | BaseCallbackHandler with lifecycle methods on_llm_start, on_llm_end, on_llm_error, on_chat_model_start, on_tool_start, on_tool_end, on_tool_error, on_chain_start, on_chain_end, on_chain_error, on_agent_action, on_agent_finish, on_retriever_start, on_retriever_end, on_retriever_error, on_retry, on_text. |
libs/core/langchain_core/callbacks/base.py |
| LangGraph | BaseCheckpointSaver.list(config, *, filter, before, limit) -> Iterator[CheckpointTuple] (line 234) returns the full sequence of super-step checkpoints with metadata, supplying audit trail and replay state from one surface. |
libs/checkpoint/langgraph/checkpoint/base/__init__.py:157 |
| CrewAI | Crew.step_callback: SerializableCallable | None (line 257) and
Crew.task_callback (line 261) plus before_kickoff_callbacks /
after_kickoff_callbacks; crewai/telemetry/ module ships an
event bus. |
src/crewai/crew.py:257 |
| Google ADK | BasePlugin exposes 12 lifecycle hooks: on_user_message_callback, before_run_callback, on_event_callback, after_run_callback, before_agent_callback, after_agent_callback, before_model_callback, after_model_callback, on_model_error_callback, before_tool_callback, after_tool_callback, on_tool_error_callback. |
src/google/adk/plugins/base_plugin.py:37 |
| Semantic Kernel | Three filter chains registered through kernel_filters_extension:
function_invocation_filters, prompt_rendering_filters,
auto_function_invocation_filters; each filter is invoked with a context
and a next awaitable. |
filters/kernel_filters_extension.py:33 |
Source-grounded observations on data and compliance surfaces.
None of the five frameworks ship semantic input validation for the user-facing message.
Validation lives at the tool boundary via Pydantic args_schema (LangChain,
CrewAI), annotated function signatures (ADK, SK), or whatever the developer-defined
state schema enforces (LangGraph).
PIIMiddleware at middleware/pii.py:32 ships built-in detectors
for email, credit-card, IP, MAC, URL, and four redaction strategies
("block", "redact", "mask", "hash").
The other four frameworks expose generic hooks where a developer-supplied detector runs.
ADK after_model_callback (line 391) returns the modified response. SK
filters and LangChain v1 middlewares wrap the call middleware-style with
next(context). CrewAI Task.guardrail + guardrails
(lines 244, 255) uses a validate-or-retry loop bounded by
guardrail_max_retries (line 271).
BaseCallbackHandler in langchain_core/callbacks/base.py defines
17 lifecycle methods covering chat models, tools, chains, agents, retrievers, and retry.
ADK's BasePlugin ships 12 hooks; SK runs three filter chains; CrewAI
provides step_callback, task_callback, and kickoff callbacks.
BaseCheckpointSaver.list(config, *, filter, before, limit) at line 234
yields CheckpointTuple entries with checkpoint, metadata, and config —
the same persistence layer that drives durable resume. No other framework conflates
state persistence with audit recording.
filters/kernel_filters_extension.py:33 registers
function_invocation_filters, prompt_rendering_filters, and
auto_function_invocation_filters as distinct lists. The signature is the
same — (context, next) middleware — but the trigger points are explicit.