These are the failures that originate before your code runs at all — the LLM provider is rate-limiting your account, having an outage, refusing the request on safety grounds, rejecting an expired API key, or dropping a streaming connection partway through. Your agent loop receives an exception or an unexpected response and has to decide what to do about it. This chapter walks through each failure in plain language, then catalogs the retry, fallback, and post-response hooks the five frameworks expose for these conditions.
Every LLM provider caps how often you can call them. When you exceed the cap they return
HTTP 429 — sometimes with a Retry-After header telling you how long
to wait. To your agent loop this looks like an exception thrown by the model client. The
framework's job is to decide whether to back off and try again, switch to a different model,
or give up and propagate the error to the caller.
Your agent is humming along when traffic spikes. The third tool-call cycle hits OpenAI's per-minute token budget for your account; the next request gets a 429. If your framework retries automatically with exponential backoff, the user sees a 2–8 second delay and the run finishes. If it doesn't retry, the exception bubbles up and the user gets a generic error — even though the provider explicitly told you "wait 4 seconds and try again."
Two things to check in any framework's behaviour. First, does it retry at all by
default? Some ship retry off and require explicit configuration. Second, does it
honour the Retry-After header? A framework whose backoff is hard-coded
will retry too aggressively when the provider says "wait longer", and too slowly when it
says "you can come back immediately." None of the five frameworks parse this header
automatically — the retry intervals come from each framework's own backoff configuration.
| Framework | Surface | Default | Source |
|---|---|---|---|
| LangChain | v1 ModelRetryMiddleware · v1 ModelFallbackMiddleware |
Off until added; ModelFallbackMiddleware walks an ordered list of alternative models on exception. |
libs/langchain_v1/langchain/agents/middleware/model_fallback.py |
| LangGraph | RetryPolicy(retry_on=…) attached to the agent node |
max_attempts=3, backoff_factor=2.0, initial_interval=0.5, max_interval=128.0, jitter=True — when attached. |
libs/langgraph/langgraph/types.py:406 |
| CrewAI | RPMController (preventive throttle via Agent.max_rpm) |
No retry of the 429 itself; pauses outgoing requests when the per-minute counter is reached. | lib/crewai/.../utilities/rpm_controller.py:12 |
| Google ADK | on_model_error_callback on LlmAgent |
Callback receives the exception; can return an alternative LlmResponse or re-raise. No automatic backoff. |
src/google/adk/agents/llm_agent.py:405 |
| Semantic Kernel | RetryMechanismBase (pluggable) |
Default is PassThroughWithoutRetry — no retry. |
python/semantic_kernel/reliability/{retry_mechanism_base,pass_through_without_retry}.py |
Sometimes the provider itself is having a bad day — its servers return 500,
502, 503. Retrying against the same provider may or may not help;
what helps reliably is sending the same request to a different provider. This is
where multi-model fallback matters: instead of failing the run, the framework walks down a
pre-configured list of alternative models until one responds.
OpenAI is down for the next eleven minutes. Your agent is in the middle of step 4 of a multi-tool task. With a plain retry policy, the framework retries against the same endpoint two or three times — each one fails — and then the run aborts. With multi-model fallback configured, the same exception triggers a switch to your second-choice model (Anthropic, Google, an OSS model on a different provider), the call succeeds, and the run continues. The user never sees the outage.
The trade-off is consistency. Different models behave differently — they format JSON slightly differently, follow tool schemas with different fidelity, hallucinate at different rates. A run that started on GPT-4o and finished on Claude Sonnet may produce a slightly different answer than one that ran end-to-end on either model. The fallback is a reliability mechanism, not an equivalence guarantee.
| Framework | Multi-provider fallback | Source |
|---|---|---|
| LangChain | v1 ModelFallbackMiddleware(first_model, *additional_models) — receives a list of alternative models; iterates on exception until success or list exhausted. |
libs/langchain_v1/langchain/agents/middleware/model_fallback.py |
| LangGraph | No first-class fallback. RetryPolicy targets the same node; switching models requires a custom node. |
— |
| CrewAI | Per-agent llm attribute is bound at agent construction; no built-in switch on failure. |
— |
| Google ADK | on_model_error_callback can return an LlmResponse from any source — including a different provider — but the routing logic is in the user's callback. |
src/google/adk/agents/llm_agent.py:405 |
| Semantic Kernel | Multiple chat services can be registered with the kernel and selected by service ID; no built-in switch-on-failure. | — |
Most security teams require API keys to be rotated regularly — every 30, 60, or 90 days.
When a key expires, the next request returns HTTP 401. To the agent loop this
looks like a generic exception from the model client, indistinguishable from a transient
network error. There's no provider convention that says "this is auth, refresh your
credentials" — the agent just sees a failed call.
It's 9:00 AM on a Monday. Your platform team rotated the OpenAI key over the weekend, but the secrets-manager refresh hasn't propagated to all your agent pods yet. The first request from a stale pod gets a 401. With no auth-aware handling, the framework's retry policy kicks in and retries the same request with the same expired key — three times, with backoff — and finally raises. The user gets an error; the on-call engineer gets a page. With explicit auth-rotation logic (a credential refresh hook in front of the model client), the 401 triggers a fetch of the new key and the retry succeeds silently.
None of the five frameworks ships a dedicated handler for this. The 401 is treated as whatever exception the underlying LLM client raises and flows through the same retry or error-callback paths as any other transient error. Credential refresh is implemented one layer below the framework — in the model client wrapper or the platform's secrets layer.
When a provider's safety system blocks a request, the API call doesn't fail in the programmatic sense — you get back a perfectly valid response. The model just says "I can't help with that" instead of doing the task. Because nothing throws, the agent loop continues as if everything succeeded. Detecting a refusal means inspecting the model's response after it returns, before it propagates as the agent's answer.
A user asks your customer-service agent for help with a billing dispute. The agent calls a "lookup_account" tool, gets back the customer's record, and on the next model call the provider's safety filter triggers — perhaps the customer's last name resembles a sanctions-list entry. The model returns: "I'm not able to assist with this request." The agent loop sees a successful response, doesn't recognise it as a refusal, and forwards the apology to the user as the final answer. The user concludes the bot is broken.
Catching this requires a post-response hook that reads the actual text and either
substitutes a useful fallback message, escalates to a human, or retries with a different
prompt. Every framework provides such a hook — a middleware in LangChain v1, a node in
LangGraph, a guardrail in CrewAI, an after_model_callback in ADK, a filter in
SK — but the detection logic itself (refusal-classifier, regex, or LLM judge) is
developer-supplied.
| Framework | Post-LLM hook | Source |
|---|---|---|
| LangChain | v1 middleware — wrap_model_call override on an AgentMiddleware sees the model's response object before it's used. |
libs/langchain_v1/langchain/agents/middleware/types.py |
| LangGraph | A node placed after the agent node sees the last AIMessage; conditional edges can route on its content. |
libs/prebuilt/langgraph/prebuilt/chat_agent_executor.py |
| CrewAI | Task.guardrail / Task.guardrails — callable validators run on each task's output before it propagates; failure causes re-execution. |
lib/crewai/.../task.py:244, 255 |
| Google ADK | after_model_callback on LlmAgent — receives the LlmResponse, can return an alternative response. |
src/google/adk/agents/llm_agent.py:391 |
| Semantic Kernel | Auto-function-invocation filter sees the model's function_call_content; for a refusal (no function call) the filter chain doesn't fire. |
python/semantic_kernel/filters/auto_function_invocation/auto_function_invocation_context.py |
Streaming responses are how chat UIs feel responsive — tokens arrive one at a time over a long-lived HTTP connection. That connection can drop midway: a load-balancer timeout, a transient network blip, a provider-side hiccup. When that happens you have most of an answer and no way to ask the model to "continue from where you left off" without losing context. Recovery means re-invoking the model with the partial output already in scope.
Your agent is generating a 600-word summary as a streaming response. After 350 words the connection drops. With no resumption support, the framework raises an exception and the user sees a half-finished answer with an error overlay. With resumption support, the framework reconnects to the model — typically using a session handle the provider issued with the original request — and the model picks up where the stream cut off. The user sees a brief stutter and then the rest of the answer.
The hard part is that resumption is a provider-coupled capability: the model has
to support a session handle, and the framework has to know how to use it. Most providers
don't expose this; Google's Live API does. Of the five frameworks, only ADK ships a
reconnect loop wired to that handle (BaseLlmFlow.run_live with
session_resumption_handle). For everyone else, a streaming disconnect raises
and recovery is the caller's problem.
| Framework | Surface | Source |
|---|---|---|
| LangChain | No native resumption. The exception propagates to the caller. | — |
| LangGraph | Checkpointer captures messages already received as state; resumption replays from the last checkpoint, the model regenerates the truncated message. | libs/langgraph/langgraph/checkpoint/ |
| CrewAI | No native resumption. step_callback sees partial output as it arrives. |
lib/crewai/.../agent/core.py · step_callback |
| Google ADK | BaseLlmFlow.run_live implements automatic reconnect using session_resumption_handle: on disconnect the loop increments an attempt counter and reconnects with the saved handle. |
src/google/adk/flows/llm_flows/base_llm_flow.py:475–517 |
| Semantic Kernel | Streaming is exposed via invoke_stream; disconnects raise and the caller decides how to recover. |
python/semantic_kernel/agents/chat_completion/chat_completion_agent.py · invoke_stream |
Every model call has a maximum output length — the max_tokens setting plus
whatever ceiling the model itself imposes. When the model hits that ceiling mid-sentence,
it stops. The response is technically complete (no exception, no error), but the content
is half-finished. Providers signal this by setting finish_reason="length" on
the response — but the agent loop has to actually look at that field to notice.
A user asks your agent to draft an email. The model generates 4,096 tokens — exactly the configured ceiling — and the response cuts off in the middle of the closing paragraph. The agent loop sees a successful response and forwards the half-email to the user. The user has no idea the model wanted to keep going; they just see a weird, incomplete message and assume the agent broke.
Detecting this is one line of code: check whether finish_reason equals
"length". Acting on it is harder. The "right" recovery is to issue a
continuation call where the prompt now includes the partial output and asks the model
to keep going — but the agent loop doesn't do this for you. Every framework surfaces
finish_reason on its response object; none of them auto-continues.
| Framework | Where finish_reason is exposed |
|---|---|
| LangChain | On AIMessage.response_metadata; not acted on by the agent loop. |
| LangGraph | Same — flows through state metadata; no built-in handler. |
| CrewAI | Not surfaced in the standard executor flow. |
| Google ADK | Available on LlmResponse; visible in after_model_callback. |
| Semantic Kernel | Available on FunctionResult.metadata. |
Concrete observations from the source on what each framework exposes for provider failures.
ModelFallbackMiddleware takes (first_model, *additional_models)
and walks the list on exception. The other four frameworks expose hooks where a custom
callback could route to a different model, but no first-class fallback class is shipped.
LangGraph RetryPolicy: max_attempts=3, backoff_factor=2.0,
jitter=True. LangChain v1 ModelRetryMiddleware: opt-in, no
attempt cap by default. SK reliability: PassThroughWithoutRetry. ADK: per-callback,
no automatic backoff. CrewAI: RPMController throttles outbound rather than retries.
BaseLlmFlow.run_live at base_llm_flow.py:475–517 reconnects using a
session_resumption_handle and an attempt counter. No equivalent
ships in LangChain, LangGraph, CrewAI, or Semantic Kernel — disconnects raise and recovery
is the caller's responsibility.
A content-moderation refusal is a successful API response. Each framework exposes a
post-LLM hook where the response can be inspected: LangChain v1 middleware
wrap_model_call, LangGraph post-agent node, CrewAI Task.guardrail,
ADK after_model_callback, SK auto-function-invocation filter.
A 401 from the provider surfaces as whatever exception the underlying LLM client raises and is routed through the same retry / error-callback paths as any other transient error. No dedicated mechanism for credential rotation appears in any of the five repos.
Every framework surfaces finish_reason="length" on its response object
(AIMessage.response_metadata in LangChain/LangGraph, LlmResponse
in ADK, FunctionResult.metadata in SK). None of the agent loops continues
generation automatically when truncation is detected.