Performance

What this looks like in practice

A user types a question into your chat UI. Without streaming, the request opens, the server holds the connection for 6 seconds while the model generates a 600-token answer, then the entire answer arrives at once. The user has stared at a loading dot the whole time. With streaming, the first token reaches the browser ~200 ms after the model starts, and the user sees the answer appear progressively while later tokens are still being generated. The total time hasn't changed; the perceived time has dropped to almost nothing.

Frameworks differ on three things: (1) what the API looks like in your code (async generator, callback, IAsyncEnumerable, Go channel, Tokio stream), (2) what's available to stream — just text tokens, or tool-call deltas and reasoning steps as well, and (3) what you have to build for HTTP streaming — every framework expects you to wire up SSE / WebSockets / chunked transfer yourself.

What this means in practice

All ten frameworks support streaming, so the practical question is how much glue you have to write between the framework's stream and the user's browser. Mastra is the standout — its Vercel AI SDK foundation ships SSE handlers tuned for Next.js / edge deployment, so you can ship a streaming chat UI in a couple of lines. ADK and Genkit Go give you ready-made HTTP servers with streaming built in. Everyone else expects you to write the HTTP transport yourself, which is straightforward but adds a layer to test.

On API shape, the pull-based async iterator (Python / TypeScript / Rust) is the easiest to reason about — you write a for await / while let Some(chunk) loop and you're done. LangChain4j's push-based handler interface is the outlier — it works, but reactive-stream conversions are needed to plug into the rest of the JVM ecosystem.

What this looks like in practice

At scale, a meaningful percentage of agent traffic is repeat or near-repeat requests — common questions, repeat document lookups, identical summarisation prompts. A well-placed response cache can drop the answer for those down to milliseconds and shave a real percentage off the model bill at the same time. The trade-off is staleness: a cached answer might be wrong if the underlying data has changed.

Two layers worth distinguishing. Exact-match cache — keyed by a hash of the prompt, returns the previous response if the prompt is byte-identical. Cheap and safe; only catches true repeats. Semantic cache — keyed by an embedding, returns the previous response if the new prompt is similar enough. Catches paraphrases too, but you have to tune the similarity threshold to avoid wrong answers.

What this means in practice

LangChain ships the most complete cache story — BaseCache with backends for nearly every store you'd consider, plus a separate semantic-cache implementation. LangGraph layers a per-node cache policy on top of the same backends. Mastra ships a TTL primitive aimed at TS / edge deployment. Everyone else delegates: you put your favourite cache (Redis with a hash-of-prompt key, Azure prompt caching, your edge runtime's KV store) in front of the model client and the agent framework never knows the difference.

The "delegate to your cache" pattern is fine in production but it costs you the ability to express cache rules at the agent level — e.g. "cache this tool response for 5 minutes but not this other one." The frameworks that ship a cache primitive let you express that without leaving the framework.

What this looks like in practice

A research agent gets asked to compare three vendors. The model responds with three tool calls at once: search("Vendor A pricing"), search("Vendor B pricing"), search("Vendor C pricing"). Each search takes ~1.5 seconds. With serial execution, the next model call doesn't fire for 4.5 seconds; with parallel execution, all three searches run together and the next model call fires after 1.5 seconds. Across the rest of the agent loop the user sees a 3-second improvement on every turn.

Parallel execution requires both concurrency (the runtime can issue overlapping I/O) and orchestration (the framework knows that multiple tool calls in one response are independent and can be fanned out). All ten frameworks have the concurrency; only some default to fanning out. The rest leave it to you to write the parallel orchestration with whatever async / threading primitive your runtime exposes.

What this means in practice

Three frameworks fan out tool calls automatically: LangChain (classic AgentExecutor on the async path), LangGraph (prebuilt ToolNode), and LangChain4j (configurable ExecutorService; effectively free on Java 21+ with virtual threads). Everywhere else, the language runtime can do it but the framework leaves the orchestration to you. For multi-tool agents that's a meaningful piece of glue you have to write — and easy to forget. If parallel-tool dispatch is critical to your latency budget, the LangChain family and LangChain4j get you there with the least effort.

What this looks like in practice

You're building a chat backend that handles 200 concurrent users. With an async-first framework on Python's asyncio, Node.js's event loop, .NET's Task, Go's goroutines, or Rust's Tokio, those 200 users share a small pool of OS threads — each agent call yields control while waiting for the model, and the runtime multiplexes. With a sync-first framework, each user occupies a thread for the duration of their request; you scale by adding threads, which costs memory.

The async-vs-sync distinction matters less than it used to thanks to runtime advances — Java 21+ virtual threads make sync code on the JVM behave like async without code changes. But the API shape still matters for ergonomics: an async API integrates cleanly with FastAPI, Next.js, ASP.NET Core, Tokio-based servers; a sync API requires a thread pool to bridge.

What this means in practice

Eight of ten frameworks are async-first or async-only. LangChain4j is the lone sync-first holdout — and on Java 21+ that's largely a non-issue thanks to virtual threads, which let blocking code scale like async without rewrites. LangChain (and by extension LangGraph) is the only Python framework that exposes both sync and async surfaces equally — useful when an existing sync codebase is being incrementally modernised.

What this looks like in practice

You have 1,000 customer reviews to classify. With a batch API, you call batch(reviews) and the framework dispatches them with controlled concurrency (configurable max in-flight) and returns the results in order. With no batch API, you write a loop that spawns 1,000 tasks, manages a semaphore to bound concurrency, handles partial failures, and reassembles results — which works but is repeat infrastructure work every team writes from scratch.

What this means in practice

Batching lives in only one part of the catalogue: LangChain's Runnable interface and LangGraph's reuse of it. If your workload is "process N items independently and gather results," LangChain saves you a clean piece of concurrency-management work. Every other framework expects you to write the bounded-parallel loop in your host language's idiom — well-trodden territory in each language, but not framework-managed.