An agent in production is opaque by default. The model picks tools and arguments based on reasoning you cannot directly inspect; tools call other services that themselves call models; a single user request fans out into a tree of LLM calls, retries, and database hits. Observability is what makes this tree legible — what tracing surfaces does each framework ship, how does it integrate with OpenTelemetry, what metrics does it emit, and what vendor dashboards can you point it at? This chapter walks through each surface in plain language, then catalogs the concrete primitives across ten frameworks spanning Python, TypeScript, .NET, Java, Go, and Rust.
The ten frameworks under comparison: LangChain, LangGraph, CrewAI, Google ADK, Microsoft Semantic Kernel (Python), Mastra (TypeScript), Microsoft Agent Framework (.NET), LangChain4j (Java), Genkit (Go), and Rig (Rust).
A trace is the structured record of an agent run — who called whom, in what order, and how long each step took. The shape of the trace determines what kind of debugging you can do after the fact: a flat list of "events" tells you what happened; a properly nested span hierarchy lets you see why a particular tool call took eight seconds when the rest of the run took two.
A user asks your agent a question. Behind the scenes the run produces: one outer "agent run" span, three "model call" spans (one per reasoning step), four "tool call" spans (search, lookup, search again, summarise), and one final "agent response" span. With a properly nested trace, your dashboard renders this as a waterfall: you can see the third model call took 6 seconds, and inside it the only thing that ran was a single tool call that took 5.8 seconds — so the slowdown was the downstream service, not the LLM. Without a properly nested trace, you have a flat list of timestamped events and have to reconstruct the parent-child relationship by hand.
Frameworks differ on three axes here: (1) do they emit spans at all out of the box, (2) what's the span hierarchy (flat events, or modelled run → step → tool / agent → model call), and (3) do they emit through OpenTelemetry's standard APIs so any OTel-compatible backend can ingest them, or through a framework-specific callback you have to translate yourself.
| Framework | What you get out of the box | Trace shape | Wiring effort | Source |
|---|---|---|---|---|
| LangChainPython | Callback events, not spans. To see a real trace you either use LangSmith (which translates callbacks to spans for you) or write your own translator. | Flat events; the tree is reconstructed downstream. | High — sign up for LangSmith, or write a custom callback-to-tracer adapter. | callbacks/base.py |
| LangGraphPython | Same as LangChain — callback events, no spans of its own. LangSmith remains the path of least resistance. | Flat events; super-step boundaries are implicit. | High — same as LangChain. | pregel/_messages.py |
| CrewAIPython | OTel spans named at each layer (Crew Execution, Task Execution, Tool Usage). Plug in any OTel-compatible backend. | Named-but-flat — uses tracer.start_span() without with blocks, so spans are emitted top-level rather than as nested children. |
Low — point an OTLP endpoint env var at your collector. | telemetry/telemetry.py |
| Google ADKPython | Production-ready OTel spans following Gen AI semantic conventions. | Naturally nested — agent → tool / model spans because the flow itself nests. | Low — wire an OTel exporter, or use the bundled Cloud Trace integration. | telemetry/tracing.py |
| Semantic KernelPython | OTel spans applied via decorators on agent and model methods. | Two-tier — agent layer + model layer; not as deeply nested as CrewAI / ADK. | Low — wire an OTel exporter; integrates cleanly with Application Insights. | utils/telemetry/agent_diagnostics/ |
| MastraTypeScript | Production-ready OTel spans across agent, tool, memory, and workflow layers. | Naturally nested — workflow steps and tool calls roll up under the agent run. | Low — pick from nine bundled vendor packages, or wire a generic OTel exporter. | core/src/observability/ |
| MS Agent Framework.NET | Production-ready spans via .NET's built-in ActivitySource (OTel-compatible). Token usage and latency emitted out of the box. |
Two-tier — outer agent activity wraps inner chat-client activities. | Low — register the OTel exporter in your ASP.NET Core startup. | OpenTelemetryAgent.cs |
| LangChain4jJava | No tracing in core — only listener events. To get spans you add the Micrometer adapter package. | Whatever Micrometer + your exporter produce; no agent-level hierarchy out of the box. | Medium — add the metrics module, configure Micrometer with an OTel registry. | chat/listener/ChatModelListener.java |
| GenkitGo | Production-ready OTel spans for flows, tool calls, and model calls. | Naturally nested per flow invocation. | Low — wire an OTel exporter, or use the bundled Google Cloud plugin. | core/tracing/tracing.go |
| RigRust | Structured log events from the tracing crate, but no agent-level spans. The developer instruments their own application code. |
Whatever the surrounding application's tracing-subscriber configures. |
High — set up tracing-subscriber, add the tracing-opentelemetry bridge, and decide where the spans go yourself. |
crates/rig-core/src/streaming.rs |
Six frameworks (CrewAI, ADK, Semantic Kernel, Mastra, MS Agent Framework, Genkit Go) give you usable spans on day one — pick a backend, wire an exporter, you're done. Three require meaningful additional work: LangChain and LangGraph need LangSmith or a custom translator to turn callback events into spans; LangChain4j needs the Micrometer adapter to turn listener events into anything observable. Rig sits at the far end — you get structured log events and nothing else; agent-level tracing is something you build.
Trace nesting matters less than it sounds when reading source code, but it matters a
lot when reading a dashboard. ADK, Mastra, and Genkit Go produce naturally nested
traces because their flows nest (ADK uses start_as_current_span inside
with blocks). SK and MS Agent Framework give you a clean two-tier view.
CrewAI emits spans named like a hierarchy (Crew Execution, Task Execution, Tool Usage)
but they aren't actually nested — they show up as separate top-level spans you can
filter by name. The flat-events + downstream-reassembly pattern (LangChain, LangGraph,
LangChain4j) works once your tracer is configured but means more steps before the
first useful waterfall renders.
OpenTelemetry is the de facto standard for distributed tracing — vendor-neutral APIs and
semantic conventions that let you swap dashboards without rewriting instrumentation. For
agents specifically, the Gen AI semantic-convention working group has standardised
attribute names like gen_ai.client.token.usage and gen_ai.operation.name.
Eight of the ten frameworks ship native or first-party OTel; two integrate via plugins.
Your platform team standardised on Datadog APM. A new team wants to ship an agent
feature and they're using a different framework than the rest of the company. With OTel
as a native dependency, the new team gets working dashboards on day one — Datadog's
OTel-compatible ingestion sees the spans, recognises the gen_ai.* attributes,
and renders token-usage and latency panels automatically. Without OTel, the team writes
a translator from the framework's callback interface to Datadog's API, maintains it
forever, and inevitably discovers that retries / nested calls / streaming aren't quite
right in the translation.
"Native OTel" means the framework imports the OTel library directly, declares spans through it, and follows the Gen AI semantic conventions. "First-party plugin" means the framework ships a separate package that bridges its own callback model to OTel. "Plugin" means OTel is reachable but the developer wires it. "Absent" means there's no official path and you have to write your own.
| Framework | Will any OTel backend work? | Standard attribute names? | What you have to do | Source |
|---|---|---|---|---|
| LangChainPython | Through LangSmith — LangSmith is the bridge between callbacks and OTel. | No — uses LangChain's own callback event names. | Use LangSmith, or write a callback-to-OTel translator yourself. | callbacks/base.py |
| LangGraphPython | Through LangSmith — same path as LangChain. | No. | Same as LangChain. | — |
| CrewAIPython | Yes — direct OTel SDK; ships a hardened OTLP exporter. | Mostly — token/latency follow standard names; adds CrewAI-specific span attrs. | Set CREWAI_TELEMETRY_BASE_URL to your OTel collector. |
telemetry/telemetry.py |
| Google ADKPython | Yes — direct OTel SDK. | Yes — imports Gen AI semantic conventions directly. | Wire any OTel exporter, or use the bundled Cloud Trace plugin. | telemetry/_experimental_semconv.py |
| Semantic KernelPython | Yes — direct OTel SDK. | Yes — emits standard token-usage and operation attributes via Meter. | Register an OTel exporter (Application Insights, or any OTel-compatible backend). | — |
| MastraTypeScript | Yes — direct @opentelemetry/api dependency. |
Partial — token attributes match standards; adds Mastra-specific attribute names that vendor exporters translate. | Pick from nine bundled vendor exporters, or wire a generic OTel exporter. | core/src/observability/ |
| MS Agent Framework.NET | Yes — uses .NET's built-in ActivitySource, which is OTel-compatible. |
Yes — Gen AI semantic conventions v1.37. | Register OTel in your ASP.NET Core startup. | OpenTelemetryConsts.cs |
| LangChain4jJava | Through Micrometer — add the metrics module to bridge listeners to OTel. | Yes — Micrometer ships OTelGenAiAttributes / OTelGenAiMetricName that match standards. |
Add the langchain4j-micrometer-metrics module and configure Micrometer with an OTel registry. |
langchain4j-micrometer-metrics/ |
| GenkitGo | Yes — direct Go OTel imports. | Partial — uses standard Go OTel attributes plus Genkit-specific ones. | Wire any OTel exporter, or use the bundled Google Cloud plugin. | core/tracing/tracing.go |
| RigRust | Through the tracing-opentelemetry bridge. |
No — agent-level attributes aren't modelled out of the box. | Set up tracing-subscriber, add the OTel bridge, define your own attribute schema. |
Cargo.toml |
Six frameworks (CrewAI, ADK, SK, Mastra, MS Agent Framework, Genkit Go) work directly with any OTel-compatible backend the moment you register an exporter. LangChain and LangGraph route through LangSmith; LangChain4j routes through Micrometer; Rig routes through the Rust tracing-opentelemetry bridge. All three of those routes work, but each adds a layer of translation you have to set up and maintain.
On standard attribute names — the ones a Datadog or Grafana Gen AI dashboard expects — ADK, MS Agent Framework, Semantic Kernel, and LangChain4j are the most aligned. Mastra, CrewAI, and Genkit Go follow the conventions partially and add their own. LangChain, LangGraph, and Rig don't follow them at all without extra work. If you're picking a framework specifically because you want existing dashboards to "just work," the first group is the safest starting point.
Tracing tells you what happened in one specific run. Metrics tell you what's been happening across millions of runs — token usage rates, p99 latency, error counts per tenant. Six of the ten frameworks emit first-class metrics out of the box; the others surface the data through traces or callbacks but don't aggregate it.
It's Tuesday morning. Your CFO asks why the OpenAI bill spiked 40% over the weekend.
With first-class metrics emission, you open Grafana, filter the
gen_ai.client.token.usage histogram by tenant and model, and within a minute
you see one customer ramped their usage of GPT-4o-mini by 5×. Without metrics emission,
you're querying every individual trace, summing token counts by hand, and presenting an
estimate three days later.
Out-of-the-box metrics matter because retrofitting them after the fact is laborious. A framework that ships metrics with sensible defaults (token usage as a histogram, not a counter; tagged with model and operation; emitted at the chat-client layer not the agent layer) gives you a running production view from day one.
| Framework | Token & latency dashboards day one? | Wiring effort | Source |
|---|---|---|---|
| LangChainPython | No — token counts are on individual response objects but never aggregated. | Use LangSmith for aggregated metrics, or build your own Meter from callbacks. | — |
| LangGraphPython | No — same as LangChain. | Same as LangChain. | — |
| CrewAIPython | Through span attributes — your backend computes the aggregates from spans. | Configure the OTLP endpoint; your backend (Datadog, Grafana, etc.) builds the dashboard. | telemetry/utils.py |
| Google ADKPython | Yes — first-class metric definitions for tool calls, model responses, latency. | Wire an OTel meter exporter. | telemetry/_metrics.py |
| Semantic KernelPython | Yes — Meter-based token-usage histograms and latency, with error type attributes. | Wire an OTel meter exporter. | utils/telemetry/model_diagnostics/ |
| MastraTypeScript | Yes — through bundled vendor packages (token usage, latency, custom counters). | Pick a vendor package (Arize, BrainTrust, Datadog) or wire a generic OTel meter. | core/src/observability/ |
| MS Agent Framework.NET | Yes — input/output token usage and latency that follow Gen AI semantic conventions. | Add OTel SDK in your DI startup. | OpenTelemetryAgent.cs |
| LangChain4jJava | Yes — gen_ai.client.token.usage histogram via Micrometer. |
Add the langchain4j-micrometer-metrics module; pick any Micrometer registry (Prometheus, Datadog, New Relic, etc.). |
MicrometerMetricsChatModelListener.java |
| GenkitGo | Yes — OTel metrics API for tool calls, model responses, latency. | Wire an OTel meter exporter. | core/tracing/telemetry.go |
| RigRust | No — no agent-level metrics. | Define your own Meter and emit token/latency from your tools. | — |
Six frameworks give you a token-usage and latency dashboard with minimal wiring (ADK, Semantic Kernel, Mastra, MS Agent Framework, LangChain4j, Genkit Go). CrewAI gets you most of the way there through span attributes — your backend does the aggregation, but the data is all present. LangChain, LangGraph, and Rig don't aggregate metrics out of the box: you either route through their canonical product (LangSmith for the LangChain family) or build the metrics layer yourself.
Logs are the third leg of the observability tripod, alongside traces and metrics. The
question isn't whether the framework writes logs — they all do — but whether those logs
are structured (key-value pairs you can filter on in your log search) or plain text you
have to grep. The shape of the logging layer also tells you how it integrates with your
host language's standard library: Python's logging, Go's slog,
.NET's ILogger, Rust's tracing, and so on.
A user reports their agent gave a wrong answer. With structured logs, you search your
log aggregator for {trace_id: "abc123"}, get every log line from that run
in chronological order, and see the model's tool selection and the tool's response. With
plain-text logs, you have an approximate timestamp and a transcript of free-form lines —
you spend half an hour grepping by the user's question phrase and stitching together
what happened.
The defining feature is correlation: every log line should carry a trace ID so it can
be joined back to the trace and the metrics. Frameworks that use the host language's
modern structured logging library (slog in Go, ILogger in .NET, tracing in Rust) get
this for free; frameworks that use older logging surfaces (Python's standard
logging module, Java's SLF4J without context) require the developer to
propagate trace IDs manually.
| Framework | Log format | Auto-correlated with traces? | Source |
|---|---|---|---|
| LangChainPython | Plain text via Python logging — you grep, not filter. |
No — you propagate trace IDs yourself. | — |
| LangGraphPython | Plain text via Python logging. |
No. | _internal/_log.py |
| CrewAIPython | Plain text via Python logging. |
No — telemetry module logs failures, but log lines aren't joined to span IDs. | utilities/logger.py |
| Google ADKPython | Structured via OTel logs (key-value records). | Yes — OTel context propagation joins logs to their spans. | telemetry/ |
| Semantic KernelPython | Structured via OTel logs; error attributes follow standard OTel error semantics. | Yes — joined to OTel spans. | utils/telemetry/ |
| MastraTypeScript | Structured custom logger with pluggable sinks. | Yes — uses @opentelemetry/api context to thread trace IDs. |
core/src/logger/logger.ts |
| MS Agent Framework.NET | Structured via .NET ILogger (key-value scopes). |
Yes — ILogger and ActivitySource share context automatically. |
— |
| LangChain4jJava | Whatever your host app's SLF4J / Log4j config produces. | No — depends on host MDC configuration; not framework-managed. | — |
| GenkitGo | Structured via Go log/slog (Go 1.21+). |
Yes — context-scoped logger via logger.FromContext(ctx). |
core/logger/ |
| RigRust | Structured via the tracing crate's events. |
Yes — tracing spans automatically carry trace IDs into nested events. |
— |
Six frameworks give you structured logs that join automatically to traces (ADK, SK,
Mastra, MS Agent Framework, Genkit Go, Rig), which means a "show me everything from
this run" query in your log search just works. LangChain, LangGraph, and CrewAI default
to Python's plain logging module and require manual trace-ID propagation.
LangChain4j defers entirely to host configuration — you get whatever Spring Boot or
Quarkus is set up to do.
"Open standards" sounds good in principle, but in production you almost always want a pre-built integration with whatever observability vendor your team has already adopted. The frameworks vary wildly here: Mastra ships nine vendor integrations in-tree; LangChain ships one (LangSmith); Rig ships none and expects you to use the OTel ecosystem.
Your team standardised on Langfuse for prompt experimentation and Datadog for production APM. With a framework that ships both integrations, the agent's traces show up in both dashboards on day one — the framework knows how to translate its internal spans into each vendor's format. With a framework that ships only its own dashboard (e.g. LangSmith for LangChain), getting traces into Langfuse means writing or maintaining a bridge yourself, and you'll discover edge cases like streaming or partial-tool-call spans aren't quite right.
The number and breadth of in-tree integrations is also a signal of where each framework's contributors come from. Mastra's nine vendors include both observability SaaS (Langfuse, LangSmith, Datadog) and AI-eval platforms (Arize, BrainTrust). LangChain leans on its own LangSmith product. CrewAI ships a configurable OTLP endpoint instead of dedicated integrations. Rig leans on the broader Rust + OTel ecosystem.
| Framework | Bundled integrations | If your vendor isn't bundled | Source |
|---|---|---|---|
| LangChainPython | LangSmith (native); LangChain Hub for prompts. | OTel via LangSmith, or write your own callback adapter. | callbacks/tracers/langchain.py |
| LangGraphPython | Inherits LangSmith from LangChain; LangSmith Deployment for hosted runtime. | Same as LangChain. | — |
| CrewAIPython | None bundled — only a configurable OTLP endpoint env var. | Point CREWAI_TELEMETRY_BASE_URL at any OTel-compatible collector. |
telemetry/telemetry.py |
| Google ADKPython | Google Cloud Trace plugin. | Wire any OTel exporter. | telemetry/google_cloud.py |
| Semantic KernelPython | Azure Application Insights (via OTel exporter). | Wire any OTel exporter. | — |
| MastraTypeScript | Nine: LangSmith, Langfuse, Datadog, Arize, BrainTrust, Arthur, PostHog, Sentry, ClickHouse. | Wire a generic OTel exporter. | core/src/observability/ |
| MS Agent Framework.NET | Application Insights via OTel exporter. | Wire any OTel exporter (the .NET ecosystem has one for almost everything). | — |
| LangChain4jJava | Micrometer's full vendor catalogue: Prometheus, Datadog, New Relic, AppDynamics, Dynatrace, etc. | Add a different Micrometer registry — virtually every monitoring vendor ships one. | langchain4j-micrometer-metrics/ |
| GenkitGo | Google Cloud (native plugin). | Wire any OTel exporter. | go/plugins/googlecloud/ |
| RigRust | None in-tree. | Use the Rust tracing-opentelemetry ecosystem to send to any OTel backend. |
— |
Mastra and LangChain4j are the two extremes — Mastra ships nine specific vendor integrations covering observability and AI-eval tooling; LangChain4j ships zero agent-specific integrations but inherits Micrometer's enormous vendor reach. Most frameworks ship one or two cloud-vendor-specific integrations (Cloud Trace, Application Insights, LangSmith) and expect you to wire generic OTel for anything else. CrewAI and Rig are at the other end: neither ships dedicated vendor integrations and both rely entirely on OTel as the integration layer.
Source-grounded observations on observability surfaces across the ten frameworks.
Six frameworks ship direct OTel SDK dependencies: CrewAI, ADK, Semantic Kernel,
Mastra, MS Agent Framework, Genkit Go. The other four reach OTel through a bridge —
LangChain4j via Micrometer, Rig via tracing-opentelemetry, LangChain
and LangGraph via LangSmith. Every framework gets there; only the wiring effort
differs.
Frameworks emitting gen_ai.client.token.usage and Gen AI attributes:
ADK (_experimental_semconv.py), MS Agent Framework
(OpenTelemetryConsts.cs v1.37), SK, LangChain4j (Micrometer
OTelGenAiAttributes). Means a Datadog or Grafana dashboard built once
works across these frameworks unchanged.
ADK and Genkit Go produce nested traces via start_as_current_span
inside with blocks; Mastra's flow steps roll up under the agent run.
MS Agent Framework's OpenTelemetryAgent gives a two-level wrap. CrewAI
emits descriptively named spans (Crew Execution, Task Execution, Tool Usage) but as
flat top-level events, not parent-child children. Rig has no agent-level span concept.
ADK (_metrics.py), Semantic Kernel (OTel Meter), Mastra (vendor packages),
MS Agent Framework (OpenTelemetryChatClient token + latency),
LangChain4j (MicrometerMetricsChatModelListener), and Genkit Go (OTel
metrics API). CrewAI surfaces the data through span attributes — your backend
aggregates. LangChain, LangGraph, and Rig don't aggregate at all.
Frameworks using modern structured logging by default: ADK and SK (OTel logs), Mastra
(custom structured logger), MS Agent Framework (ILogger), Genkit Go
(log/slog), Rig (tracing events). LangChain, LangGraph,
CrewAI default to plain Python logging; LangChain4j defers to host
SLF4J/Log4j config.
Mastra: LangSmith, Langfuse, Datadog, Arize, BrainTrust, Arthur, PostHog, Sentry, ClickHouse. LangChain ships one (LangSmith); ADK and Genkit ship Google Cloud; SK and MS Agent Framework ship Application Insights; CrewAI ships only a configurable OTLP endpoint; LangChain4j leans on Micrometer's vendor reach; Rig ships none in-tree.