AI agent observability is now a deployment requirement

Instrument the run before you change the orchestrator

AI agent observability is now part of deployment, not a later analytics project. If an agent can call tools, spend tokens, cross systems and make decisions, it needs traceability before it needs another orchestration layer.

The practical order is simple. Start with session traces, span-level work, tool calls, model requests, token usage and errors. Only then compare orchestration frameworks. AWS makes the same deployment point in its August 2026 AgentCore Observability walkthrough: agents may run on Amazon EKS, ECS, Lambda, on-premises, Google Cloud, Azure or developer machines, yet still need one telemetry route into the same dashboard for traces, metrics and logs via ADOT and OTLP (AWS, 2026).

This matters for AI employees and custom automation because the failure mode is rarely a single bad answer. It is usually a chain. A support agent retrieved the wrong policy. A scheduling agent called the calendar API twice. A finance workflow used a larger model than expected. A sales agent passed a malformed object to the CRM. Logs alone show fragments. Traces show the run.

What agent tracing must capture first

A production agent run has more moving parts than a normal API request. The minimum useful trace links the business session to each model call, retrieval step, tool call and handoff.

OpenTelemetry’s GenAI conventions now define separate signals for events, exceptions, metrics, model spans and agent spans, with provider-specific conventions for Anthropic, Azure AI Inference, AWS Bedrock and OpenAI (OpenTelemetry GenAI conventions). The status is still Development, so teams should expect field names to move. The direction is clear: agents need standardised telemetry, not screenshots of chat transcripts.

Token data deserves special attention. The GenAI metrics model defines gen_ai.client.token.usage as a histogram for input and output tokens, and says billable tokens must be reported when systems expose both used and billable token counts (OpenTelemetry metrics model). That wording matters because cost controls based on estimated tokens can drift from invoices.

OpenTelemetry is the safer default between clouds

Agent platforms are adding their own dashboards. That is useful, but the telemetry format should not depend on where the agent happens to run this month.

AWS’s example sends telemetry from non-AWS environments into AgentCore Observability using the AWS Distro for OpenTelemetry running in-process, then exports to the CloudWatch OTLP endpoint with SigV4 authentication (AWS AgentCore Observability). The interesting part is not only the AWS destination. It is the assumption that agent telemetry must cross deployment boundaries.

For businesses, this is the same architectural lesson behind controlled execution in enterprise AI. A workflow that touches customers, payments, patient records or legal documents needs boundaries and evidence, not just a chat UI. Our related piece on controlled enterprise AI adoption covers the governance side. Observability supplies the operational record.

OpenTelemetry AI agents instrumentation is not complete enough to remove vendor differences. The GenAI repo is explicit that the conventions are in Development (OpenTelemetry GenAI conventions). Still, it gives engineering teams a common spine: trace ID, span hierarchy, provider, model, operation, usage, errors and events.

That is enough to avoid the worst lock-in. Keep the canonical trace in OpenTelemetry shape, then export it to LangSmith, Phoenix, CloudWatch, Datadog, Grafana or another backend as needed.

Observability tools are converging on the same model

The agent observability market looks crowded, but the primitives are becoming consistent.

LangSmith describes observability as visibility from individual traces to production-wide performance metrics, with integrations for OpenAI, Anthropic, CrewAI, Vercel AI SDK and Pydantic AI (LangSmith docs). Phoenix defines a trace as a record of a single application run, broken into spans that show how agents, tasks and tools executed (Phoenix tracing guide). OpenAI’s Agents SDK includes tracing by default for LLM generations, tool calls, handoffs, guardrails and custom events, and uses group_id to link multiple traces from the same conversation (OpenAI Agents SDK tracing).

Those are independent implementations, but they point at the same deployment requirement. You need a run graph.

The choice is not “LangSmith or Phoenix or CloudWatch?” It is “where should the system of record live, and what needs to be exportable?” A team building custom LLM pipelines and automation may start with a managed trace viewer because it shortens debugging. A company with strict cloud controls may route OTLP into its existing telemetry stack. A product team with complex dashboards may expose observability data inside a custom internal platform.

The first metrics should answer business questions

LLM observability becomes useful when technical spans map to operating questions.

Start with five questions:

1. Which agent sessions fail, and at which span?
2. Which tools are slow, flaky or retried?
3. Which model calls dominate cost?
4. Which retrieval sources influence bad answers?
5. Which user intents require human escalation?

These questions produce better instrumentation than a blanket “log everything” policy. They also limit privacy exposure.

Cost is a good example. Anthropic’s pricing page lists Sonnet 5 at $2 per million input tokens and $10 per million output tokens, and Haiku 4.5 at $1 per million input tokens and $5 per million output tokens (Anthropic pricing). If one agent quietly shifts from Haiku 4.5 to Sonnet 5 for routine classification, output-token cost doubles before any infrastructure graph turns red. A trace with model and token attributes catches the change at run level.

Latency has the same pattern. A customer-facing agent may feel slow because the model is slow, because retrieval is slow, or because a business system returns in four seconds. Span timing separates those causes. Without it, teams replace the model when the CRM connector was the bottleneck.

What not to instrument first

Do not begin with a new orchestration layer just because debugging is painful. Orchestration changes control flow. Observability explains the existing control flow. Those are different interventions.

Also avoid storing raw prompts and outputs by default. OpenTelemetry’s GenAI content fields are opt-in, including system instructions, input messages, output messages and tool definitions (OpenTelemetry spans model). That is a useful privacy signal. Many failures can be diagnosed with IDs, timings, model names, token counts, tool names, status codes and redacted argument schemas.

For regulated sectors such as healthcare, finance and legal, the default should be metadata-first traces with controlled sampling of content. Keep retention short for sensitive payloads. Store document IDs and policy versions. Capture the fact that an output was evaluated, not necessarily the full customer record inside the trace backend.

A practical deployment order

A useful first deployment does not need to be large. It needs to be complete enough that a failed run can be reconstructed.

Instrument in this order:

1. Assign a stable trace ID to every agent run.
2. Add a session or group ID for multi-turn work.
3. Wrap every model call in a span with provider, model, latency, error and token data.
4. Wrap every tool call in a span with tool name, target system, status and retry count.
5. Add retrieval spans with index, document IDs, scores and knowledge-base version.
6. Emit evaluation events for safety checks, task success and human review.
7. Export through OpenTelemetry where possible, even if the first viewer is vendor-specific.
8. Define retention, redaction and sampling rules before production traffic.

This gives operations teams the data they need to decide whether an agent is safe to expand. It also gives finance a usable cost model. Most importantly, it gives engineers a way to fix the system without guessing.

What this means if you are deciding whether to act

If your agent only drafts internal text and a person checks every output, basic application logs may be enough for now. If it calls business systems, handles customers, changes records or runs across more than one environment, AI agent observability should be part of the deployment scope.

The buying decision should focus less on dashboards and more on evidence. Ask whether the implementation captures trace IDs, span hierarchy, model calls, tool calls, retrieval context, token usage, errors and evaluation events. Ask whether those signals can move through OpenTelemetry. Ask how sensitive content is redacted, sampled and retained.

For most businesses, the right first step is not a platform migration. It is an instrumentation pass over one production workflow, followed by a review of real traces from failed and successful runs. Cloudex can help scope that pass for an AI employee, automation pipeline or internal tool. Start with Cloudex AI Solutions or contact the team through /contact.