62% of production LLM failures go undetected by HTTP monitoring for over 48 hours. A 200 OK status code means the server responded —not that the answer was correct, the retrieved context was fresh, or the agent didn’t loop through 47 redundant tool calls. Your dashboard is green while hallucinations reach paying customers and a prompt template regression poisons every response since Tuesday’s deploy.
This guide builds a three-layer observability stack —Base OTel, GenAI semantic conventions, OpenInference span kinds —that turns every user request into a forensic trace tree. One function call registers it. Tail-based sampling retains every failure trace without blowing your storage budget.
Why Your Monitoring Dashboard Is Blind to LLM Failures
Why Standard APM Fails for LLM Applications
HTTP 200 does not mean “correct answer.” It means the server responded. The OpenTelemetry project provides the wire format and collector infrastructure —but LLM applications need semantic conventions on top of that foundation. Standard APM fails because LLM applications fail in ways that HTTP status codes cannot express:
- Hallucination. The model returned a confident, well-formatted answer. Every fact in it is wrong. HTTP status: 200.
- Silent refusal. The model should have answered. It refused —politely, in perfect JSON. HTTP status: 200.
- Cost spike. One request generated 32,000 thinking tokens because reasoning effort was set to “high” for a simple classification task. HTTP status: 200. No dashboard shows the thinking token count.
You don’t need to see “the request succeeded.” You need to see “the retrieved context relevance was 0.3, which caused a generation faithfulness score of 0.4, which means the user got a wrong answer despite everything looking green.” Tracing embedding calls alongside chat completions is essential when retrieval quality drops —instrumenting both endpoints under one trace reveals the full picture.
The Three-Layer Architecture
These are not three options. You need all three.
Layer 1: Base OpenTelemetry. The wire format (OTLP), the context propagation (W3C trace context), the collector pipeline. This is the substrate —every observability backend speaks OTLP. Every microservice emits OTel spans. Without this layer, you’re locked into a vendor’s proprietary format. With it, you can switch backends without re-instrumenting a single line.
Layer 2: OTel-GenAI Semantic Conventions. Standardized span attributes for LLM operations: gen_ai.system (which provider), gen_ai.request.model (which model version), gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (token consumption), gen_ai.operation.name (chat vs. embeddings vs. tool execution). Without this layer, all your LLM spans look identical —you can’t tell a chat completion from an embedding call.
Layer 3: OpenInference Span Kinds. Fourteen LLM-aware span types that GenAI conventions don’t yet enumerate: LLM, CHAIN, RETRIEVER, TOOL, EMBEDDING, AGENT, RERANKER, GUARDRAIL, EVALUATOR, CONVERSATION, VECTOR_DB, and more. Without this layer, your RAG pipeline’s trace is a flat list of HTTP calls. With it, you see EMBEDDING —RETRIEVER —RERANKER —LLM as distinct stages —and you know exactly which stage added the 800ms latency spike.
The Trace Tree as the Minimum Unit of Understanding
An isolated log line cannot diagnose an LLM problem. The question is never “what did this one API call return?” It’s “what was the full causal chain: user query —intent classification —retrieved chunks —reranker scores —final prompt —LLM response —evaluation score?” A trace tree captures that chain. One trace = one user interaction’s complete forensic record.
Why Observability Is Non-Negotiable
Cost Without Visibility
An unmonitored agent loop burned $5,000 over a weekend at one deployment I investigated. The agent hit a tool-call loop Friday evening —search_kb("return policy") returned “no results,” so the agent called search_kb("return policy EU"), then search_kb("return policy Europe"), then 44 more variations —each one a full LLM API call with context. Nobody noticed until Monday’s billing alert.
Per-span cost attribution —gen_ai.cost.input, gen_ai.cost.output, gen_ai.cost.total attached to every LLM span —catches this in minutes, not days. Set an alert: if any single trace exceeds $2.00 in accumulated API costs, trigger a notification. The cost of the alerting infrastructure is less than one weekend incident.
Cost attribution is the detection layer. For the prevention layer — caching strategies, model tier selection, and batch processing — see our cost optimization strategies guide.
Quality Without Visibility
A model migration from GPT-4o to GPT-5.5 looked clean on HTTP dashboards. Latency improved 15%. Error rate unchanged. What the dashboard didn’t show: the new model handled structured output slightly differently —null appeared in three fields that were never null before. The format was valid JSON. The business logic consuming it broke silently.
Span-attached evaluation scores catch this in five minutes. Your eval rubric runs against production traces continuously. Any drop in faithfulness, context adherence, or format compliance triggers an alert —before users notice, before support tickets accumulate, before the quarter’s quality metrics take a hit. For the CI/CD pipeline that runs these evaluations pre-deployment, see our testing and evaluation guide.
Compliance Without Visibility
SOC 2 Type II auditors ask: “Show us the complete API call record for user X on date Y —what data was sent, what model processed it, what was returned?” If your LLM API calls don’t produce structured traces with the right retention policy, the answer is: “we can’t.” That’s not a finding. That’s a qualification —a much more expensive word in an audit report.
Structured traces satisfy the audit trail requirement. For the access control and data handling controls that complete SOC 2 readiness, structured access logging and retention policies aligned with your compliance framework close the gap.
How to Set Up LLM Observability
Step 1: One-Call Registration
One function call in your startup module. That’s it.
from fi_instrumentation import register, ProjectType, SemanticConvention
trace_provider = register(
project_name="checkout_assistant",
project_type=ProjectType.OBSERVE,
semantic_convention=SemanticConvention.OPENINFERENCE,
metadata={"git_sha": "abc123", "environment": "production"},
batch=True,
)
from openinference.instrumentation.openai import OpenAIInstrumentor
from openinference.instrumentation.langchain import LangChainInstrumentor
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)
LangChainInstrumentor().instrument(tracer_provider=trace_provider)The semantic_convention parameter is the key architectural decision here. Set it to OPENINFERENCE, OTEL_GENAI, or OPENLLMETRY —your instrumentation code doesn’t change. Only the attribute naming on emitted spans changes. This matters when you switch observability backends: Datadog expects one convention, Langfuse another, SigNoz a third. One config switch. No code changes.
Coverage: 50+ Python frameworks, 39 TypeScript packages, 24 Java modules, C#. OpenAI, Anthropic, LangChain, LlamaIndex, Haystack, DSPy —all auto-instrumented.
Step 2: Span Enrichment
A span without user_id, session_id, and prompt_version is an orphan. You can see what happened but not to whom or with which configuration.
from contextlib import contextmanager
@contextmanager
def using_attributes(**kwargs):
# Attach attributes to the current span; all child spans inherit them
with tracer.start_as_current_span("user-interaction") as span:
for key, value in kwargs.items():
span.set_attribute(key, value)
yield span
with using_attributes(
session_id="sess_a1b2c3",
user_id="user_42",
metadata={
"prompt_template": "checkout_v3.2",
"ab_bucket": "treatment",
"feature_flag": "new_upsell_logic"
}
):
response = client.chat.completions.create(...)The minimum attribute set for every trace: session.id, user.id, prompt.version, feature.id, tenant.id. Without these, your trace data can’t answer “did the checkout_v3.2 prompt cause the regression, or did the model version change cause it?” —which is the first question you’ll ask during an incident.
Step 3: Eval-as-Span-Attribute
Evaluation scores that live in a separate database, requiring a manual join against trace IDs, are evaluation scores nobody looks at. EvalTag fixes this: declare evaluators at registration time, and their scores write directly to the original span as gen_ai.evaluation.<rubric>.score attributes —with zero added request latency.
register(
project_name="checkout_assistant",
evaluators=[
"GROUNDEDNESS", # Are claims supported by retrieved context?
"CONTEXT_ADHERENCE", # Is the answer using the provided context?
"PROMPT_INJECTION", # Is there an injection attempt in the input?
"TASK_COMPLETION", # Did the model complete the requested task?
],
)The evaluator runs asynchronously —the user gets their response without waiting for scoring. The score appears on the span within seconds. Your dashboard refreshes. If GROUNDEDNESS drops below 0.7 across a 5-minute window, your alert fires. No separate eval pipeline. No manual correlation. One trace tree, one source of truth.
Step 4: Tail-Based Sampling
Head-based sampling —“keep 10% of all traces at random” —is the default in most APM setups. It’s catastrophically wrong for LLM applications. Failures are rare. Cost outliers are rare. Low-quality outputs are rare. A uniform 10% random sample throws away 90% of the traces that actually matter.
Tail-based sampling flips this: the collector sees the complete trace before deciding whether to keep it. The retention rule:
- Keep 100% of traces with errors (5xx, timeout, rate limit)
- Keep 100% of traces with any eval score below threshold
- Keep 100% of traces with cost above p95
- Keep 1-10% of clean, fast, correct traces
Your storage costs stay controlled. The traces you actually need for debugging stay available.
Step 5: Three-Tier Retention
Don’t pay ClickHouse prices for regulatory compliance data you access once a year.
| Tier | Storage | Duration | Contents |
|---|---|---|---|
| Hot | ClickHouse / InfluxDB | 14-30 days | All retained traces —live dashboards and alerts |
| Warm | Columnar (S3/Parquet) | 90 days | Full traces —compliance and retrospective debugging |
| Cold | Object storage (S3 Glacier) | 1-7 years | Compressed traces —regulatory retention |
The hot tier is for operations. The warm tier is for debugging last quarter’s incident. The cold tier is for auditors. Each tier costs roughly an order of magnitude less than the one above it.
Production Trace Patterns for Specific Workloads
RAG Trace Topology
A flat trace of a RAG pipeline is useless. You need to see each stage as a distinct span:
EMBEDDING span [model: text-embedding-3-small, tokens: 450, latency: 32ms]
→ RETRIEVER span [vector_db: pgvector, top_k: 20, index: hnsw, latency: 8ms]
→ RERANKER span [model: bge-reranker-large, candidates: 20→5, latency: 45ms]
→ LLM span [model: gpt-4o, input_tokens: 2840, output_tokens: 380, latency: 1.2s]When retrieval quality drops, you look at the RETRIEVER span —are the similarity scores low? Check if the embedding model drifted. When generation quality drops but retrieval looks fine, you look at the LLM span —are the retrieved chunks being fed in the right order? Is the system prompt intact? The topology tells you where to look, not just that something is wrong. For the full RAG pipeline architecture behind this trace model, see our RAG production guide.
Agent Trace Topology
A 6-node LangGraph agent traced flat is a debugging nightmare. You see 100 spans. You don’t know which node triggered which tool, which tool failed, or where the loop started.
The correct topology:
Root: AGENT span [session_id, user_id, task]
—Reasoning span: "plan to answer user's question about order status"
—TOOL span: lookup_order(order_id="ORD-12345") [latency: 180ms, status: success]
—Reasoning span: "order found, now check shipping"
—TOOL span: track_shipment(tracking_id="ZYX-987") [latency: 340ms, status: success]
—LLM span: synthesis [model: claude-sonnet-4, input_tokens: 1520, output_tokens: 210]For LangGraph specifically, add langgraph.node.name, langgraph.node.type, and conditional edge events to every span. Without these, when your 6-node agent gets stuck in a loop, you can’t tell which node is the problem. With them, the trace renders as a topology graph —and the looping node is visually obvious. For the multi-agent orchestration patterns that generate these traces, see our multi-agent architecture guide.
Cost Attribution Per Span
Every LLM span carries gen_ai.cost.input, gen_ai.cost.output, gen_ai.cost.cache_read, and gen_ai.cost.total. At the gateway level, maintain hierarchical budgets: org —team —user —session. Generate monthly cost breakdowns by team, by model, by use case —automatically, from trace data. No manual billing reconciliation. No “the AI line item is a black box.” For configuring per-user and per-endpoint request throttling to prevent runaway agent loops from blowing your budget, see the rate limits documentation.
Observability Mistakes That Cost You in Production
Vendor-SDK-Only Instrumentation
You instrumented with Datadog’s native SDK because it was the fastest path to a dashboard. Six months later, your team wants to evaluate Langfuse for LLM-specific tracing. Every call site needs re-instrumentation.
Fix: Instrument with OTel. It’s the abstraction layer. Switch backends by changing the exporter config, not the instrumentation code. Vendor SDKs are output targets, not instrumentation frameworks.
Uniform Random Sampling
Your sampling rate is 10%. You’re randomly discarding 90% of your traces —including the one where a user got charged twice because the agent looped, the one where a prompt injection attempt almost succeeded, and the one where a single request consumed $18 in thinking tokens.
Fix: Tail-based sampling. The collector sees the full trace, then decides. Failures, cost outliers, and low-quality outputs: keep 100%. Clean traces: keep a small percentage for baseline comparison.
No LangGraph Topology in Agent Traces
You deployed a multi-node agent. Your traces show 87 spans per user request in a flat list. Your agent got stuck in a loop last Tuesday. It took three hours to identify which node was the culprit —because “87 flat spans” doesn’t tell you the execution graph.
Fix: langgraph.node.name and langgraph.node.type on every span. Conditional edge events. Your trace viewer should render the agent as a graph, not a list.
Skipping Gateway-Emitted Spans
You trace your application code meticulously. But you access LLMs through a unified API platform —and the gateway spans (provider-side latency, routing decision, cache hit/miss, fallback trigger) are invisible to your application tracer. When latency spikes, you can’t tell if it’s your code, the gateway, or the provider.
Fix: Gateway spans are part of your trace. If your API platform emits OTel spans, configure your collector to ingest them. A unified API endpoint means one integration point for gateway observability —set it up once, every model call is covered.
For deployment patterns that pair observability with fallback chains, retry logic, and cost monitoring across models, see the production optimization guide.
FAQ
Do I need all three layers (Base OTel + GenAI + OpenInference)?
Yes. Base OTel prevents vendor lock-in. GenAI semantic conventions standardize model-specific attributes (token counts, model identity) so your dashboards don’t break when you switch providers. OpenInference span kinds give you LLM-aware topology —without them, every span is “an API call” and you can’t distinguish retrieval from generation from tool execution.
What’s the performance overhead?
Span creation and attribute setting: less than 1% latency impact. Evaluators (EvalTag): zero impact on user-facing latency —they run asynchronously after the response is sent. Tail-based sampling: runs in the collector, not in your application process. Total overhead is negligible compared to the 200ms-10s latency of the LLM API calls themselves. For cutting input costs on repeated traces, prompt caching strategies pair naturally with span-level cost attribution.
Self-host or SaaS for observability?
Self-host: SigNoz (OTel-native, GenAI dashboards) + ClickHouse + Grafana. Good if you already run OTel infrastructure. SaaS: Langfuse Cloud (trace-first, tuned ClickHouse), Datadog LLM Observability. Good if you want a dashboard in 10 minutes. The OTel abstraction means you can start with SaaS and migrate to self-hosted without re-instrumenting.
How do I redact PII from LLM traces?
Redact at the collector —not in application code. Regex patterns for credit card numbers, SSNs, and email addresses. NER classifiers for names and physical addresses. Custom rules for API keys and access tokens. The principle: raw secrets never cross your network boundary. They get stripped in the collector processor before export to any external backend.
What’s the simplest path to unified LLM observability?
One integration point. When every model call —GPT, Claude, Gemini, DeepSeek —flows through one API endpoint, you configure OTel export once. Gateway-emitted spans (provider-side latency, routing decisions, cache hit rates, fallback triggers) arrive pre-formatted alongside your application spans. No stitching together traces from three different provider SDKs. No wondering whether the latency spike is in your code, the gateway, or the provider —because all three are in the same trace tree. Start with one API key for all models and see unified traces on the TokSpan platform.
Observability for LLM applications is not a “mature organization” concern. It’s a “first production deployment” concern. The cost of not having it is measured in weekend incidents, silent quality regressions, and audit qualifications —all of which cost more than setting up the three layers described here.
The register pattern takes one function call. The eval-as-span attachment takes zero additional latency. The tail-based sampling keeps your storage bill under control while retaining every trace that matters. Start with one model in one service. Instrument it. Watch the trace tree for a day. You’ll find something you didn’t know was happening —everyone does on their first day with real observability.