3:14 AM. PagerDuty fires. “The model returned an error.” Which model? Which user? A 429, a 503, or a malformed 400? Your dashboard shows a 4xx spike. That’s all you have.
Without structured LLM-aware logging, you can’t answer a single one of those questions. And auditors for SOC 2, GDPR, or HIPAA demand trails you don’t have.
This guide gives you the fix: a JSON log schema capturing every dimension of an LLM call, a troubleshooting tree for the six codes behind 90% of production incidents, and audit trail patterns mapped to SOC 2 and GDPR Article 30.
The LLM-Aware Log Schema
Standard HTTP access logs capture status codes and latency. LLM calls have another dozen dimensions that matter: which model version, which prompt template, how many input tokens, how many output tokens, how many were cached, what stop reason, what tool calls were made, what was the generated content. Capture these or debug blind.
{
"timestamp": "2026-07-27T03:14:22.451Z",
"trace_id": "a1b2c3d4e5f6",
"span_id": "7890abcd",
"user_id": "user_42",
"session_id": "sess_8f3a",
"request": {
"provider": "openai",
"model": "gpt-5.5",
"prompt_version": "checkout_v3.2",
"input_tokens": 1240,
"cached_tokens": 980
},
"response": {
"status_code": 200,
"output_tokens": 380,
"stop_reason": "stop",
"latency_ms": 1240,
"cost": {
"input": 0.0025,
"output": 0.0057,
"cached": 0.0001,
"total": 0.0083
}
},
"tools": [
{"name": "lookup_order", "status": "success", "latency_ms": 180}
],
"error": null
}Every LLM call in your system should produce exactly this schema. Not a subset. Not “we’ll add fields when we need them.” Standardize the schema before you have 15 services logging in 15 different formats. The prompt_version field alone — knowing which prompt template generated which response — is worth the logging investment within the first production incident.
For correlating these structured logs with full LLM call traces and spans, our OpenTelemetry observability guide covers end-to-end instrumentation. The OpenTelemetry GenAI semantic conventions define the standard span attributes — gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — that map directly to the fields in the log schema above, ensuring your logs and traces speak the same language.
Error Code Troubleshooting Tree
401 Unauthorized
What it means: Invalid or expired API key. First check: did someone rotate the key and not update this service? Second check: is the key scoped to the correct project/organization? Fix: validate the key with a minimal API call. Rotate keys on a schedule — not reactively after an incident. Our 10-point API security guide covers key rotation architecture, access controls, and the full security baseline.
403 Forbidden
What it means: Valid key, insufficient permissions. Common cause: trying to access a model not enabled for your account. Fix: check model availability in your dashboard. If a model isn’t available in your region, contact our support team about regional availability options.
429 Rate Limit / Too Many Requests
What it means: You’ve exceeded RPM (requests per minute) or TPM (tokens per minute) limits. First check: is one service or user consuming disproportionate quota? Second check: are retries compounding the problem — failed requests triggering more requests? Fix: implement exponential backoff with jitter. Add client-side token-bucket throttling. Route overflow to alternative models. Full rate limit architecture in our guide to handling rate limits and 429 errors.
500 / 502 / 503 Server Error
What it means: Provider-side issue. Not your fault. Not your fix. What you can do: implement fallback chains — if provider A returns 5xx, automatically retry on provider B. A unified API endpoint makes this transparent to application code. Log every 5xx with provider, model, and region for SLA tracking.
400 Bad Request
What it means: Malformed request. Common causes: context window exceeded, invalid model name, unsupported parameter combination. Fix: validate context length before sending. Maintain a model registry with context window limits. Log the full request body (sanitized of PII) for debugging.
404 Not Found
What it means: Model ID doesn’t exist or has been deprecated. Common cause: hardcoded model string that was retired. Fix: use a model registry with aliases (chat_default → specific model ID). When a model is deprecated, change one alias definition — not 15 hardcoded strings. For current model availability and lifecycle status across all providers, check the supported models page.
Building Audit-Ready Logging
SOC 2 and GDPR auditors don’t care about your error troubleshooting. They care about: who accessed what data, when, using which model, and what was the result. Your logging needs to answer these questions with immutable, queryable records.
Required audit fields: user ID (who), timestamp (when), model and provider (which AI system), input token count and type (what data was sent), output content or summary (what was returned), stop reason (was it a normal completion, a refusal, a content filter?).
Retention by framework: SOC 2 typically requires 90 days hot, 1 year warm. GDPR requires “no longer than necessary” — define a specific retention period in your data protection policy and enforce it. HIPAA requires 6 years. Implement retention at the log infrastructure level, not in application code.
Immutability: audit logs must be append-only and tamper-resistant. Use S3 Object Lock, CloudWatch Logs with integrity validation, or a dedicated audit logging service. “We store logs in a database that engineers can write to” will not pass an audit. For the full compliance framework covering data residency, consent management, and breach notification, see our GDPR compliance checklist.
Debugging with Structured Logs
A user reports: “The AI gave me a wrong answer yesterday around 4 PM.” Your debugging workflow:
- Query logs by
user_idand timestamp range → find the specific request. 2. Extracttrace_id→ follow the full request chain through your observability platform. 3. Checkprompt_version→ was this the old prompt or the new one deployed that morning? 4. Checkstop_reason→ did the model complete normally, hit a content filter, or exceed token limits? 5. Checktoolresults → did thelookup_ordercall return correct data? 6. Checkcost→ was this request abnormally expensive (suggesting a thinking loop or tool-call spiral)?
Without structured logs, this workflow is: “search Slack for the user’s complaint → ask the engineer who handled it → they don’t remember → guess.”
Debugging War Stories: 3 AM Scenarios with Real Logs
Theory is clean. Production is messy. Here are four production debugging sessions, reconstructed from real incident timelines, with the log snippets that solved them.
2:47 AM — The cost anomaly that was a data contract change. Your cost monitoring fires: the checkout_summary endpoint’s per-request cost jumped 14x in the last hour. You pull the structured logs. Normal requests show input_tokens: 1240, cached_tokens: 980, output_tokens: 380, cost $0.008. Then you hit the spike: input_tokens: 18920, cached_tokens: 0, output_tokens: 11240, cost $0.62. Same endpoint. Same prompt_version. The stop_reason: "stop" tells you the model completed normally. The prompt version is unchanged, so the template isn’t the issue. You query the sanitized request content: the frontend team deployed a “small improvement” at 2:30 AM — enriching the cart endpoint with full product detail objects. The cart data that used to be {"sku": "NK-4721", "name": "Pegasus 41", "price": 129.99} became a 2KB product document. Multiplied by 8 items per cart. The LLM faithfully summarized all of it. The cost multiplied by 14. The fix wasn’t a code change — it was a data contract: the cart service now sends {sku, name, price, quantity} to the LLM, not the full product catalog response. Debugging took 8 minutes because the structured logs had input_tokens, cached_tokens, and prompt_version. Without those fields, you’d still be grepping access logs wondering why your bill spiked.
5:12 AM — The tool call that silently returned stale data. Your RAG pipeline’s lookup_inventory tool shows status: "success" for every call. But customers report the AI says items are in stock that are actually sold out. You pull the tool logs: the response shows {"in_stock": true, "quantity": 142, "last_updated": "2026-07-26T22:15:00Z"}. The response is valid JSON. The status is success. But last_updated is seven hours ago. The inventory sync job that runs hourly from the warehouse management system had silently failed at 11 PM. The tool’s cache was serving seven-hour-old data. The tool was “working.” The data was wrong. The fix: add a data_freshness check to your tool response schema. If last_updated is more than 2 hours old, the tool returns status: "stale_data" instead of status: "success". The LLM can then tell the user “our inventory data may not be current — let me check with a representative.” The logging caught the stale timestamp. The schema change prevented the next occurrence.
3:33 AM — The 429 spiral that broke exponential backoff. Your error dashboard shows a 429 rate limit cluster. The logs show exponential backoff working perfectly — retry delays doubling from 1s to 2s to 4s to 8s, all with incrementing retry_count values. But then you see it: twenty-seven requests fire simultaneously, all with retry_count: 0. The exponential backoff works per client instance. But twelve Kubernetes pods each independently hit the rate limit, backed off, and retried. Their retry windows overlapped. Plus, a new batch of user requests arrived and added fresh retry_count: 0 calls to the pile. The result: a thundering herd that kept the rate limit saturated for 19 minutes. The fix: add a token-bucket rate limiter at the API gateway level, before requests reach individual pods. Exponential backoff protects against transient failures within a single client. It does not protect against coordinated retry storms across a fleet. One engineering team discovered this at 3:33 AM. You can discover it by reading their logs instead.
6:47 AM — The stop_reason that wasn’t “stop.” Your content moderation pipeline has been running silently for eight months. You happen to check the stop_reason distribution and notice: "stop": 9842, "content_filter": 3, "length": 127. Wait — 127 length stop reasons? That means 127 requests hit the max_tokens limit and were truncated mid-response. Your pipeline ingests truncated JSON, fails to parse it, and writes the error to a dead-letter queue. But nobody monitors the dead-letter queue size because “structured output always works.” For eight months, 127 content items per batch have been silently dropped. That’s roughly 30,000 unreviewed items. The fix: monitor stop_reason distribution as a first-class metric. Any non-”stop” reason that exceeds 0.1% of traffic should trigger an alert. And always check your dead-letter queues — the errors you’re not monitoring are the ones accumulating quietly.
FAQ
How long should LLM API logs be retained?
SOC 2: 90 days hot, 1 year minimum. GDPR: define a specific period in your policy — 6-12 months is common, but the standard is “no longer than necessary for the purpose.” HIPAA: 6 years. Implement tiered retention: hot storage for recent logs (30 days), warm for compliance window (90 days-1 year), cold archive for long-term regulatory requirements.
What PII fields need to be redacted from logs?
Names, email addresses, physical addresses, phone numbers, credit card numbers, SSNs or national ID numbers, API keys and access tokens. Redact at ingress — before the data touches any LLM, log, or trace. Regex patterns for structured PII. NER classifiers for unstructured PII in free-text prompts. Raw sensitive values should never exist in your logging pipeline. For the authentication patterns that keep API keys out of logs and code, see the security documentation.
How do I correlate LLM API logs with application logs?
Use a trace_id that propagates through every service in the request chain. Every log entry — application, LLM API call, database query — carries the same trace ID. OpenTelemetry’s W3C trace context does this automatically when your services are instrumented. Without trace IDs, correlating “user clicked checkout” with “LLM generated cart summary” requires timestamp guessing. With trace IDs, it’s a single query.
What’s the difference between observability spans and audit log records?
Observability spans are for engineers debugging at 3 AM — they need trace IDs, latency breakdowns, and error messages. Audit log records are for compliance officers and auditors — they need immutable who-did-what-when records with tamper resistance. They have different schemas, different retention requirements, and different access controls. Don’t try to make one system serve both purposes — you’ll end up with a system that’s mediocre at both. The observability guide linked above covers spans. This guide covers audit logs. Both should be structured. Neither should be ad-hoc.
What’s the one log field most teams forget that causes the most pain during incidents?
prompt_version. Without it, you cannot answer “was the bad output caused by a prompt change, a model change, or a data change?” A deployment at 2 PM, a model config update at 4 PM, a prompt template change at 6 PM — and a user complaint at 8 PM. Which change caused it? With prompt_version, you query one field and know. Without it, you bisect deployments and guess. Two hours of debugging versus thirty seconds. Add prompt_version to your log schema today.
For automating log validation as part of your deployment pipeline, see our CI/CD testing and evaluation guide.
LLM logging isn’t exciting infrastructure — until it saves you at 3 AM. Implement the schema above. Test your error code troubleshooting tree against a simulated incident. The hour you spend standardizing log fields now will repay itself tenfold the first time you debug a production issue with structured data instead of grep and guesswork. For more production engineering patterns, our blog covers observability, security, and reliability — linked throughout this guide.