In June 2025, a compromised PyPI package in the LiteLLM supply chain exfiltrated API keys from 95 million monthly package installs. In March 2026, a company burned $500 million in a single month from an API key with no spending cap. In between, dozens of smaller incidents —keys committed to public repos, exposed in client-side code, shared in Slack messages —cost teams thousands of dollars and weeks of remediation. Five of the first six entries in our 23 LLM API mistakes guide are security failures —leaked keys, missing budget caps, flat key architectures.
LLM API security in 2026 isn’t theoretical. The attack surface is real. The financial blast radius of a single leaked key is measured in dollars per minute. The OWASP Top 10 for LLM Applications catalogs the most critical risks —this article maps the ten most actionable security baselines to those risks. This is the authority reference for API authentication and key management across the TokSpan blog —other articles link here for full implementation details.
Baseline 1: Centralized Secrets Vault
No API keys in .env files. No keys in source code. No keys in Slack, Notion, or chat logs. Every provider key lives in an encrypted secrets vault —AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, or Doppler.
Keys are injected at runtime —never baked into container images at build time. A container image with embedded credentials is a leaked credential the moment that image escapes your private registry. The Kubernetes Secrets Store CSI Driver syncs secrets from your vault into your pods without the keys ever touching a Kubernetes Secret object —plain K8s Secrets are trivially exposed by anyone with get secrets RBAC.
The minimum viable implementation: one vault —HashiCorp Vault for self-hosted, Doppler for managed. All provider keys stored encrypted. Applications fetch keys at startup via the vault’s SDK. Keys never appear in configuration files, environment variables on disk, or version control. The time investment is proportional to the cost of a leaked key —and a leaked GPT-5.5 key with no budget cap can cost $500 million in a month.
Baseline 2: Scoped Key Architecture
The “flat key” anti-pattern —one API key per provider shared across every environment, every application, every developer —is incompatible with audit readiness, cost control, and incident containment.
Implement a scoped hierarchy:
| Scope | Purpose | Budget Cap | Model Allowlist |
|---|---|---|---|
| Production (customer-facing) | Live user traffic | $5,000/month | Pinned, versioned model IDs only |
| Production (internal tools) | Internal dashboards, analytics | $1,000/month | Broader, but no experimental models |
| Staging | Pre-release testing | $200/month | Production models + evaluation candidates |
| Development | Experimentation | $50/month | Broadest, with per-developer caps |
Each scope gets its own API key, model allowlist, and budget. Benefits: attribution —every request is traceable to a specific application and environment. Blast-radius containment —a compromised development key can’t access production models or budgets. Safe rotation —keys rotate per scope on independent schedules without org-wide coordinated redeployments.
The simplest path to scoped keys is a platform that supports virtual API keys —create separate keys for each environment with per-key budgets and model restrictions. No changes to your provider accounts. Provider keys stay in the platform vault. Applications use short-lived virtual keys scoped to their specific needs. See the TokSpan authentication docs for how virtual API keys, scoped permissions, and per-key budgets work in practice.
Baseline 3: Proxy/Gateway Layer
Applications should never hold provider API keys directly. They authenticate to a gateway with short-lived, scoped virtual keys. The gateway holds the actual provider keys, enforces model allowlists, applies budget checks, logs every request, and forwards to providers.
Architecture: Application —Virtual Key —Gateway —Provider Key —LLM Provider.
This ensures provider keys never ship to browsers, mobile apps, or client-side code. They never appear in application logs. If an application’s virtual key is compromised, you revoke it at the gateway —provider keys were never exposed. Propagation is per-request with near-instant effect.
The gateway can be self-hosted (LiteLLM) or managed (aggregation platform). The security properties are similar. The operational overhead differs —self-hosting requires maintaining the gateway infrastructure; managed platforms handle it for you. TokSpan’s security architecture documents how the gateway layer is implemented in practice —key isolation, request-level audit logging, and per-key budget enforcement.
Baseline 4: Automated Key Rotation
Schedule: Every 90 days for all provider keys —more frequently for keys with broad scope. Emergency rotation: immediately when a developer leaves, a laptop is lost, or exposure is detected —even suspected.
Blue/green rotation procedure: Generate a new key at the provider. Add it to your vault alongside the old key. Deploy —applications pick up both keys. Monitor for 15–30 minutes —all requests succeeding with the new key. Revoke the old key at the provider. The transition is seamless because the gateway handles key selection. Applications never know the keys changed.
Without a gateway layer, rotation requires coordinated redeployment across every application that uses the key —a multi-hour operation with downtime risk. With a gateway, it’s a 30-minute, zero-downtime operation.
Baseline 5: Hard Budget Caps
Set spending limits at three levels. Provider dashboard: the backstop —caps at the source that even a platform compromise can’t exceed. Platform/gateway level: operational control —per-environment, per-application caps that prevent runaway spending before it reaches the provider. Per-key: attribution —per-developer, per-feature caps that contain the blast radius of a rogue loop or compromised key.
Alert at 80% of each cap. Hard-reject at 100%. The $500M monthly bill happened because one organization had zero caps at any level. One configuration change would have prevented it.
Baseline 6: Least-Privilege Model Allowlists
Each key should only access the models it needs. Production customer-facing keys: pinned, versioned model IDs —gpt-5.5-2025-06-15, never the alias gpt-5.5. Aliases silently upgrade to new snapshots that may change behavior. Development keys: broader access for experimentation, but never to models that aren’t approved for your use case.
Operation restrictions: inference-only keys should not be able to call fine-tuning, administrative, or billing endpoints. Default-deny posture: new keys start with zero access. Models and operations are explicitly granted.
Baseline 7: CI/CD Secret Detection
Deterministic scanning that catches LLM API keys before they merge. Detect sk-proj-* (OpenAI), sk-ant-* (Anthropic), and other provider key patterns across Python, JavaScript, Java, C#, Go, .env files, YAML configs, JSON configs, and CI pipeline logs. Block merges when critical or high-confidence secrets are detected. Treat LLM keys as tier-zero credentials —the same severity as cloud IAM credentials.
Baseline 8: Full Audit Trail
Every API call must be traceable across these dimensions: user ID, application ID, environment, model, provider, tokens consumed, cost, timestamp, request ID, and guardrail results. Log to a centralized system with minimum 90-day hot retention —1-year cold for regulated workloads. Redact sensitive patterns (API keys, PII) from logs before writing to persistent storage.
Here’s what audit logs catch in practice: In January 2026, a developer at a Series A SaaS company accidentally committed a development-scoped API key to a public GitHub Gist while debugging a CI failure. The key leaked at 2:14 AM UTC. By 6:30 AM, the SOC received an automated alert from the gateway —request volume on that key had spiked 40x. Because every request was logged with user ID, application ID, model, and token count, the team traced the full exposure in 11 minutes: 37 requests over 4 hours, all against a pinned GPT-4.0 model, none touching production data or fine-tuning endpoints. The scoped key and per-key budget cap held losses to $18. Without those logs, the team would have spent days reconstructing the blast radius —or assumed the worst and triggered an unnecessary breach disclosure to every customer. The audit trail turned a credential leak into a confirmed non-event.
Under HIPAA, the audit trail must answer: “Which systems accessed PHI on this date? Which model processed it? Under which BAA?” A flat-key architecture with no per-user attribution cannot answer these questions. A scoped-key architecture with gateway-level logging can.
Baseline 9: PII Redaction & Data Privacy
Redact personally identifiable information from prompts before they leave your infrastructure. Use gateway-level PII detection —Portkey’s guardrails, custom middleware, or dedicated tools. Understand each provider’s data-use policy: does your tier allow training on API data? Is there an opt-out? For sensitive workloads, use providers with contractual data-processing agreements —or route through a platform that provides them.
Baseline 10: Documented Incident Response
When a key leaks, the response is time-sensitive —a compromised key with no budget cap can generate thousands of dollars in unauthorized usage per hour. Procedure: Revoke the key at the gateway immediately —propagation is per-request with near-instant effect. Rotate the provider key that may have been exposed. Cap spend at the provider dashboard as emergency backstop. Audit the scope’s request logs for the exposure period: what data was in the prompts? Any anomalous activity? Patch the gap —usually a missing guardrail, an over-broad CORS origin, or a tool lacking an explicit consent gate. Disclose to affected parties if PII or PHI was involved.
LLM API Security Maturity Model
You don’t need all ten baselines on day one. The maturity model below maps baselines to your organization’s stage —prioritize based on what a breach would cost you right now.
Basic (Startup / Solo Developer): Baselines 1, 5, 7. A centralized secrets vault keeps keys out of .env files and version control. Hard budget caps at the provider dashboard prevent catastrophic spend from a single leak. CI/CD secret detection catches keys before they merge to public repos. These three baselines prevent the two most common failure modes —committed keys and uncapped spending. Implementation time: one afternoon with Doppler + provider dashboard caps + a pre-commit hook.
Standard (Growth Team / Multi-Environment): Basic + Baselines 2, 3, 4, 6. Scoped keys per environment contain blast radius. A gateway layer ensures applications never hold provider keys directly —a leaked virtual key exposes nothing. Automated rotation reduces key-exposure windows from months to 90 days. Model allowlists enforce least-privilege access, blocking staging keys from calling production-only models. These baselines become necessary the moment you have more than one environment —a single flat key per provider is the “root user with no MFA” of API security. Implementation time: one week with a managed gateway.
Enterprise (Regulated / Compliance-Required): Standard + Baselines 8, 9, 10. Full audit trails with per-user attribution satisfy HIPAA, SOC 2, and ISO 27001 requirements —you can answer “who accessed what data, when, and through which model” for any audit window. Gateway-level PII redaction scrubs sensitive data before prompts leave your infrastructure. A documented, tested incident response playbook means the SOC executes the revoke-rotate-audit workflow in under 5 minutes, not under 5 hours. At this tier, security is no longer a feature —it’s the control plane for your entire LLM infrastructure.
FAQ
What’s the #1 security mistake with LLM APIs?
Hardcoded keys in source code or .env files that get committed to version control. The LiteLLM supply-chain attack exfiltrated keys from 95 million monthly installs —but the more common vector is a simple git push to a public repo. Use a secrets vault. Never hardcode.
Do I really need per-environment API keys?
Yes. A compromised development key must not grant access to production models or budgets. Scoped keys contain the blast radius. A single flat key per provider is the “root user with no MFA” of LLM API security.
Are aggregation platforms more or less secure than direct API?
Well-implemented platforms are more secure: virtual keys never expose provider credentials, per-key budgets and allowlists are built in, audit trails are unified, and key rotation is centralized. Poorly implemented platforms are less secure. Evaluate a platform’s security documentation before committing. For self-hosted requirements, LiteLLM provides the same security properties with full control over infrastructure.
How do I comply with HIPAA when using LLM APIs?
Use a gateway with BAA support, audit logging that attributes every request to a user and purpose, data-residency options that keep PHI within compliant infrastructure, and contractual guarantees that providers won’t train on your data. Direct API requires per-provider BAAs —each negotiated separately. Aggregation platforms can consolidate this into one agreement.
What should I do if my API key leaks?
Immediate: revoke at the gateway. Rotate the provider key. Cap spend. Then: audit logs to determine what was accessed. Patch the gap. Disclose if data was exposed. The first three steps should take under 5 minutes. The last three may take days. Having the procedure documented before you need it is the difference between “incident” and “catastrophe.”
The ten baselines covered here form a specific checklist for LLM API security. Zoom out, and a broader pattern emerges across the industry: security is ceasing to be a feature you bolt on and becoming the substrate everything else runs on. The same shift happened with cloud IAM a decade ago —what started as a compliance checkbox became the control plane for the entire infrastructure. LLM API security is on the same trajectory. The teams that treat these baselines as foundational architecture, not a post-launch audit item, are the ones that will move fast without breaking things —or budgets.
Secure your API keys —Virtual keys, scoped permissions, per-key budgets, and unified audit logging. Seven of ten baselines enforced by default.