LLM API Key Management for Teams: Rotation, Scoping, and Gateway-Level Secrets Governance

A 30-second answer: the biggest API key risk for teams using multiple LLM providers is not a single leaked key — it is key sprawl. Five providers × three environments × four teams = 60 keys scattered across env files, CI secrets, and Slack DMs. A routing gateway collapses that to one upstream key per provider (managed centrally) and one gateway credential per team (scoped and rotatable), while a secrets manager handles lifecycle. This guide walks through the practical steps.

Why LLM API Keys Are Harder Than Regular API Keys

Traditional SaaS API keys are annoying to manage but manageable: one provider, one key, one billing account. LLM deployments break this pattern in three ways:

1. Multiple providers are the norm. Most production setups route to at least two providers — a primary and a fallback. OpenAI for GPT-5, Anthropic for Claude, maybe DashScope for Qwen models in China. Each provider issues its own key format, its own dashboard, its own rotation mechanism.

2. Keys carry spending authority. Unlike a read-only analytics API key, an LLM API key authorizes token consumption that can run to thousands of dollars per day. A leaked key is not just an access problem — it is a billing problem.

3. AI coding agents multiply the blast radius. Tools like Cursor, Claude Code, and Codex each need credentials. When developers paste the same production key into their IDE config, every laptop becomes an unaudited entry point.

Step 1 — Audit Your Key Inventory

Before fixing anything, find out what you have. For each provider:

Most teams discover that the answer to "who has access" is "everyone who was on the project when the key was created." That is your starting point.

Step 2 — Scope Keys: One Per Provider Per Environment

The flat-key anti-pattern — one key shared across dev, staging, and production — makes rotation terrifying because you cannot test the new key without risking production traffic.

The fix: create separate keys for each environment at every provider:

# Instead of this:
OPENAI_API_KEY=sk-prod-shared-across-everything

# Do this:
OPENAI_API_KEY_PROD=sk-prod-xxxxxxxx
OPENAI_API_KEY_STAGING=sk-staging-yyyyyyyy
OPENAI_API_KEY_DEV=sk-dev-zzzzzzzz

This gives you three things for free:

• Safe rotation: rotate the staging key first, verify, then rotate production.
• Blast radius control: a leaked dev key cannot run up production bills.
• Attribution: provider usage dashboards show which environment generated which traffic.

OpenAI supports project-scoped API keys that restrict a key to specific models and rate limits within a project. Anthropic offers workspace-level key isolation. DashScope uses RAM-based access control per key. Use the native scoping controls each provider offers.

Step 3 — Move Keys Into a Secrets Manager

Hardcoded keys in .env files are a liability. A secrets manager adds rotation automation, access control, and audit trails:

The integration pattern is straightforward: your application reads the key from the vault at startup (or on each request if you can tolerate the latency), and the vault handles rotation on a schedule you define.

# Example: reading an OpenAI key from AWS Secrets Manager at runtime
import boto3, json

def get_openai_key():
    client = boto3.client("secretsmanager", region_name="us-east-1")
    resp = client.get_secret_value(SecretId="prod/openai/api-key")
    return json.loads(resp["SecretString"])["OPENAI_API_KEY"]

Step 4 — Put a Gateway in Front of Providers

Even with per-environment keys and a vault, you still have a coordination problem: every application that calls an LLM provider needs to know the key, the endpoint, and the rotation schedule for that provider. When you use five providers, every service needs five sets of credentials.

A routing gateway eliminates this:

Before (N apps × M providers = N×M key pairs):
  App A → OpenAI key, Anthropic key, DashScope key
  App B → OpenAI key, Anthropic key, DashScope key
  App C → OpenAI key, Anthropic key, DashScope key

After (N apps × 1 gateway credential):
  App A → Gateway key (team-a-prod)
  App B → Gateway key (team-b-prod)
  App C → Gateway key (team-c-prod)
  Gateway → OpenAI key, Anthropic key, DashScope key (centrally managed)

The gateway holds all upstream provider keys in one place. Applications authenticate with a single gateway credential scoped to their team and environment. When a provider key rotates, you update it once in the gateway — no application redeployments.

OpenAI-compatible means a provider exposes a chat-completions endpoint whose request and response shape matches the OpenAI API contract closely enough that an unmodified OpenAI SDK call works against it after swapping three values: API key, base URL, and model name. The minimum surface in practice is POST /v1/chat/completions with messages, model, and an OpenAI-shaped streaming response.

Because the gateway exposes an OpenAI-compatible endpoint, applications do not even need to know which upstream provider is handling their request. The same base_url + gateway key works regardless of whether the request routes to OpenAI, Anthropic, or DeepSeek.

Step 5 — Implement Rotation With Dual-Key Overlap

The safest rotation pattern keeps two keys active simultaneously during the transition window:

Timeline:
  T+0    Generate new key (KEY_B) in provider dashboard
  T+0    Add KEY_B to vault/gateway alongside KEY_A
  T+1h   Verify KEY_B works (send test requests)
  T+24h  Update all consumers to prefer KEY_B
  T+48h  Revoke KEY_A in provider dashboard

The 48-hour grace period matters because:

• Cached credentials in long-running processes expire naturally.
• Any CI/CD pipelines that cached the old key will pick up the new one on next run.
• If KEY_B has issues, you can roll back to KEY_A without downtime.

When to rotate immediately (no grace period):

• Key confirmed leaked in a public repository.
• Anomalous spending spike detected.
• Team member with key access leaves the organization.

SOC 2 and ISO 27001 typically require rotation every 90 days. For LLM keys with high spending authority, we recommend 30–60 days.

Step 6 — Set Up Audit Logging and Alerts

The final piece is visibility. You need to answer three questions at any time:

1. Who is making requests? (Which team, which application, which environment?)
2. How much are they spending? (Per key, per day, per model?)
3. Is anything abnormal? (Sudden spike in a dev key? Requests from an unexpected IP?)

A gateway with observability features gives you this without instrumenting every application. Each gateway credential maps to a team and environment, so every request is automatically tagged.

Set alerts for:

• Spending threshold breached — per-key daily/weekly cap.
• Unusual request patterns — sudden volume spike, new model accessed, requests outside business hours.
• Failed authentication — repeated 401s may indicate a revoked key still in use somewhere.
• Key age exceeded — automated reminder when a key passes its rotation deadline.

Common Mistakes (and How to Avoid Them)

Production Checklist

Use this as a go/no-go gate before shipping an LLM-powered feature:

• Every provider key is stored in a secrets manager (not in .env, not in CI env vars)
• Keys are scoped per environment (dev / staging / prod)
• Keys are scoped per team or application where the provider supports it
• A rotation schedule exists and is enforced (≤ 90 days, ideally 30–60)
• Rotation uses dual-key overlap — no big-bang cutover
• A revocation runbook exists and has been tested
• Spending limits are set per key at the provider level
• A gateway consolidates upstream keys — applications hold only gateway credentials
• Audit logs capture team, app, environment, and model per request
• Alerts fire on spending spikes, auth failures, and key age expiry

TheRouter Integration

TheRouter routes OpenAI-compatible requests through configured providers, which means it naturally serves as the gateway layer described in this guide. Your applications authenticate with TheRouter using a single credential, and TheRouter holds the upstream provider keys for OpenAI, Anthropic, DashScope, SiliconFlow, and DeepSeek.

When a provider key rotates, you update it in TheRouter's configuration — no application changes needed. Combined with fallback routing, this means a revoked key on one provider does not take down your application: traffic falls back to the next configured provider while you provision a new key.

For teams already managing keys in a vault, the pattern is: vault stores provider keys → TheRouter reads them at configuration time → applications call TheRouter with a team-scoped gateway key. One vault, one gateway, one credential per team.

Further Reading

• Claude API Key Governance for Enterprise Security — Anthropic-specific key governance patterns
• LLM API Cost Optimization Through Smart Routing — cost controls that pair with key governance
• LLM API Observability and Monitoring Tools — the logging layer that makes audit work
• Unified LLM API Providers and Gateway Comparison — gateway options beyond TheRouter
• Agentic Coding Model Governance for Operators — key management for AI coding agents