An AI API gateway gives an application one stable endpoint while the infrastructure behind that endpoint can use multiple models, providers, accounts, or regions. The useful part is not merely hiding several API keys behind one key. The useful part is creating a controlled decision point for every request.
That decision point can answer operational questions before traffic reaches a model provider:
- Is this client allowed to call the requested model?
- Which upstream currently satisfies the request's capability, latency, and cost requirements?
- Is that upstream healthy enough to receive more traffic?
- Can the request be retried safely?
- Which fallback preserves the response contract?
- How will the team explain the route, cost, and failure afterward?
This guide maps those responsibilities into a production architecture. It also shows where a single API key helps, where it does not, and how to migrate an OpenAI-compatible client without turning the gateway into an invisible source of routing surprises.
The reference architecture in one request path
A practical AI gateway request moves through five layers:
- Client contract: the application sends an authenticated request to one stable base URL.
- Admission controls: the gateway validates identity, quota, model permissions, payload limits, and request metadata.
- Routing policy: a policy engine converts the requested model or capability into eligible upstream targets.
- Execution controls: health, concurrency, timeout, retry, fallback, and streaming rules determine how the selected target is called.
- Telemetry and accounting: the gateway records the selected route, response status, latency, token or media usage, and cost attribution.
Application / agent
|
| one API key + stable request schema
v
AI API gateway
├─ authentication and tenant policy
├─ model alias and capability registry
├─ routing policy and budget rules
├─ health, timeout, retry, and fallback controls
└─ logs, traces, usage, and cost attribution
|
├────────> Provider or deployment A
├────────> Provider or deployment B
└────────> Provider or deployment C
The gateway is therefore both a control plane and a data plane. The control plane stores policies, credentials, aliases, quotas, and routing configuration. The data plane handles live requests, streaming responses, retries, and telemetry. Keeping those responsibilities conceptually separate makes changes safer: operators can update routing policy without asking every application team to ship new client code.
What “one key” should mean
“One key” should mean one application-facing credential contract, not one credential shared by every person, service, and environment.
A sound design issues separate gateway credentials for production, staging, local development, CI, and independent workloads. Each key should have a narrow scope, an owner, a quota, and a revocation path. The gateway then keeps provider credentials server-side and maps an incoming identity to the upstream credentials it is allowed to use.
This creates a useful security boundary:
| Boundary | Client can see | Gateway can see | Provider can see |
|---|---|---|---|
| Application credential | Its own gateway key | Client identity and policy | Not required |
| Provider credential | Nothing | Encrypted upstream secret or managed identity | Provider account identity |
| Routing policy | Requested public model or alias | Eligible targets and selection reason | Only the chosen request |
| Billing context | App-level usage if exposed | Tenant, project, route, usage, and price mapping | Provider-side usage |
The gateway key should never be treated as a reason to weaken key hygiene. Put it in a secret manager, never in browser code or a public repository, rotate it, and separate it by environment. For a deeper operational checklist, see secure API key management for AI products.
Model aliases separate the client contract from providers
The first routing abstraction is a model alias. Instead of hard-coding a provider-specific model identifier throughout an application, the client requests a stable name such as:
support-fast
reasoning-high
code-review-default
image-generation-standard
The registry behind each alias defines a capability contract. A text alias might specify tool calling, structured output, minimum context size, streaming support, and an approved fallback family. An image or video alias needs different fields, such as accepted input types, output dimensions, asynchronous job behavior, and safety constraints.
An alias should not promise that every candidate model behaves identically. It should define the minimum behavior the application can depend on.
alias: support-fast
contract:
modality: text
streaming: true
tools: optional
structured_output: required
maximum_latency_ms: 3500
routes:
- target: provider-a/model-fast
priority: 1
- target: provider-b/model-balanced
priority: 2
This indirection is what makes a stable base URL valuable. Applications integrate with the alias contract; platform owners can change the target set after evaluation, a provider incident, a pricing change, or a regional requirement.
The routing decision should be explicit
Production routing usually combines hard filters and soft ranking.
1. Apply hard eligibility filters
Remove any target that cannot satisfy the request. Common filters include:
- Required modality and input type
- Context-window or output-size requirement
- Tool calling or structured-output support
- Data residency or regional availability
- Tenant or project allowlist
- Safety or compliance policy
- Current quota, rate-limit, or concurrency state
- Streaming compatibility
A target that fails a hard requirement should never win because it is cheaper.
2. Rank the eligible targets
After filtering, score the remaining routes. A simple policy can be easier to operate than an opaque optimizer:
route score =
quality_weight × evaluation_score
- latency_weight × predicted_latency
- cost_weight × estimated_cost
- risk_weight × recent_error_rate
The weights should differ by workload. Interactive chat may favor time to first token. A nightly extraction job may favor cost per successful structured record. A coding agent may value tool reliability and long-context behavior more than a small price difference.
3. Record the reason
Every routing decision should produce machine-readable metadata such as:
{
"requested_alias": "support-fast",
"selected_target": "provider-a/model-fast",
"policy_version": "support-fast-2026-07-29.3",
"selection_reason": "healthy_primary_within_latency_budget",
"fallback_count": 0
}
If a team cannot reconstruct why a route was selected, it cannot debug cost drift, quality regressions, or provider incidents.
Health checks need more than an HTTP 200
An upstream can return successful health probes while failing real model traffic. AI gateway health therefore needs several signals:
- Transport health: connection failures, TLS errors, DNS errors, and upstream timeouts
- API health: rate-limit responses, authentication failures, provider errors, and malformed responses
- Model health: empty output, invalid structured output, broken tool calls, or incompatible streaming chunks
- Performance health: time to first token, total latency, queue time, and throughput
- Capacity health: concurrent requests, token-per-minute pressure, account balance, or deployment quota
Use a rolling window rather than one failure. A circuit breaker can temporarily remove a target after its failure or latency threshold is crossed, then allow limited probes before restoring full traffic. Outlier detection can also eject one unhealthy deployment while leaving healthy deployments from the same provider available.
The principle is well established in gateway and service-mesh infrastructure: retries, circuit breaking, and outlier detection are separate controls, and each needs a bounded policy. Envoy documents these mechanisms independently in its HTTP retry, circuit breaking, and outlier detection guidance.
Retry only when the request is safe
Retries improve reliability only when they do not multiply work or create duplicate side effects.
For a non-streaming text completion that failed before any response bytes arrived, one retry against the same target may be reasonable. For a request that triggers a tool, starts an image or video job, charges an external account, or has already streamed partial output, a blind retry may create duplicates or corrupt the user experience.
Define retry eligibility using three questions:
- Was the request accepted upstream? A connection failure before acceptance is different from a timeout after the provider started work.
- Has any output reached the client? Once streaming begins, switching providers can produce a discontinuous answer.
- Is there an idempotency key or deduplication record? Long-running media and agent workflows need a stable operation identity.
A conservative retry matrix looks like this:
| Failure | Same-target retry | Cross-target fallback | Notes |
|---|---|---|---|
| Connection failure before response | Usually safe, bounded | Usually safe | Apply jitter and deadline budget |
| Provider rate limit | Sometimes | Often | Respect retry hints and capacity state |
| Provider 5xx before output | Bounded | Often | Exclude unhealthy target temporarily |
| Invalid structured output | Only with a repair policy | Only to contract-compatible target | Count against quality SLO |
| Partial streaming response | Usually no | Usually no | Return a clear stream error or resume only with an explicit protocol |
| Async media job accepted | No blind retry | No blind fallback | Poll by operation ID; deduplicate submissions |
Keep one end-to-end deadline. If the client allows eight seconds, the gateway cannot spend seven seconds on the primary and then give the fallback another eight. Each attempt consumes the same request budget.
Fallbacks must preserve the contract
A fallback is not simply “try another model.” It is an agreement about what may change when the primary route fails.
Define fallbacks at three levels:
- Same model, different deployment or account: lowest behavioral risk; useful for quota or regional failures.
- Equivalent model family: moderate risk; requires regression tests for schema, tools, safety, and output style.
- Degraded capability: highest risk; may disable tools, reduce context, or return a queued response instead of a live one.
For each alias, document:
- Which failure classes trigger fallback
- Which targets are contract-compatible
- Whether the client is told that fallback occurred
- Maximum attempts and total deadline
- How quality and cost changes are measured
- Whether the response may be cached or replayed
Regional provider access adds another dimension. A provider or model may be available in one geography, account type, or commercial arrangement and unavailable in another. Regional LLM provider routing explains the separate access, policy, and failover checks needed for those routes.
Streaming is part of the gateway contract
OpenAI-compatible request shapes can simplify client migration, but streaming compatibility requires deliberate translation. The gateway must preserve event order, finish reasons, usage metadata, tool-call fragments, error signaling, and connection cancellation.
Before routing two models behind one streaming alias, test:
- Time to first event and heartbeat behavior
- Incremental text delta format
- Tool-call argument assembly
- Usage reporting in the final event
- Client cancellation propagation
- Timeout behavior before and after the first event
- Error format after headers have already been sent
Do not hide a stream restart inside one response unless the protocol explicitly supports resumption. In most clients, mixing a partial answer from one model with a second answer from another is worse than returning a clear error.
Observability connects routing to outcomes
Gateway dashboards are useful, but production diagnosis requires structured telemetry that can join a model request to the surrounding application trace.
At minimum, capture:
| Dimension | Example fields |
|---|---|
| Identity | tenant, project, environment, key ID, workload |
| Request | request ID, operation ID, alias, modality, input size |
| Routing | policy version, eligible targets, selected target, fallback count |
| Reliability | status class, provider error code, retries, timeout stage |
| Performance | queue time, time to first token, total latency, output throughput |
| Usage | input, output, cache, image, audio, or video units |
| Economics | estimated cost, billed cost, budget rule, price version |
| Quality | evaluation label, schema validity, tool success, user outcome |
Avoid logging raw prompts and outputs by default. Record content only when the use case, retention policy, and user expectations permit it. The OpenTelemetry project maintains evolving semantic conventions for generative AI systems that can help teams use consistent span and metric names rather than inventing a separate schema for every provider.
Cost controls belong before the upstream call
Post-hoc spend reports cannot prevent an incident. Admission and routing policy should evaluate cost before sending traffic.
Useful controls include:
- Per-key and per-project hard quotas
- Soft budget alerts
- Maximum input or output units
- Model allowlists by environment
- Cost-aware routing for flexible workloads
- Cache policy for repeatable requests
- Concurrency limits for expensive media jobs
- Kill switches for a model, provider, tenant, or route
The routing engine needs a versioned price table and a consistent usage normalization layer. Otherwise, a “cheapest model” policy may compare incompatible units or stale prices. For a framework that separates provider rates, platform fees, and operational controls, see AI gateway pricing.
A minimal OpenAI-compatible migration
The smallest client change is usually a new API key, base URL, and model name. With an OpenAI-compatible gateway, the application code can keep the same client library:
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["FLATKEY_API_KEY"],
base_url="https://router.flatkey.ai/v1",
)
response = client.chat.completions.create(
model="your-model-or-alias",
messages=[
{"role": "user", "content": "Summarize this incident report."}
],
)
That code change is the easy part. A safe migration has four stages:
- Inventory the current contract. Record models, parameters, streaming behavior, tools, schemas, timeouts, and error handling.
- Run shadow or offline evaluations. Compare output quality, schema validity, latency, and cost on representative requests.
- Canary one workload. Start with a bounded traffic percentage and an immediate rollback path.
- Enable routing features separately. First change the endpoint, then add aliases, then health-based failover, then cost or quality optimization.
Separating those changes makes incidents diagnosable. If the endpoint migration, model replacement, retry policy, and cost optimizer all launch at once, the team will not know which variable caused a regression. The Flatkey integration starter covers the base-URL migration pattern in more detail.
Production readiness checklist
Use this checklist before treating the gateway as shared infrastructure.
Client contract
- Stable base URL and versioned request schema
- Named aliases with documented minimum capabilities
- Consistent error envelope and request IDs
- Tested streaming, tool calls, and structured output
Identity and security
- Separate keys by service and environment
- Server-side provider credentials
- Key scopes, quotas, rotation, and revocation
- Prompt and response logging disabled or governed explicitly
Routing and reliability
- Hard eligibility filters before cost ranking
- Versioned routing policies and price data
- Health based on real request behavior
- Bounded retries with one end-to-end deadline
- Contract-compatible fallback targets
- Circuit breaker and recovery probes
Operations
- Route reason, provider error, latency, and usage telemetry
- Alerts for fallback rate, error rate, cost drift, and quota pressure
- Per-model and per-route kill switches
- Runbook for provider outage and gateway outage
- Direct or alternate emergency path for critical workloads
How Flatkey fits this architecture
Flatkey provides one API key, one OpenAI-compatible base URL, and one dashboard for supported model access, usage, and billing. Its router is designed to reduce separate provider accounts and fragmented integration paths while supporting upstream switching and load balancing.
For an application team, the architectural benefit is a stable client boundary: point an OpenAI-compatible client at https://router.flatkey.ai/v1, select a supported model, and keep model access behind the same gateway endpoint. Teams should still define their own application-level contracts, evaluation thresholds, key scopes, failure budgets, and fallback expectations.
The best gateway architecture does not make routing invisible. It makes routing changeable, bounded, and explainable.
FAQ
What is an AI API gateway?
An AI API gateway is an intermediary between applications and model providers. It centralizes authentication, model access, routing, reliability controls, usage tracking, and policy while exposing a stable client-facing API.
Does one API key mean every service shares the same key?
No. It means applications use gateway-issued credentials instead of handling each provider credential directly. Production services, environments, and teams should still receive separate scoped keys.
What is model routing?
Model routing is the process of filtering eligible models or deployments and selecting a target according to capability, policy, health, latency, quality, cost, region, or capacity.
What is the safest fallback strategy?
Start with the same model on another healthy deployment or account. Cross-model fallback should happen only after tests show that the alternate target preserves the application's schema, tool, streaming, safety, and quality contract.
Can a gateway retry a streaming response on another model?
Usually not after output has reached the client. Switching midstream can combine incompatible partial responses. Use a clear stream error unless the client and gateway implement an explicit resume protocol.
Is an OpenAI-compatible API enough for zero-change migration?
It reduces SDK and request-shape changes, but teams still need to verify supported parameters, errors, streaming events, tool calls, structured output, token accounting, and model behavior.
