API Gateway Architecture and Key Features (2026 Guide)
An API gateway is a single entry point that sits between clients and your backend services, receiving every request, applying security and traffic policies, and routing it to the right service. Its architecture has two parts: a data plane that handles live traffic (routing, authentication, rate limiting, transformation, caching, logging) and a control plane where you configure those policies. The key features to look for are routing, authentication and authorization, rate limiting, request and response transformation, caching, observability, and resilience controls such as timeouts, retries, and circuit breakers.
This guide walks through API gateway architecture with a diagram, explains each component and feature, compares common deployment patterns and products, and shows working configuration and code. The last section covers how the same architecture applies to gateways for large language model (LLM) APIs. Product facts are from official documentation checked October 24, 2026.
What is an API gateway?
In a microservices system, clients would otherwise need to know the address, protocol, and auth scheme of every service. The API gateway pattern puts one component in front of them. Clients call the gateway; the gateway decides where each request goes and what rules apply.
Without a gateway, every service reimplements authentication, rate limits, and logging, and every client must change when a service moves. With a gateway, those cross-cutting concerns live in one place and services can change behind a stable public API.
API gateway architecture diagram
Clients (web, mobile, partners, internal jobs)
|
v
[ DNS / CDN / WAF / Load balancer ]
|
v
+--------------------- API GATEWAY (data plane) ---------------------+
| TLS termination -> AuthN/AuthZ -> Rate limit & quota -> |
| Request validation & transform -> Cache lookup -> Router |
| -> Timeouts / retries / circuit breaker -> Response transform |
| -> Logs, metrics, traces |
+--------------------------------------------------------------------+
| | |
v v v
Orders service Users service Payments service (or Lambda, queues, LLM providers)
Control plane: config store, admin API/UI, policy and key management, deploys config to data plane
Core components
| Component | Role |
|---|---|
| Listener / TLS termination | Accepts HTTP/1.1, HTTP/2, gRPC, or WebSocket connections and decrypts TLS |
| Authentication and authorization | Validates API keys, OAuth 2.0 / JWT tokens, or mTLS certificates, then checks permissions |
| Rate limiter and quota | Caps requests per client, key, or route; returns HTTP 429 when exceeded |
| Request validation and transformation | Checks schemas, rewrites paths and headers, converts protocols (REST to gRPC, for example) |
| Cache | Serves repeat responses without calling the backend |
| Router | Matches path, method, host, or header to an upstream service and load-balances across instances |
| Resilience layer | Timeouts, retries with backoff, circuit breakers, and fallbacks |
| Observability | Access logs, metrics (latency, error rate), and distributed traces |
| Control plane | Stores configuration and pushes it to gateway instances; exposes an admin API or UI |
Request lifecycle
- The client sends
GET /orders/42with a bearer token. - The gateway terminates TLS and validates the token.
- It checks the caller's rate limit and quota.
- It validates the request and adds headers such as a request ID.
- It checks the cache, and on a miss routes to a healthy
ordersinstance. - It enforces a timeout, retries once on a connection error, and records the result.
- It transforms the response if needed and returns it, logging status, latency, and caller.
Key features of an API gateway
These are the features most teams need, roughly in order of how often they matter:
- Request routing. Path, host, method, and header-based routing, plus load balancing and canary or weighted releases.
- Authentication and authorization. API keys, OAuth 2.0 and JWT validation, mTLS, and per-route access policies.
- Rate limiting and quotas. Per-consumer limits that protect backends; HTTP 429 "Too Many Requests" is defined in RFC 6585.
- Request and response transformation. Header injection, payload mapping, protocol translation, and API versioning.
- Caching. Response caching with TTLs to cut latency and backend load.
- Security. TLS, IP allow and deny lists, request size limits, and integration with a web application firewall.
- Observability. Structured access logs, metrics, and trace propagation to tools like Grafana Tempo or Datadog.
- Resilience. Timeouts, retries, circuit breakers, and health checks.
- Developer experience. API keys and plans, a developer portal, and usage analytics; often bundled as "API management."
- Monetization and metering. Counting usage per consumer for billing or chargeback.
Deployment patterns
| Pattern | How it works | When to use it |
|---|---|---|
| Single central gateway | One gateway cluster fronts all services | Small to mid-size systems, one team owns the edge |
| Backend for frontend (BFF) | A separate gateway per client type (web, mobile, partner) | Clients need very different APIs |
| Gateway per domain | Each business domain runs its own gateway | Large organizations with autonomous teams |
| Kubernetes ingress / Gateway API | Gateway configured with Kubernetes resources | Services already run on Kubernetes |
| Gateway plus service mesh | Gateway handles north-south (client) traffic; mesh handles east-west (service-to-service) | Many internal services needing mTLS and traffic policy |
The Kubernetes Gateway API is now the standard way to configure gateways on Kubernetes, and implementations such as Envoy Gateway build on it.
API gateway vs load balancer vs service mesh
- A load balancer spreads traffic across instances of one service, mostly at the network or HTTP level, without API-aware policy.
- An API gateway understands APIs: it authenticates consumers, enforces quotas per key, transforms payloads, and routes across many services.
- A service mesh manages traffic between internal services, usually with sidecars or node proxies.
Most production systems use a load balancer in front of the gateway, and some add a mesh behind it.
Popular API gateway products
| Product | Type | Notes (October 2026) |
|---|---|---|
| Amazon API Gateway | Managed (AWS) | REST, HTTP, and WebSocket APIs; the free tier includes 1M REST calls, 1M HTTP calls, and 1M WebSocket messages plus 750,000 connection minutes per month for 12 months |
| Kong Gateway | Open source and enterprise | Plugin-based; runs DB-less with declarative config or with a database |
| Envoy Gateway | Open source | Envoy proxy managed through the Kubernetes Gateway API |
| Azure API Management | Managed (Azure) | Managed and self-hosted gateways plus a developer portal |
Example: Kong declarative configuration
This DB-less Kong config routes /orders to a backend, requires an API key, and limits each consumer to 60 requests per minute:
_format_version: "3.0"
services:
- name: orders
url: http://orders:8080
routes:
- name: orders-route
paths: ["/orders"]
plugins:
- name: key-auth
- name: rate-limiting
config:
minute: 60
policy: local
consumers:
- username: mobile-app
keyauth_credentials:
- key: replace-with-a-long-random-key
Example: a minimal gateway in Python
To see the architecture in code, here is a small, runnable gateway with API-key auth, per-key rate limiting, routing, and a timeout. It is for learning; use a production gateway for real traffic.
# pip install fastapi uvicorn httpx ; run: uvicorn gateway:app --port 8080
import time, httpx
from collections import defaultdict
from fastapi import FastAPI, Request, Response, HTTPException
ROUTES = {"/orders": "http://localhost:9001", "/users": "http://localhost:9002"}
KEYS = {"key-mobile": "mobile-app"}
LIMIT_PER_MIN = 60
hits = defaultdict(list)
app = FastAPI()
@app.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE"])
async def gateway(path: str, request: Request):
consumer = KEYS.get(request.headers.get("x-api-key", ""))
if not consumer:
raise HTTPException(401, "invalid API key")
now = time.time()
hits[consumer] = [t for t in hits[consumer] if now - t < 60]
if len(hits[consumer]) >= LIMIT_PER_MIN:
raise HTTPException(429, "rate limit exceeded", headers={"Retry-After": "60"})
hits[consumer].append(now)
prefix = next((p for p in ROUTES if f"/{path}".startswith(p)), None)
if not prefix:
raise HTTPException(404, "no route")
async with httpx.AsyncClient(timeout=5.0) as client:
upstream = await client.request(request.method, ROUTES[prefix] + f"/{path}"[len(prefix):],
content=await request.body(),
headers={"x-consumer": consumer})
return Response(upstream.content, upstream.status_code,
media_type=upstream.headers.get("content-type"))
From API gateway to LLM gateway
When the backends are AI model providers instead of microservices, the same architecture applies, with a few extra jobs. An LLM gateway (or AI gateway) adds:
| Classic gateway feature | LLM gateway extension |
|---|---|
| Routing by path | Routing by model name, cost, latency, or provider health |
| Rate limiting by requests | Limits by tokens per minute and spend per key |
| Request transformation | Translating between OpenAI, Anthropic, and Gemini request formats |
| Caching responses | Prompt caching and semantic caching |
| Retries and circuit breakers | Fallback to another provider or model on 429, 5xx, or timeout |
| Access logs | Token counts, cost per request, and model used |
| API keys per consumer | One key that works across many providers, with per-team budgets |
App / agent --OpenAI-format request--> LLM gateway
auth + budget check -> model map -> route (primary provider)
-> on 429/5xx/timeout: fallback provider -> usage + cost log
-> Anthropic / OpenAI / Google / DeepSeek / self-hosted vLLM
You can run this layer yourself (for example with LiteLLM or Kong's AI plugins) or use a managed one. Flatkey is a managed option: one OpenAI-compatible endpoint at https://router.flatkey.ai/v1, one key across 300+ models, usage logs with tokens and cost per call, and spend caps per key or group, per its documentation as of October 2026. Because it speaks the OpenAI format, existing clients only change the base URL:
curl https://router.flatkey.ai/v1/chat/completions \
-H "Authorization: Bearer $FLATKEY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"claude-sonnet-5-5","messages":[{"role":"user","content":"Summarize the API gateway pattern in one sentence."}]}'
For a deeper design, read our guides on AI API gateway architecture and model routing, circuit breakers for LLM gateways, AI gateway vs API management, and managed AI gateway vs self-hosted proxy. If you are new to the topic, the LLM gateway beginner guide is a good starting point, and the rate limit handling guide covers 429 strategies.
Design checklist
- Put TLS, auth, and rate limiting at the gateway, not in each service.
- Set a timeout on every upstream route; never wait indefinitely.
- Retry only idempotent requests (GET, PUT, DELETE per RFC 9110) unless the backend supports idempotency keys.
- Propagate a request ID and trace context to every backend.
- Run at least two gateway instances behind a load balancer so the gateway is not a single point of failure.
- Keep business logic out of the gateway; it should enforce policy, not implement features.
- Version your gateway configuration in git and deploy it like code.
FAQ
What is the API gateway pattern in microservices?
It is a design where a single gateway is the entry point for all client requests. The gateway routes each request to the right microservice and handles shared concerns such as authentication, rate limiting, and logging, so clients do not call services directly.
What are the key features of an API gateway?
The core features are request routing, authentication and authorization, rate limiting and quotas, request and response transformation, caching, security controls, observability, and resilience features such as timeouts, retries, and circuit breakers.
What are the main components of API gateway architecture?
A data plane that processes traffic (listener, auth, rate limiter, transformer, cache, router, resilience layer, and telemetry) and a control plane that stores and distributes configuration.
Is an API gateway the same as a load balancer?
No. A load balancer spreads traffic across instances of a service. An API gateway adds API-aware features such as per-consumer authentication, quotas, payload transformation, and routing across many services. Most systems use both.
What is an LLM gateway?
An LLM gateway is an API gateway specialized for AI model traffic. It routes by model and provider, limits by tokens and spend, translates between provider API formats, falls back when a provider fails, and logs token usage and cost.
Is the API gateway a single point of failure?
It can be if you run one instance. Run multiple instances across zones behind a load balancer, keep configuration in a replicated store, and monitor the gateway itself.
