LLM Rate Limits Explained: RPM, TPM, and Retries

LLM rate limits decide how much traffic your application can send to a model within a time window. The two limits engineers encounter most often are RPM (requests per minute) and TPM (tokens per minute). A workload can stay under one and still exceed the other.

That distinction matters. If you treat every 429 as a generic request-count problem, you may add retries that increase token pressure, inflate latency, and make an outage worse. A production-safe design identifies the constrained resource first, then chooses among pacing, queuing, retrying, reducing tokens, or routing elsewhere.

This guide explains the mechanics, provides capacity-planning formulas, and includes a bounded TypeScript retry pattern for OpenAI-compatible APIs.

RPM vs. TPM: the quick answer

Limit	Measures	Workloads that hit it first	Best first response
RPM	Requests admitted during a provider-defined time window	Many small calls, agent tool loops, high fan-out evaluations	Pace requests, batch work, or queue bursts
TPM	Input and/or output tokens admitted during a time window	Long context, large outputs, retrieval-heavy prompts, parallel evaluations	Reduce token volume, cap output, or route capacity
RPD	Requests per day	Scheduled crawls, broad offline evaluations, free-tier workloads	Reschedule or raise the service tier
Concurrent requests	Requests in flight at once	Slow generations and streaming workloads	Bound workers and apply backpressure
429	A limit or capacity policy rejected the request	Any workload that exceeds an active bucket	Classify the error before retrying

RPM controls frequency. TPM controls throughput. Concurrency controls simultaneous work. They interact, but they are not interchangeable.

Why a request can fail below the headline limit

A published limit such as 600 RPM does not necessarily mean a client can send 600 requests in the first second of every minute. Providers commonly enforce limits with rolling windows or token-bucket-style controls. A short burst can exhaust the immediately available capacity even when the arithmetic for the whole minute looks safe.

Other reasons a 429 can appear early include:

• The limit applies to a project, organization, account, model family, or service tier rather than one API key.
• Input and output tokens use separate buckets.
• Multiple workers, services, or users share the same quota pool.
• Retries from previous failures are consuming the same limit.
• The provider is applying an acceleration or burst limit while traffic ramps sharply.
• A model-specific pool is full even though another model still has capacity.

This is why the application should not infer the cause from its own request counter alone. Read the response body and headers, preserve provider request IDs, and log the model, account, token estimates, attempt number, and queue delay.

A practical capacity formula

Start with two independent ceilings.

request_ceiling = RPM × safety_factor

token_ceiling = (TPM × safety_factor) ÷ average_tokens_per_request

safe_requests_per_minute = min(request_ceiling, token_ceiling)

Use a safety factor below 1.0—for example 0.7 to 0.9—to absorb token variance, retries, shared consumers, and uneven arrival patterns.

Example

Assume a model pool allows:

• 1,000 RPM
• 2,000,000 TPM
• 4,000 average total tokens per request
• 80% operating safety factor

request_ceiling = 1,000 × 0.8 = 800 requests/minute

token_ceiling = (2,000,000 × 0.8) ÷ 4,000
              = 400 requests/minute

safe_requests_per_minute = min(800, 400) = 400

TPM is the binding constraint. Adding more workers will not increase sustainable throughput; it will only create a larger queue or more 429 responses.

For online systems, translate throughput into a concurrency starting point with Little’s Law:

target_concurrency ≈ requests_per_second × average_request_seconds

If the safe rate is 400 requests per minute (6.67 per second) and average model latency is 3 seconds, a starting concurrency is about 20. Add headroom carefully, then tune against real p95 latency and token distributions.

Token limits are often the hidden bottleneck

Teams frequently monitor request counts but overlook token volume. TPM pressure rises when you:

• Add more retrieved documents to each prompt.
• Keep long conversation histories.
• Run several candidate completions per task.
• Increase output caps.
• Send the same large system prompt repeatedly.
• Launch parallel evaluation suites against one project quota.

Measure at least four token values per successful request:

1. Input tokens.
2. Output tokens.
3. Total tokens.
4. A rolling p50, p95, and maximum by workload and model.

Capacity planning with only the average is optimistic. A safer scheduler reserves against a high percentile or a workload-specific estimate, then reconciles the reservation with actual usage after completion.

What a 429 means—and what it does not

HTTP 429 Too Many Requests tells you the server rejected the request under an active limit or capacity policy. It does not automatically mean “sleep one second and try again.”

Classify a 429 into an operational bucket:

429 class	Evidence	Correct action
Short burst	Recent spike; retry header is short; queue is otherwise healthy	Wait for the server hint, then retry with jitter
Sustained RPM exhaustion	Request rate stays near the ceiling	Pace or queue; retries alone cannot fix it
Sustained TPM exhaustion	Token rate is high; long prompts or outputs dominate	Reduce tokens, delay work, or route to another eligible pool
Daily or tier limit	Error identifies daily quota, billing, or tier restriction	Stop retries; reschedule or change account capacity
Acceleration limit	Traffic ramped quickly from a low baseline	Ramp gradually and smooth bursts
Provider capacity event	Normal client rate but repeated transient rejection	Use a small retry budget, then contract-safe fallback

The body and headers differ by provider. Prefer an explicit Retry-After or rate-limit reset signal when supplied. Otherwise, use exponential backoff with random jitter.

Exponential backoff with jitter

Exponential backoff increases the delay after each failed attempt. Jitter randomizes that delay so hundreds of workers do not retry at the same instant.

One common full-jitter formula is:

delay_ms = random(0, min(cap_ms, base_ms × 2^attempt))

A production retry policy also needs boundaries:

• Maximum attempts: usually a small number, not an infinite loop.
• Maximum elapsed time: stop when the caller’s latency budget is exhausted.
• Retryable status list: commonly 429, selected 5xx responses, and safe network failures.
• Server hints: honor Retry-After when it is valid.
• Cancellation: stop immediately when the upstream request is aborted.
• Observability: record attempt count, wait time, final status, and provider request ID.

Unsuccessful requests may still consume rate-limit capacity. Aggressive retries can therefore extend the period of throttling.

TypeScript: a bounded retry helper

The following example uses the OpenAI-compatible Flatkey base URL. It retries only before a successful response body is consumed and stops when either the attempt limit or total time budget is exhausted.

type ChatRequest = {
  model: string;
  messages: Array<{ role: "system" | "user" | "assistant"; content: string }>;
  max_tokens?: number;
};

const sleep = (milliseconds: number) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

function retryAfterMilliseconds(response: Response): number | null {
  const value = response.headers.get("retry-after");
  if (!value) return null;

  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);

  const date = Date.parse(value);
  return Number.isNaN(date) ? null : Math.max(0, date - Date.now());
}

export async function createChatCompletion(
  apiKey: string,
  request: ChatRequest,
  options: { maxAttempts?: number; maxElapsedMs?: number } = {},
) {
  const maxAttempts = options.maxAttempts ?? 4;
  const maxElapsedMs = options.maxElapsedMs ?? 30_000;
  const startedAt = Date.now();

  for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
    const response = await fetch(
      "https://router.flatkey.ai/v1/chat/completions",
      {
        method: "POST",
        headers: {
          Authorization: `Bearer ${apiKey}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify(request),
      },
    );

    if (response.ok) return response.json();

    const retryable = response.status === 429 || response.status >= 500;
    const finalAttempt = attempt === maxAttempts - 1;
    if (!retryable || finalAttempt) {
      throw new Error(`LLM request failed with ${response.status}: ${await response.text()}`);
    }

    const hintedDelay = retryAfterMilliseconds(response);
    const exponentialCap = Math.min(8_000, 500 * 2 ** attempt);
    const jitteredDelay = Math.random() * exponentialCap;
    const delayMs = hintedDelay ?? jitteredDelay;

    if (Date.now() + delayMs - startedAt >= maxElapsedMs) {
      throw new Error("LLM retry budget exhausted");
    }

    await sleep(delayMs);
  }

  throw new Error("Unreachable retry state");
}

For production use, add request timeouts, structured error types, metrics, and your application’s cancellation signal. If the operation can create external side effects—such as sending an email or executing a tool—make the business operation idempotent before retrying it.

Retry, queue, reduce, or route?

Use the cause, not the status code alone, to choose the control.

Situation	Retry	Queue	Reduce tokens	Route elsewhere
One isolated transient 429	Yes, bounded	Optional	No	Usually no
Repeated RPM exhaustion	Limited	Yes	No	Sometimes
Repeated TPM exhaustion	Limited	Yes	Yes	Often useful
Daily quota exhausted	No	For later	Optional	Yes, if policy allows
Provider 5xx incident	Yes, bounded	Yes	No	Yes after retry budget
Partial streaming response	No automatic replay	Application-specific	No	Only with explicit recovery semantics

Retry

Retry when the failure is transient and the caller still has time. Keep a per-request retry budget and a service-level retry budget so a provider incident cannot multiply total traffic.

Queue

Queue when arrival rate temporarily exceeds sustainable service rate. A useful queue exposes:

• Oldest-item age.
• Estimated start time.
• Per-tenant fairness.
• Cancellation for stale jobs.
• A maximum depth with explicit shedding behavior.

Reduce tokens

When TPM binds, remove irrelevant context, summarize history, reduce candidate count, lower output caps, cache reusable prompt prefixes where supported, and separate short interactive traffic from long batch jobs.

Route elsewhere

Routing is appropriate when another model or provider satisfies the same contract and has healthy capacity. The fallback must preserve required capabilities such as structured output, tool calling, context length, safety policy, and latency target. For a deeper implementation framework, use the LLM API fallback routing playbook.

Rate limits in model evaluations

Poor throttling can invalidate an evaluation.

Suppose model A is tested with 10 workers while model B is tested with 100. If model B spends more time throttled, its measured latency includes queueing and retry delay that model A never faced. The result may describe your harness configuration rather than model performance.

For defensible comparisons:

1. Separate model latency, queue delay, and retry delay.
2. Apply the same arrival process or normalized utilization level to every provider.
3. Warm traffic gradually when providers use acceleration controls.
4. Record tokens and attempts per completed task.
5. Report both first-attempt success rate and eventual success rate.
6. Compare cost per accepted task, not only cost per token.
7. Run overload tests separately from quality and latency benchmarks.

If you are evaluating providers on both reliability and cost, pair this process with the AI API pricing comparison.

A production rate-limit architecture

A robust request path usually has five control layers:

1. Admission control rejects or defers work that cannot meet its deadline.
2. Token reservation estimates the request’s likely quota cost.
3. Rate limiter paces each provider, model pool, tenant, and priority class.
4. Retry controller spends a bounded retry budget with jitter.
5. Router selects a contract-compatible alternative after the retry budget or capacity policy says to move.

client
  → admission control
  → priority queue
  → RPM + token reservation limiter
  → provider/model route
  → bounded retry
  → contract-safe fallback
  → usage and latency logs

Do not place an unbounded retry loop in every application worker. Centralize policy so all callers share the same understanding of available capacity.

Metrics worth alerting on

Track these by provider, model, project, route, tenant, and workload:

• Requests per minute and tokens per minute.
• Estimated tokens reserved versus actual tokens used.
• First-attempt success rate.
• Retry attempts per successful request.
• 429 rate by classified cause.
• Queue depth and oldest-item age.
• Time spent waiting for rate capacity.
• End-to-end p50, p95, and p99 latency.
• Fallback rate and fallback outcome.
• Cost per successful or accepted task.

An alert on total 429 count alone is noisy. A better signal combines throttling rate with queue age, retry amplification, and final failure rate.

Common mistakes

Treating RPM as a concurrency limit

RPM measures admissions over time; concurrency measures in-flight work. Slow requests can create high concurrency at a modest RPM.

Retrying every 429 immediately

Immediate retries synchronize workers and consume more capacity. Honor server timing when available and add jitter.

Using one limiter for every model

Providers may use separate or shared pools. Model-specific policy should follow the provider’s documented quota scope and observed headers.

Ignoring shared consumers

A dashboard, batch job, and production API may share one project quota. Reserve capacity by workload and isolate critical traffic where possible.

Retrying a partial stream

Once tokens have reached the user, replaying the request can duplicate content or tool actions. Define explicit continuation or restart semantics instead of silently retrying.

How Flatkey changes the operational model

Flatkey provides one API key and an OpenAI-compatible base URL for access across supported model providers. That gives an application one integration surface while still allowing routing policy to consider model fit, capacity, reliability, and cost.

The gateway does not remove upstream limits. It makes it easier to implement consistent controls around them: one client integration, centralized request logs, and the option to move eligible traffic when a route is constrained. Review the AI API gateway architecture guide for the broader routing design, or see current Flatkey pricing before selecting production routes.

FAQ

What is the difference between RPM and TPM?

RPM limits how many requests are admitted over time. TPM limits how many input and/or output tokens are admitted. Small prompts usually pressure RPM first; large-context or high-output workloads often pressure TPM first.

Why do I get 429 errors below my RPM limit?

The provider may enforce shorter rolling windows, token buckets, shared project quotas, separate token limits, acceleration limits, or model-specific pools. Your local request counter may not represent the full quota scope.

Should I retry every 429?

No. Retry short-lived throttling with a small budget and jitter. Do not repeatedly retry daily quota exhaustion, billing restrictions, or sustained overload that has no time to recover.

Does exponential backoff guarantee success?

No. Backoff reduces collision and gives transient capacity time to recover. It cannot create quota. Persistent exhaustion requires lower demand, more capacity, delayed work, or another eligible route.

How many retries should an LLM request use?

There is no universal number. Set attempts from the user-visible latency budget and failure mode. Many interactive applications should allow only a few short attempts before failing or routing elsewhere; offline jobs can tolerate longer queues.

Do retries count against rate limits?

They can. Providers may count unsuccessful attempts toward active limits, so retry amplification must be monitored and bounded.

Final checklist

• Model RPM, TPM, daily limits, and concurrency separately.
• Calculate capacity from the minimum of request and token ceilings.
• Operate below the published maximum with a safety factor.
• Use queues and pacing for sustained load.
• Honor Retry-After and use exponential backoff with jitter.
• Cap attempts and total retry time.
• Do not automatically replay partial streams or side effects.
• Route only to models that preserve the required contract.
• Separate queue and retry delay from model latency in evaluations.
• Alert on retry amplification and queue age, not only raw 429 counts.

Rate limits are a capacity-planning problem before they are a retry problem. Once you measure request frequency, token throughput, concurrency, and retry amplification independently, 429 errors become actionable signals instead of unpredictable production noise.