LLM API Fallback Routing: A Production Failover Playbook

Fallback routing for LLM APIs sounds simple until the first real incident: catch an error, switch models, and try again. In production, that rule can turn one provider problem into duplicate tool calls, broken JSON, mixed streaming output, runaway retry traffic, or a response that is technically successful but no longer satisfies the product contract.

A safer design treats fallback as a bounded state machine, not a list of backup model names. Every request moves through a small set of decisions:

1. Is the failure retryable?
2. Is it safe to repeat this request?
3. Should the next attempt use the same target or a different one?
4. Can the fallback preserve the required contract?
5. Has the request already produced output or side effects?
6. Has the end-to-end latency and attempt budget been exhausted?

This playbook turns those questions into an error matrix, routing policy, TypeScript controller, test plan, and rollout checklist for multi-provider LLM applications.

The four actions behind reliable LLM API fallback routing

Do not send every error into the same retry loop. A production router needs four distinct actions.

Action	Use it when	Typical examples
Retry the same target	The failure looks transient and the current deployment may recover within the request deadline	Connection reset before headers, isolated timeout, short rate-limit wait
Fail over to an equivalent target	The provider, region, deployment, or account is unhealthy but the same model contract is available elsewhere	Regional outage, exhausted deployment quota, repeated 5xx responses
Fall back to another model	An evaluated alternate model can preserve the application's minimum capability and output contract	Primary model unavailable and a tested secondary model supports the same tools and schema
Stop and surface the error	Repeating the request will not fix it, may create side effects, or cannot preserve the contract	Invalid authentication, malformed request, unsupported parameter, policy block, partial stream

The distinction between failover and fallback matters. Failover keeps the logical model contract and changes infrastructure. Fallback changes the model or capability tier. Failover is usually the lower-risk move.

If you need the broader request-path design around aliases, health scoring, billing, and observability, start with the AI API gateway architecture guide. This article focuses on the controller that runs after a target has been selected.

Build an error-to-action matrix before writing retry code

Provider SDKs expose different exception classes and response bodies, but the router should normalize them into a small internal taxonomy.

Normalized failure	Retry same target?	Equivalent failover?	Cross-model fallback?	Notes
Connect failure before request acceptance	Yes, once	Yes	Maybe	Stay inside one end-to-end deadline
Timeout before response headers	Maybe	Yes	Maybe	Only repeat requests that are safe to replay
429 rate limit	After bounded delay	Yes	Maybe	Respect server guidance when available; do not create a retry storm
Provider 5xx or overload	Once at most	Yes	Maybe	Open the circuit after a defined failure threshold
Authentication or permission error	No	No	No	Fix credentials or policy; model switching does not help
Malformed request or unsupported parameter	No	No	No	Correct the client contract
Context length exceeded	No blind retry	No	Only with an explicit adaptation	Truncation, summarization, or a larger-context route changes the request
Safety or policy rejection	No blind retry	No	Usually no	Switching providers to evade a policy decision is not a reliability strategy
Output-schema validation failure	Maybe with repair	No	Only if evaluated	Keep schema repair separate from transport retries
Stream fails before first token	Maybe	Yes	Maybe	No user-visible output exists yet
Stream fails after output begins	No automatic switch	No	No automatic switch	Do not splice two model responses together
Tool call may already have executed	No blind retry	No	No blind retry	Require idempotency keys or tool-level deduplication

Official provider documentation reinforces why normalization is necessary. Anthropic documents distinct rate-limit, API, and overload errors and notes that a streaming request can still fail after an initial successful response. OpenAI likewise separates invalid requests, rate limits, and server-side failures. Your application should translate provider-specific signals into stable internal decisions instead of embedding provider names throughout business logic.

Put one retry budget around the whole request

Retries often exist in several places at once: the HTTP client, provider SDK, gateway, background job, and application service. If each layer performs three attempts, a single user action can multiply into far more upstream calls than the team intended.

The safer pattern is:

• Choose one layer to own LLM retries and fallback.
• Set one end-to-end deadline for the user request or job.
• Set a maximum number of upstream attempts.
• Reserve part of the deadline for the fallback target.
• Use exponential backoff with jitter for transient failures.
• Stop when the remaining time cannot support another meaningful attempt.

AWS's guidance on timeouts, retries, backoff, and jitter describes how retries can amplify overload and recommends bounded behavior rather than constant immediate repetition. The same principle applies to model APIs, where a provider under load is least able to absorb synchronized retry traffic.

A practical interactive budget might be expressed as policy rather than hard-coded sleeps:

type RetryBudget = {
  deadlineMs: number;
  maxAttempts: number;
  maxSameTargetAttempts: number;
  reserveForFallbackMs: number;
};

The exact values depend on the product. A chat interface, coding agent, batch evaluator, and asynchronous video workflow should not share the same budget.

Use circuit breakers to stop routing into known failures

A circuit breaker prevents every new request from rediscovering the same outage.

The standard states are:

• Closed: requests flow normally while the router measures failures and latency.
• Open: the target is temporarily ineligible because its recent behavior crossed a threshold.
• Half-open: a small number of probe requests test whether the target has recovered.

Azure's circuit-breaker pattern describes this closed/open/half-open lifecycle. For LLM routing, the breaker key should be specific enough to isolate the failing surface. Useful dimensions include provider, model, region, deployment, account, and capability. A text-completion deployment can be healthy while a tool-calling route or regional endpoint is failing.

Avoid opening circuits on every client error. Invalid authentication, malformed requests, context overflow, and policy rejection usually say more about the request than provider health. Breakers should react primarily to transient infrastructure signals such as connection failures, timeouts, overload, and server errors.

Preserve a capability contract across models

A fallback model is not safe merely because it accepts an OpenAI-compatible request. Define the minimum contract for each route alias.

route: support-agent-v3
requires:
  modalities: [text]
  streaming: true
  tools: true
  parallel_tool_calls: false
  structured_output: json_schema
  context_window_min: 64000
  max_output_tokens_min: 4000
quality_gates:
  task_success_rate_min: 0.94
  schema_valid_rate_min: 0.995
policy:
  same_model_failover_first: true
  cross_model_fallback_allowed: true

Before adding a target to the fallback set, test at least:

• Supported request parameters
• Tool definition and tool-call behavior
• Structured-output validity
• Streaming event shape
• Context and output limits
• Safety behavior appropriate to the application
• Token accounting fields used by cost controls
• Latency and quality on representative prompts

This contract-first approach is especially important for workflows that cross modalities. The multimodal agent routing guide covers additional checks for text, image, audio, and video routes.

A TypeScript fallback controller

The following example is intentionally provider-neutral. It assumes upstream adapters normalize errors and responses before the routing layer sees them.

type FailureKind =
  | "connect"
  | "timeout"
  | "rate_limit"
  | "overloaded"
  | "server_error"
  | "invalid_request"
  | "auth"
  | "policy"
  | "context_overflow"
  | "partial_stream"
  | "unknown";

type Target = {
  id: string;
  contractId: string;
  healthy: boolean;
  circuit: "closed" | "open" | "half_open";
};

type RequestState = {
  attempt: number;
  sameTargetAttempts: number;
  deadlineAt: number;
  outputStarted: boolean;
  sideEffectsPossible: boolean;
};

function canReplay(state: RequestState): boolean {
  return !state.outputStarted && !state.sideEffectsPossible;
}

function isTransient(kind: FailureKind): boolean {
  return [
    "connect",
    "timeout",
    "rate_limit",
    "overloaded",
    "server_error",
  ].includes(kind);
}

function chooseNextAction(
  kind: FailureKind,
  state: RequestState,
  current: Target,
  equivalent: Target | undefined,
  fallback: Target | undefined,
) {
  if (!canReplay(state) || kind === "partial_stream") return { type: "stop" };

  if (!isTransient(kind)) return { type: "stop" };

  if (state.attempt >= 3 || Date.now() >= state.deadlineAt) {
    return { type: "stop" };
  }

  if (
    state.sameTargetAttempts < 1 &&
    current.circuit === "closed" &&
    current.healthy
  ) {
    return { type: "retry", target: current };
  }

  if (equivalent?.healthy && equivalent.circuit !== "open") {
    return { type: "failover", target: equivalent };
  }

  if (
    fallback?.healthy &&
    fallback.circuit !== "open" &&
    fallback.contractId === current.contractId
  ) {
    return { type: "fallback", target: fallback };
  }

  return { type: "stop" };
}

Production code also needs jittered delays, cancellation propagation, request IDs, circuit-breaker updates, telemetry, and adapter-specific error parsing. The important property is that replay safety and contract compatibility are checked before another target is selected.

Treat streaming fallback as a separate protocol

Streaming creates a hard boundary: once content reaches the client, the gateway can no longer pretend the attempt never happened.

If the upstream fails before the first event is forwarded, a retry or fallback may still be transparent. After the first token, tool delta, image event, or audio chunk is delivered, automatic model switching risks combining two incompatible responses.

Use one of these explicit strategies:

1. Fail the stream clearly. Return a stable error event with the request ID and let the client offer retry.
2. Buffer before release. For short structured responses, validate the complete result before sending it downstream. This sacrifices time-to-first-token.
3. Implement application-level resume. Start a new turn with explicit context saying the previous response was interrupted. Treat it as a new model generation, not a continuation of the same byte stream.

Do not silently concatenate output from two models.

Separate tool-call reliability from model-call reliability

An LLM request may be replayable while the tool it selected is not. A payment, email, deployment, database write, or ticket creation can succeed even if the model connection fails before the application records the result.

Protect write-side tools with:

• An idempotency key derived from the user operation, not the provider attempt
• A durable tool-execution record
• Deduplication at the tool boundary
• A clear distinction between planned, started, succeeded, and unknown
• Human review for uncertain high-impact side effects

If side effects are possible and their result is unknown, stop automatic fallback. First reconcile the tool state.

Observe fallback as a product outcome

A low provider error rate does not prove that fallback is working. Track the full route outcome.

Metric	What it reveals
Primary-target success rate	Baseline provider or deployment health
Retry recovery rate	Whether same-target retries are useful
Equivalent failover recovery rate	Value of redundant deployments or regions
Cross-model fallback recovery rate	Value of the alternate model set
Contract rejection rate	How often candidate targets fail eligibility checks
Post-fallback schema validity	Whether “successful” responses remain usable
Post-fallback task success	Whether users still complete the intended job
Added fallback latency	Reliability cost paid by the user
Fallback cost delta	Billing impact of the recovery path
Circuit open duration and probe success	Whether breaker thresholds and recovery timing are sensible

Log a route reason for every attempt: selected target, normalized error, retry delay, circuit state, fallback reason, request deadline remaining, and final outcome. Avoid logging sensitive prompts or outputs unless the product's data policy explicitly allows it.

Test the failure paths before enabling automatic fallback

Run failure injection in a staging environment and then canary the policy in production.

Transport and provider tests

• Drop the connection before response headers.
• Return repeated rate limits with and without retry guidance.
• Simulate overload and server errors.
• Delay the primary until the request deadline is nearly exhausted.
• Open a target circuit and verify traffic moves to an eligible route.
• Recover the target and verify half-open probes do not restore full traffic too early.

Contract tests

• Remove a required tool from the fallback adapter.
• Return invalid structured output.
• Change a streaming event shape.
• Exceed context or output limits.
• Compare fallback quality on a fixed evaluation set.

Replay-safety tests

• Fail before and after the first streamed event.
• Fail after a write-side tool begins.
• Repeat the same idempotency key.
• Cancel the client request while the fallback attempt is pending.

The test passes only when the router chooses the expected action and records why.

Where Flatkey fits

Flatkey provides one API key and an OpenAI-compatible base URL for supported models, with centralized usage and billing. That creates a stable integration boundary for multi-model access and routing.

Application teams should still own the route contract described in this playbook: which errors may retry, which targets are equivalent, whether cross-model fallback is allowed, how tools are deduplicated, and what quality threshold a recovered response must meet.

For the shortest integration path, use the Flatkey integration starter. If you are migrating an existing client, the OpenAI-compatible API gateway checklist covers base URL, parameters, streaming, and error-shape verification.

Production rollout checklist

• Normalize provider errors into a stable internal taxonomy.
• Define retry, equivalent failover, cross-model fallback, and stop actions.
• Assign one component to own the retry budget.
• Enforce one end-to-end deadline and maximum attempt count.
• Add exponential backoff with jitter for transient failures.
• Key circuit breakers by the smallest useful failure domain.
• Define a versioned capability contract for every route alias.
• Block automatic switching after partial output begins.
• Add idempotency and reconciliation for write-side tools.
• Record route reasons and final task outcomes.
• Inject transport, overload, contract, streaming, and side-effect failures.
• Canary equivalent failover before enabling cross-model fallback.
• Add kill switches for each target and fallback policy.

FAQ

What is fallback routing for LLM APIs?

Fallback routing for LLM APIs is a reliability policy that selects another eligible model or provider when the preferred route cannot complete a request. Safe fallback checks replay safety, capability compatibility, circuit health, latency budget, and output state before switching.

What is the difference between an LLM retry and fallback?

A retry repeats the request against the same target. Failover moves to equivalent infrastructure while preserving the logical model contract. Cross-model fallback changes the model and therefore requires stronger compatibility and quality testing.

Should an LLM API retry every 429 or 5xx error?

No. Retries should be bounded by an end-to-end deadline, attempt limit, backoff policy, circuit state, and replay-safety check. Equivalent failover may be better than repeatedly calling an unhealthy target.

Can an LLM router switch models during a stream?

Not transparently after output has reached the client. The safe default is to fail the stream clearly or start a new application-level turn. Concatenating partial outputs from different models can corrupt the response contract.

When should cross-model fallback be disabled?

Disable it when the alternate model cannot preserve required tools, structured output, context limits, safety behavior, quality thresholds, or side-effect guarantees. Also disable automatic replay after partial output or uncertain tool execution.

How many fallback attempts should an LLM request make?

There is no universal number. Use the smallest bounded attempt count that fits the product's latency budget and test evidence. The router should stop when the remaining deadline cannot support another useful attempt.

Reliable fallback does not mean “try everything.” It means making the next action explicit, compatible, replay-safe, observable, and easy to stop.