Fallback routing for LLM APIs is straightforward when every request returns text. If the primary route times out, you retry once, move to a backup model, validate the schema, and log the reason.
That logic breaks down when the same product also routes image generation, audio transcription, speech output, and video generation. A backup that is safe for JSON extraction is not automatically safe for brand imagery. A speech-to-text fallback is not the same thing as a text-to-speech fallback. A video fallback can create duplicate spend, queue confusion, or an output that still needs human approval.
This guide turns Fallback Routing for LLM APIs: Multimodal Agent Routing for Text, Image, Audio, and Video into an implementation playbook for AI product teams. The goal is not to pick one universal model. The goal is to make route choice, verification, fallback, and cost review explicit for every modality your agent touches.
As of September 12, 2026, Flatkey's live public pages frame the product around one key, one balance, one invoice, 100+ official AI models, 1,000+ tools, OpenAI-compatible setup, usage logs, and automatic failover. Flatkey's public model and pricing surfaces also expose text, image, audio, and video-oriented model routes. Treat those live pages as the current commercial checkpoint before rollout because model availability, prices, and route health can change.
Quick answer: what changes in multimodal fallback routing?
In text-only systems, fallback usually asks one question: "Can another route produce an equivalent answer?"
In multimodal systems, fallback routing for LLM APIs has to ask four questions:
- What artifact is being produced? Text, image, audio, and video do not share one quality standard.
- How is the artifact verified? A JSON parser, visual reviewer, transcript checker, and video approval queue are different verification systems.
- What failure is actually recoverable? Rate limits, timeouts, invalid inputs, safety rejections, and poor-quality outputs need different actions.
- Can the backup preserve the contract? A fallback route is safe only when it preserves the modality, output shape, latency budget, cost ceiling, and review path.
The practical rule is simple: fallback stays inside the workflow class unless a human or policy layer explicitly approves the move.
Why generic retries fail for text, image, audio, and video
Generic retries treat every failed request as a transport problem. That is too shallow for multimodal agent routing.
| Modality | Common failure shape | Why generic retry is risky |
|---|---|---|
| Text | Timeout, rate limit, parser failure, tool-call mismatch, weak answer | Retrying may be safe, but switching models can break schemas, citations, tools, or tone. |
| Image | Safety block, wrong aspect ratio, brand mismatch, low fidelity, bad edit adherence | Another image model may return a valid file that still fails creative review. |
| Audio | Unsupported format, poor transcript, wrong language, voice mismatch, latency spike | Transcription, translation, and speech output are separate jobs with separate fallback rules. |
| Video | Long queue, failed task, expired asset, inconsistent motion, expensive rejected output | Resubmitting blindly can create duplicate jobs, duplicate spend, and more assets to review. |
The router needs to know whether a failure is a retryable service event, an invalid request, a safety or policy result, or a quality miss. Those categories lead to different actions.
Start with route classes, not model names
The safest fallback routing for LLM APIs starts with route classes. Models can move in and out of a class as availability and quality change; the workflow policy should stay stable.
| Route class | Primary jobs | Verification class | Default fallback behavior |
|---|---|---|---|
| Text reasoning | Extraction, classification, summarization, coding, structured output, tool calls | Schema checks, parser tests, rubric sampling, tool success | Auto-fallback can be allowed when the backup preserves response shape and tool behavior. |
| Vision understanding | Image analysis, document review, screenshot interpretation | Text output checks plus visual input coverage | Fallback to another vision-capable reasoning route, not to an image-generation route. |
| Image generation or editing | New images, edits, product variants, creative drafts | Visual QA, brand checks, prompt-template review | Fallback only to an approved image route with matching aspect, style, and review rules. |
| Audio input | Transcription, translation, meeting summaries, call QA | Transcript accuracy checks, language checks, timestamp review | Fallback to another transcription or audio-understanding route with the same input limits. |
| Audio output | Speech synthesis, voice agent response, narration | Playback QA, voice policy, latency budget | Fallback to an approved voice route; keep it separate from transcription fallback. |
| Video generation | Text-to-video, image-to-video, preview clips, creative assets | Human approval, duration/fidelity checks, queue monitoring | Keep fallback narrow; often fail to manual review instead of automatically creating another job. |
This route-class map prevents a common mistake: treating "multimodal model" as one capability bucket. A model that accepts images for analysis does not necessarily generate images. A model that understands audio is not automatically a production voice model. A model that can produce a video in a demo is not automatically safe for customer-facing automation.
Use a fallback decision matrix before you write code
A decision matrix gives product, engineering, and operations one shared policy. Start with the matrix below and replace the examples with your own models, providers, and approval rules.
| Condition | Text action | Image action | Audio action | Video action |
|---|---|---|---|---|
| Primary route timeout before output starts | Retry within budget, then fallback to approved text route | Retry once if no asset was created, then fallback to approved image route | Retry or fallback within the same audio job type | Check task state first; do not create a duplicate job until the original state is known. |
| Rate limit or capacity error | Fallback when backup meets schema and cost limits | Fallback when backup meets prompt, size, and review limits | Fallback when backup supports the same file format and language | Fallback only if duplicate cost and queue behavior are acceptable. |
| Invalid request or unsupported input | Stop and fix request | Stop and fix prompt, input type, size, or aspect | Stop and fix media format, duration, language, or sample rate | Stop and fix duration, reference media, prompt, or endpoint fields. |
| Safety or policy rejection | Do not hide the rejection with fallback | Do not bypass policy with another model | Do not bypass policy with another audio provider | Do not bypass policy with another video provider. |
| Output fails automated verification | Fallback if the contract is machine-checkable | Send to visual review or regenerate under approved route policy | Send to transcript/playback review or fallback inside the same job type | Send to human review; use regeneration rules, not blind failover. |
| Partial streaming output already reached the user | Usually stop and reconcile; avoid transparent model switch | Not applicable for most image routes | Stop and reconcile user-visible state | Do not switch invisibly after a visible job state exists. |
| Tool call may have created side effects | Reconcile idempotency before retry or fallback | Reconcile any asset/storage writes | Reconcile transcript, call, or voice-session state | Reconcile task ID, storage, billing, and approval state first. |
This matrix is the article's main value asset. It turns fallback routing for LLM APIs from an engineering afterthought into a policy that can be reviewed before production traffic moves.
A reference architecture for multimodal agent routing
A production router does not need to be complicated, but it does need clean boundaries.
flowchart LR
A[Agent request] --> B[Classify workflow]
B --> C[Select primary route]
C --> D[Call provider or gateway]
D --> E[Verify artifact]
E -->|pass| F[Return and log]
E -->|retryable failure| G[Fallback policy]
E -->|policy or invalid input| H[Stop with actionable error]
G --> I[Approved backup route]
I --> D
G --> J[Human review or manual queue]
The key pieces are:
- Workflow classifier: labels the job as text, vision, image generation, audio input, audio output, or video generation before route selection.
- Capability registry: records which routes support each modality, endpoint, input format, output shape, maximum context, streaming behavior, and cost class.
- Fallback policy: defines retryable errors, allowed backups, anti-bypass rules, and maximum attempts.
- Verification layer: checks the artifact before returning it to the user or moving it downstream.
- Ledger and observability: logs model, route, owner, status, fallback reason, latency, usage, cost, and reviewer outcome.
Flatkey can sit in the provider-or-gateway layer when a team wants one OpenAI-compatible router, one shared balance, and one place to review supported model routes. Your application still owns the policy: which modality class is allowed to fallback, when to stop, and when a human should decide.
Define the policy as data
Keep routing policy out of scattered if statements. A small data object is easier to review and safer to change.
{
"workflow": "support_ticket_with_screenshot",
"route_class": "vision_understanding",
"primary_model": "primary-vision-reasoning-route",
"fallback_models": ["backup-vision-reasoning-route"],
"fallback_on": ["timeout_before_output", "rate_limit", "upstream_5xx"],
"do_not_fallback_on": ["safety_rejection", "invalid_input", "partial_stream_sent"],
"verification": {
"type": "structured_text",
"required_fields": ["summary", "severity", "evidence"]
},
"max_attempts": 2,
"max_incremental_cost_usd": 0.05,
"log_fields": [
"request_id",
"workspace_id",
"route_class",
"model",
"fallback_reason",
"latency_ms",
"cost_usd",
"verification_status"
]
}
The same structure can represent media workflows:
{
"workflow": "product_video_preview",
"route_class": "video_generation",
"primary_model": "primary-video-route",
"fallback_models": ["approved-video-preview-route"],
"fallback_on": ["provider_5xx", "capacity_unavailable"],
"do_not_fallback_on": ["client_timeout_without_task_state", "safety_rejection", "asset_created"],
"verification": {
"type": "human_review",
"checks": ["duration", "motion", "brand_safety", "reference_consistency"]
},
"max_attempts": 1,
"manual_queue_after_failure": true
}
Notice the difference: the text policy can often fallback automatically because the result is machine-checkable. The video policy is narrower because a second generation is not merely a retry; it is another artifact and another cost event.
Per-modality fallback rules
Text: preserve the response contract
Text is the easiest place to start because verification can be automated. For fallback routing for LLM APIs, a text backup should preserve:
- endpoint shape;
- streaming event shape if used;
- JSON or structured-output behavior;
- tool-call behavior;
- citation or retrieval requirements;
- latency ceiling;
- cost ceiling.
The mistake is falling back from a predictable structured-output route to a more creative model that sounds better but breaks downstream parsing. If the route serves product automation, parser stability matters more than prose style.
Vision: separate image understanding from image generation
Vision jobs often produce text about an image. Image-generation jobs produce a new visual artifact. They should not share a fallback bucket.
OpenAI, Google Gemini, and Anthropic all document image-understanding or image-related API capabilities, but those docs describe different request surfaces and model behaviors. For multimodal agent routing, the policy question is not simply "can this route handle images?" It is "does this route handle this image job with the same input constraints and output contract?"
For example, a screenshot triage workflow may need a vision-capable reasoning route that returns structured text. A brand creative workflow needs an image-generation route that returns a file and passes visual review. Those are separate route classes.
Image generation: do not bypass creative review
Image fallback should protect the creative contract:
- aspect ratio;
- resolution or size class;
- edit versus net-new generation;
- prompt-template compatibility;
- brand and compliance review;
- storage and asset lifecycle;
- cost per accepted asset.
If the primary image route fails because the input is invalid, fallback is the wrong move. Fix the request. If it fails because the provider is temporarily unavailable before any asset was created, fallback may be reasonable when the backup route has already passed visual QA.
Audio: split transcription, understanding, and speech
Audio routing often hides three different jobs:
- Speech-to-text: convert audio to transcript.
- Audio understanding: reason over audio content, speaker intent, or meeting context.
- Text-to-speech: generate spoken output.
Those jobs have different failure modes. A transcription route can be judged by word error rate, domain vocabulary, speaker labels, timestamps, and latency. A speech route is judged by voice quality, pronunciation, latency, allowed voice policy, and playback experience.
Fallback should stay inside the specific audio job type. Do not use "audio-capable" as a single policy label.
Video: treat fallback as another production event
Video is usually the highest-risk modality for automatic fallback because a failed or slow request may already have created a provider-side task. Before creating another task, your system should know:
- whether the original task exists;
- whether it is queued, processing, succeeded, failed, or expired;
- whether an output URL was created;
- whether any usage or cost was recorded;
- whether a reviewer has seen the asset;
- whether a second generation would violate the budget or approval flow.
For video generation, a safe default is: retry transport only before a task exists, poll the existing task until terminal state, then use a manual or narrowly approved regeneration path.
What to log when fallback happens
Fallback that is invisible in logs becomes impossible to improve. At minimum, every request should record:
| Field | Why it matters |
|---|---|
request_id |
Ties user-visible behavior to provider and gateway evidence. |
route_class |
Shows whether the job was text, vision, image, audio, or video. |
primary_model and final_model |
Reveals which route answered or produced the artifact. |
fallback_reason |
Distinguishes timeout, rate limit, upstream error, verification failure, or manual override. |
attempt_count |
Prevents unbounded retry loops. |
latency_ms |
Shows whether fallback helped or harmed user experience. |
usage and cost |
Makes backup route economics visible. |
verification_status |
Connects fallback to output quality, not just API success. |
reviewer_outcome |
Needed for image, audio, and video acceptance tracking. |
For AI product teams, the most useful metric is often not raw uptime. It is accepted output rate after fallback. A backup route that returns quickly but produces rejected assets is not reliable in product terms.
Where Flatkey fits
Flatkey is relevant to fallback routing for LLM APIs because the operational pain is bigger than one API call. Multimodal products often accumulate provider-specific keys, direct billing accounts, scattered model IDs, inconsistent error handling, and separate dashboards for text, image, audio, and video experiments.
Flatkey's public docs and site give teams a single OpenAI-compatible router URL for supported model calls, a model catalog, usage logs, one balance, and one invoice surface. The live site also frames reliability around automatic failover and public status/SLA surfaces. That does not remove the need for your own modality policy. It gives the policy one place to operate instead of spreading route choice across feature code.
Use Flatkey when you want:
- one router endpoint while comparing supported model routes;
- one place to check model availability and commercial context;
- one balance and invoice instead of scattered provider accounts;
- usage logs for model, token, cost, latency, and owner review;
- route governance as more agents and workflows adopt AI APIs.
Use direct provider accounts when one provider is the only requirement, the workload does not need fallback, and the simplest vendor path is more important than cross-model control.
Rollout checklist for fallback routing for LLM APIs
Before you enable automatic fallback in a multimodal product, complete this checklist:
- Inventory modalities. List every workflow that produces text, analyzes images, generates images, accepts audio, speaks, or generates video.
- Assign route classes. Do not let one "multimodal" label cover different artifact types.
- Define verification. Every route class needs a pass/fail method before fallback can be trusted.
- Separate retry from fallback. Retry repeats the same route. Fallback changes route. Regeneration creates a new artifact.
- Write anti-bypass rules. Safety rejections, invalid inputs, side effects, and visible partial outputs should not silently fallback.
- Set cost ceilings. A fallback that saves the request but breaks the budget still failed the product.
- Log the route decision. Store the primary route, final route, failure reason, cost, latency, and verification outcome.
- Test failure injection. Simulate timeouts, rate limits, 5xx errors, invalid inputs, parser failures, rejected images, bad transcripts, and slow video jobs.
- Start with one low-risk workflow. Do not turn on cross-modality fallback across the whole product at once.
- Review accepted-output rate weekly. Optimize for usable results, not just successful HTTP responses.
This is the difference between reliable fallback routing for LLM APIs and blind retries across incompatible routes.
FAQ
What is fallback routing for LLM APIs?
Fallback routing for LLM APIs is the policy that decides when a failed, degraded, or unavailable model route should retry, switch to an approved backup, stop with an actionable error, or move to human review.
Why is multimodal agent routing harder than text-only routing?
Multimodal agent routing is harder because text, image, audio, and video outputs have different verification methods, cost curves, latency expectations, and review risks. One generic fallback rule cannot safely cover all four.
Can one API gateway handle text, image, audio, and video?
Yes, one gateway can centralize access and usage review across supported routes. The application still needs separate route classes and fallback policies for each modality and workflow.
When should fallback be automatic?
Automatic fallback is safest when the output contract is machine-checkable, the backup route has already been tested, no partial output reached the user, no side effects are uncertain, and the backup stays within latency and cost limits.
When should fallback stay manual?
Fallback should stay manual when the artifact requires subjective review, the request may have created side effects, the cost of duplicate work is high, or a policy/safety rejection occurred. Video generation and brand-sensitive image generation often belong in this category.
Is retry the same as fallback?
No. Retry sends the same request to the same route again. Fallback changes the route. Regeneration creates a new artifact. Treat those as separate policy actions because they have different cost and quality implications.
How should teams measure fallback quality?
Measure success rate, fallback rate, latency, incremental cost, parser failures, reviewer acceptance, rejected output rate, and accepted output rate after fallback. HTTP 200 is only the beginning of the measurement.
Build the routing policy before the next model change
The strongest reason to formalize fallback routing for LLM APIs is that model access changes faster than application architecture. New text, image, audio, and video routes will keep launching. Some will disappear, slow down, change price, or behave differently under load.
If your product has route classes, verification rules, cost ceilings, and observable fallback decisions, adding or removing a model is a controlled policy update. If your routing logic is hidden in feature code, every model change becomes a migration risk.
Start with one contained workflow, define its modality and verification class, test approved backups, and then expand. When you are ready to centralize access and billing across supported routes, use the Flatkey API quickstart, review the unified AI API workflow, and compare current model availability on the Flatkey model directory.
