LangGraph Multi-Model Routing: Gateway Checks for Agent Workflows

LangGraph multi-model routing is a graph design problem first and a gateway problem second. The graph decides what kind of work is happening, the model client decides which OpenAI-compatible route to call, and the gateway should prove whether each call stayed inside the budget, tool, retry, logging, and stop-condition rules.

Flatkey fits at that gateway boundary. Its public site shows an OpenAI-compatible chat route at https://router.flatkey.ai/v1/chat/completions, with one key, usage visibility, billing controls, and model routing in one place. This guide shows a practical LangGraph multi-model routing setup for agent workflows without assuming that every model alias, tool behavior, or fallback route has already been validated in your account.

Quick answer for LangGraph multi-model routing

A safe LangGraph multi-model routing setup separates routing into three layers.

Layer	What it decides	Gateway check
Graph route	Whether the workflow should use a simple answer node, complex reasoning node, tool-heavy node, or stop path	Store the route name and reason with the request timestamp.
Model route	Which OpenAI-compatible model alias each node calls through Flatkey	Verify the alias exists for the key and endpoint family.
Operational guardrail	Whether a request should proceed, retry, fall back, or stop	Check budget, tool-call count, retry count, timeout, and request logs.
Evidence	Whether the route actually worked	Match the app response to Flatkey usage or request logs.

Do not hide LangGraph multi-model routing inside one long prompt. Put the policy in graph nodes, conditional edges, and gateway checks that your team can inspect.

Why model routing belongs at graph boundaries

Most LangGraph examples focus on graph control flow: state, nodes, edges, tools, and persistence. That is the right starting point, but production agents also need route boundaries that platform and finance teams can reason about.

Use graph-level routes when the workflow has clearly different cost, latency, or tool-risk profiles:

Work type	Example	Route shape
Simple answer	Summarize a short support note	Fast, low-cost model alias.
Complex reasoning	Reconcile conflicting policy text	Stronger model alias with a higher per-call budget.
Tool-heavy action	Query systems, create tickets, update records	Tool-capable model alias with stricter call limits.
Stop path	Budget exhausted, too many retries, unsafe state	Return a controlled message instead of calling another model.

That boundary is where Flatkey can help. Instead of scattering separate provider keys through each node, each route can call an OpenAI-compatible API base URL while Flatkey keeps gateway-level visibility. For adjacent policy design, see the Flatkey guide to AI agent gateway controls.

Source checks before you wire the route

Use current docs and live account checks before treating templates as production code.

Check	What to verify
LangGraph state and edges	Your graph can route with StateGraph, node functions, and conditional edges.
LangChain model client	Your installed LangChain version supports init_chat_model with an OpenAI provider, base_url, and api_key, or the equivalent ChatOpenAI constructor.
Tool calling	The selected model route supports the tool schema and tool-call behavior your node needs.
Flatkey endpoint	Your key can call the selected model alias through the OpenAI-compatible Flatkey base URL.
Budget and logs	A real request appears in Flatkey usage or request logs with the expected key, model alias, and timestamp.

The code below is a template. Run it with your current langchain, langchain-openai, and langgraph versions before shipping a LangGraph multi-model routing policy.

Configure Flatkey-backed chat models

Install current packages in the environment that runs your graph.

pip install -U langgraph langchain langchain-openai

Keep the API root separate from the endpoint path. LangChain clients should receive the base URL; the client adds the Chat Completions path for OpenAI-compatible chat calls.

FLATKEY_API_KEY=sk-fk-your-key
FLATKEY_BASE_URL=https://router.flatkey.ai/v1
FLATKEY_FAST_MODEL=your-fast-model-alias
FLATKEY_REASONING_MODEL=your-reasoning-model-alias
FLATKEY_TOOL_MODEL=your-tool-capable-model-alias

Then create explicit model clients for each route.

import os
from langchain.chat_models import init_chat_model

BASE_URL = os.getenv("FLATKEY_BASE_URL", "https://router.flatkey.ai/v1")
API_KEY = os.environ["FLATKEY_API_KEY"]

fast_model = init_chat_model(
    model=os.environ["FLATKEY_FAST_MODEL"],
    model_provider="openai",
    base_url=BASE_URL,
    api_key=API_KEY,
)

reasoning_model = init_chat_model(
    model=os.environ["FLATKEY_REASONING_MODEL"],
    model_provider="openai",
    base_url=BASE_URL,
    api_key=API_KEY,
)

tool_model = init_chat_model(
    model=os.environ["FLATKEY_TOOL_MODEL"],
    model_provider="openai",
    base_url=BASE_URL,
    api_key=API_KEY,
)

This is the smallest useful shape for LangGraph multi-model routing: each route has a named model client, but all calls use the same gateway boundary. Do not assume that a model alias supports streaming or tools just because another alias does. Test each route separately.

Build a LangGraph routing template

Use a classifier node to write the route into state. Then let conditional edges decide which node runs next.

from typing import Annotated, Literal, TypedDict

from langchain_core.messages import AIMessage, BaseMessage
from langchain_core.tools import tool
from langgraph.graph import END, StateGraph
from langgraph.graph.message import add_messages


class RouteState(TypedDict):
    messages: Annotated[list[BaseMessage], add_messages]
    route: Literal["simple", "complex", "tools", "stop"]
    budget_cents: float
    errors: int
    tool_calls: int


@tool
def lookup_account_status(account_id: str) -> str:
    """Template tool. Replace with your real read-only lookup."""
    return "active"


tool_ready_model = tool_model.bind_tools([lookup_account_status])


def choose_route(state: RouteState) -> dict:
    last_message = state["messages"][-1].content.lower()

    if state.get("budget_cents", 0) <= 0:
        return {"route": "stop"}
    if state.get("errors", 0) >= 2:
        return {"route": "stop"}
    if "lookup" in last_message or "account" in last_message:
        return {"route": "tools"}
    if len(last_message) > 600:
        return {"route": "complex"}
    return {"route": "simple"}


def simple_answer(state: RouteState) -> dict:
    return {"messages": [fast_model.invoke(state["messages"])]}


def complex_answer(state: RouteState) -> dict:
    return {"messages": [reasoning_model.invoke(state["messages"])]}


def tool_answer(state: RouteState) -> dict:
    if state.get("tool_calls", 0) >= 3:
        return {"messages": [AIMessage(content="Tool-call limit reached.")] }
    return {"messages": [tool_ready_model.invoke(state["messages"])]}


workflow = StateGraph(RouteState)
workflow.add_node("choose_route", choose_route)
workflow.add_node("simple_answer", simple_answer)
workflow.add_node("complex_answer", complex_answer)
workflow.add_node("tool_answer", tool_answer)

workflow.set_entry_point("choose_route")
workflow.add_conditional_edges(
    "choose_route",
    lambda state: state["route"],
    {
        "simple": "simple_answer",
        "complex": "complex_answer",
        "tools": "tool_answer",
        "stop": END,
    },
)
workflow.add_edge("simple_answer", END)
workflow.add_edge("complex_answer", END)
workflow.add_edge("tool_answer", END)

graph = workflow.compile()

This template keeps LangGraph multi-model routing visible. A reviewer can see why a request went to simple_answer, complex_answer, tool_answer, or stop without reverse-engineering a prompt.

The tool_answer node above verifies the model route that can propose tool calls. If your graph should execute tools, add a LangGraph ToolNode or your own executor after that model node, then route the tool result back into the graph with the same budget and stop-condition checks.

Add gateway checks around every model call

The graph route should not be the only guardrail. Add a small wrapper that checks budgets before a model call and records evidence after a model call.

from dataclasses import dataclass
from time import time


@dataclass
class GatewayDecision:
    route: str
    model_alias: str
    allowed: bool
    reason: str
    started_at: float


def preflight(route: str, model_alias: str, state: RouteState) -> GatewayDecision:
    if state.get("budget_cents", 0) <= 0:
        return GatewayDecision(route, model_alias, False, "budget_exhausted", time())
    if route == "tools" and state.get("tool_calls", 0) >= 3:
        return GatewayDecision(route, model_alias, False, "tool_limit_reached", time())
    if state.get("errors", 0) >= 2:
        return GatewayDecision(route, model_alias, False, "retry_limit_reached", time())
    return GatewayDecision(route, model_alias, True, "allowed", time())


def record_gateway_result(decision: GatewayDecision, response: BaseMessage) -> None:
    # Replace this with structured logging. Include a redacted key label,
    # route name, model alias, request timestamp, and response metadata.
    print(
        {
            "route": decision.route,
            "model_alias": decision.model_alias,
            "allowed": decision.allowed,
            "reason": decision.reason,
            "started_at": decision.started_at,
            "response_type": response.type,
        }
    )

In production, do not stop at local logs. Match the timestamp and model alias against Flatkey usage or request logs. That is the proof that LangGraph multi-model routing used the gateway route you intended.

Check tools before you trust a tool-heavy route

Tool-heavy nodes deserve their own verification packet. A model may be good at plain chat but fail your tool schema, stream tool deltas differently, or produce tool arguments that your executor rejects. For deeper compatibility checks, use the Flatkey guide to tool-calling compatibility across providers.

Tool check	Pass condition
Schema accepted	The model route accepts the tool definition without a request error.
Tool selected	The response contains the expected tool call for a deterministic prompt.
Arguments parse	The tool arguments validate against your schema before execution.
Tool count capped	The route stops after your maximum number of tool calls.
Failure handled	Tool errors produce a controlled graph state, not an infinite retry loop.
Log visible	Flatkey logs show the request under the expected key and model alias.

This is also where you decide whether a route belongs on Chat Completions or Responses. Keep endpoint families separate; if you are evaluating Responses-style agent routes, pair this guide with the Flatkey OpenAI-compatible Responses API router.

Retry, fallback, and stop-condition policy

Retries can hide routing mistakes. A strict LangGraph multi-model routing policy should say when to retry, when to fall back, and when to stop.

Condition	Recommended action	Why
401 or 403	Stop and fix the key	Retrying will not repair auth.
404 model not found	Stop and verify model alias or endpoint family	The route may be calling the wrong model.
429 or capacity error	Retry once with backoff or fall back to an approved alias	Avoid runaway cost and user-visible loops.
Tool validation error	Stop the tool route and return a controlled message	Repeating invalid tool arguments is rarely useful.
Budget threshold reached	Stop before the next model call	The gateway policy should protect spend before the call.

LangChain middleware can help with agent-level retries, fallback, and call limits. For hand-built LangGraph workflows, keep the same policy near the node that calls the model, then prove the outcome in Flatkey logs.

Pre-publish test checklist

Run these checks before making Flatkey the default route for an agent.

1. Send one direct curl request to https://router.flatkey.ai/v1/chat/completions with the intended key and model alias.
2. Invoke each LangChain model client once outside LangGraph.
3. Run the graph with one prompt for the simple route, one for the complex route, and one for the tool-heavy route.
4. Force the stop path by setting budget_cents to 0.
5. Force the retry limit by setting errors to the maximum allowed value.
6. Confirm each request appears in Flatkey logs with the expected timestamp, key label, route name, and model alias.
7. Compare cost or usage fields with the owner who approves the workflow budget. The Flatkey pricing page is the right public place to start that review.

The output of this checklist becomes the acceptance packet for LangGraph multi-model routing. It is more useful than a screenshot of a working prompt because it proves the graph, model client, and gateway all agree.

Troubleshooting LangGraph multi-model routing

Symptom	Likely cause	What to check
Every request uses the same model	Conditional edge never changes state	Print the route value after the classifier node.
401 or 403	Wrong, missing, or revoked Flatkey key	Recreate the key and keep it out of source control.
404	Wrong API root, duplicate /v1, or unavailable model alias	Use https://router.flatkey.ai/v1 as the base URL and verify the alias.
Tool route fails	Model alias does not support your tool schema	Test bind_tools with one deterministic prompt before graph execution.
Retries multiply cost	Retry policy lives in too many places	Choose one retry owner: graph node, LangChain middleware, or gateway policy.
Logs do not match	Request bypassed Flatkey or logs are filtered	Search by timestamp, key label, model alias, and route.
Finance cannot approve rollout	Cost owner cannot map routes to spend	Label keys by workload and keep model aliases explicit in route config.

If the first route fails, keep the debugging surface small: key, base URL, model alias, one model client, one graph node, and one Flatkey log lookup.

FAQ

Is LangGraph multi-model routing the same as provider failover?

No. LangGraph multi-model routing decides which route fits the task before or during graph execution. Provider failover is what happens when a selected route cannot complete. You can use both, but they should be tested separately.

Should every LangGraph node use a different model?

No. Start with two or three explicit routes. Too many model aliases make logs, budgets, and rollback harder to review.

Can I use Flatkey with both Chat Completions and Responses routes?

Flatkey exposes OpenAI-compatible endpoint families, but each client path still needs its own test. Keep Chat Completions and Responses checks separate, especially for tool calls, streaming, usage fields, and fallback behavior.

Final checklist

Your LangGraph multi-model routing setup is ready to move beyond a template when:

1. The graph route is visible in state and logs.
2. Each route maps to a named model alias.
3. The Flatkey key can call each selected alias through the OpenAI-compatible base URL.
4. Tool-heavy routes pass schema, argument, cap, and error tests.
5. Budget and retry stop conditions run before another model call.
6. Flatkey usage or request logs confirm the selected route.
7. Rollback is one config change, not an application rewrite.

When you are ready to route LangGraph agent traffic through one gateway, get a Flatkey key, start with a single low-risk graph route, and expand only after the logs prove the policy works.