OpenAI client migration: preserve RPM, TPM, and retry behavior

An OpenAI client migration can look complete after two configuration changes: replace the API key and point the SDK at a new base URL. The first request succeeds, the response shape looks familiar, and the pull request appears ready to merge.

That proves interface compatibility. It does not prove production behavior.

The harder part of an OpenAI client migration is preserving what happens when traffic becomes uneven: requests arrive in bursts, prompts grow, streams run longer than expected, a provider returns 429, or a response times out after work may already have started. If the SDK, your application, and a job queue all retry independently, one failed call can become several near-simultaneous attempts.

This tutorial shows how to migrate an existing OpenAI-style Python or TypeScript integration to a unified gateway while making rate-limit and retry behavior explicit. The examples use Flatkey's OpenAI-compatible base URL, but the review method applies to any gateway migration.

Quick answer: what should change?

For a safe OpenAI client migration, review these settings together rather than treating the base URL as the whole change.

Migration surface	What to review	Safe starting decision
API endpoint	Base URL and authentication	Change through environment variables, not scattered literals
Model selection	Exact model identifiers and supported parameters	Pin one known model for the canary
SDK retries	Automatic retry count and retryable status codes	Choose whether the SDK or your application owns retries
Application retries	Backoff, jitter, attempt cap, and retry budget	Keep one retry owner and log every attempt
RPM control	Request arrival rate and burst size	Add a concurrency or queue limit before the cutover
TPM control	Prompt plus expected output tokens	Test realistic large prompts, not only a one-line smoke test
Timeouts	Connect, read, and total request duration	Set explicit values for synchronous and streaming calls
Observability	Request IDs, attempts, tokens, latency, and final outcome	Compare client logs with gateway usage logs

If you need the acronym-level explanation first, read LLM rate limits explained: RPM, TPM, and retries. This guide starts where that explainer ends: at the migration diff and the production test plan.

Why a base-URL swap is necessary but insufficient

Flatkey's quickstart documents the minimal client change: keep the OpenAI SDK request pattern and set the base URL to https://router.flatkey.ai/v1. It also recommends checking Usage Logs after the request so you can verify the model, token counts, latency, and cost.

That is the correct smoke test. A production OpenAI client migration needs four additional questions:

1. Does the SDK retry 429, timeout, or server errors automatically?
2. Does another layer also retry the same failed operation?
3. Is concurrency limited by request rate, token rate, or both?
4. Can you distinguish one logical operation from its individual attempts?

The official OpenAI Python and Node SDK documentation currently says selected failures are retried twice by default, including 429 responses, connection errors, timeouts, and some server errors. Both SDKs expose retry and timeout configuration. That default is convenient for a direct integration, but it can become invisible amplification when your own code already implements backoff.

The migration goal is not “disable every retry.” The goal is “know which layer owns the retry.”

Step 1: inventory every retry layer before changing code

Start by drawing the actual call path.

user action or job
  -> application retry wrapper
  -> queue delivery retry
  -> OpenAI SDK retry
  -> gateway
  -> provider

For each layer, record:

• Which errors trigger another attempt.
• The maximum number of attempts.
• Whether delay uses fixed sleep, exponential backoff, or jitter.
• Whether a server-provided Retry-After value is honored.
• Whether the same operation identifier is preserved across attempts.
• Whether a timed-out request is assumed to have failed before any work occurred.

The last assumption is risky. A client timeout only tells you the client stopped waiting. The upstream system may still have accepted or completed the request. For generated content, a retry can therefore create another result and another billable request even when your application observed only one logical task.

Estimate worst-case amplification

Suppose a queue can deliver a job three times, the application wrapper allows three attempts, and the SDK performs the initial call plus two retries. In the worst case, one logical job can trigger:

3 queue deliveries × 3 application attempts × 3 SDK attempts = 27 HTTP attempts

You may never hit the full number, but the multiplication explains why a brief 429 can turn into a retry storm. Write the number into the migration review. It makes hidden defaults visible.

Step 2: move endpoint settings into configuration

Keep the migration diff reversible. Do not replace endpoint strings throughout the codebase.

Python before and after

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.openai.com/v1"),
    max_retries=0,
    timeout=45.0,
)

For the Flatkey canary, configure:

export LLM_API_KEY="$FLATKEY_API_KEY"
export LLM_BASE_URL="https://router.flatkey.ai/v1"

TypeScript before and after

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: process.env.LLM_BASE_URL ?? "https://api.openai.com/v1",
  maxRetries: 0,
  timeout: 45_000,
});

These examples set SDK retries to zero because the next section gives the application explicit retry ownership. If your application has no retry layer, you can instead keep bounded SDK retries. Do not keep both by accident.

For the broader compatibility checklist, see OpenAI Compatible API Gateway: Migration Checklist for Minimal Code Changes.

Step 3: give one layer explicit retry ownership

A useful retry policy has five parts:

1. A short list of retryable failures.
2. A strict attempt cap.
3. A maximum total retry time.
4. Exponential backoff with jitter.
5. Structured logs for every attempt.

Here is a small Python wrapper for a synchronous chat call:

import random
import time
from openai import APITimeoutError, APIConnectionError, APIStatusError, RateLimitError

RETRYABLE_STATUS_CODES = {408, 409, 429, 500, 502, 503, 504}


def create_chat_with_retry(client, *, model, messages, max_attempts=4):
    started_at = time.monotonic()

    for attempt in range(1, max_attempts + 1):
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
            )
        except (RateLimitError, APITimeoutError, APIConnectionError) as error:
            retryable = True
            caught_error = error
        except APIStatusError as error:
            retryable = error.status_code in RETRYABLE_STATUS_CODES
            caught_error = error

        if not retryable or attempt == max_attempts:
            raise caught_error

        exponential_delay = min(2 ** (attempt - 1), 16)
        jitter = random.uniform(0, 0.5 * exponential_delay)
        sleep_seconds = exponential_delay + jitter

        print({
            "event": "llm_retry",
            "attempt": attempt,
            "next_delay_seconds": round(sleep_seconds, 2),
            "elapsed_seconds": round(time.monotonic() - started_at, 2),
            "error_type": type(caught_error).__name__,
        })
        time.sleep(sleep_seconds)

    raise RuntimeError("unreachable")

Treat this as a reviewable starting point, not a universal policy. In production, parse and honor a valid Retry-After response header before falling back to locally calculated delay. Add a total elapsed-time budget so retries cannot exceed the latency your product can tolerate.

Do not retry every error

Failure	Default action	Why
400 invalid request	Do not retry unchanged	The payload must change
401 authentication	Do not retry unchanged	The key or header must change
404 model not found	Do not retry unchanged	The model identifier or access must change
429 rate limit	Retry with delay and jitter	Capacity may become available
500 or 503	Retry within a small budget	The failure may be temporary
Client timeout	Retry cautiously	The upstream request may already have executed

Flatkey's quickstart gives the same high-level guidance for 429: retry with exponential backoff and jitter. The migration-specific addition is to make sure only one layer performs that policy.

Step 4: size concurrency for both RPM and TPM

An OpenAI client migration can preserve request syntax while changing the capacity envelope. RPM and TPM constrain different workloads:

• RPM becomes the bottleneck when you send many small requests.
• TPM becomes the bottleneck when prompts, outputs, or parallel evaluations are large.

Use observed traffic rather than a single average. Collect at least:

• Requests per minute at median and peak.
• Input tokens at p50, p95, and maximum.
• Output tokens at p50 and p95.
• Median and p95 request duration.
• Number of concurrent streams.

A rough concurrency ceiling can be estimated from each limit:

RPM-based concurrency ≈ (RPM / 60) × average request seconds

TPM-based concurrency ≈ (TPM / average tokens per request / 60)
                        × average request seconds

Use the lower result as the initial ceiling, then leave headroom for bursts and retries.

Example: assume a route allows 600 RPM and 300,000 TPM, the average request uses 1,500 total tokens, and average duration is 3 seconds.

RPM ceiling: (600 / 60) × 3 = 30 concurrent requests
TPM ceiling: (300,000 / 1,500 / 60) × 3 = 10 concurrent requests

TPM is the tighter constraint in this example. Starting at 30 concurrent requests because RPM looked generous would create avoidable 429 responses.

This calculation is directional, not a provider guarantee. Providers may use rolling windows, token buckets, separate input and output token limits, model-specific pools, or acceleration controls. The test plan must verify real behavior for the selected model and account.

Step 5: test streaming and timeout behavior separately

Do not treat a successful non-streaming call as proof that streaming is safe.

For streaming requests, test:

• Time to first token.
• Maximum quiet interval between chunks.
• Client read timeout.
• Behavior when the consumer disconnects.
• Whether your retry wrapper can accidentally start a second stream.
• Whether partial output is retained, discarded, or shown to the user.

A stream that fails after partial output is not equivalent to a request that failed before any output. Retrying it automatically may show duplicated text or produce a different continuation. Decide whether the product should retry, ask the user, or surface the partial result.

Also remember that SDK timeouts and infrastructure timeouts can differ. A reverse proxy, serverless platform, job worker, or browser connection may terminate before the client library reaches its own timeout. During the OpenAI client migration, record the smallest timeout in the complete request path.

Step 6: run a canary matrix before broad traffic

Use one pinned model and a small percentage of traffic. The first canary should answer whether the new route preserves behavior, not whether every model works.

Test case	Input	Expected evidence
Authentication	Valid and invalid keys	Success plus a non-retried 401
Model validation	Valid and misspelled model IDs	Success plus a non-retried model error
Small request burst	Many short prompts	Controlled queueing without a retry spike
Large prompt burst	Fewer high-token prompts	TPM pressure is visible and bounded
Forced 429	Temporarily exceed the canary limit	One retry owner, jittered delays, capped attempts
Forced timeout	Set an intentionally short client timeout	Logged timeout with no unbounded replay
Streaming interruption	Disconnect during a stream	Explicit partial-output behavior
Server error	Inject or simulate 503	Bounded retries and final error reporting
Rollback	Restore the previous base URL	Configuration-only rollback succeeds

For every logical operation, log:

operation_id
attempt_number
base_url_name
model_requested
http_status
input_tokens
output_tokens
latency_ms
retry_delay_ms
final_outcome

Then compare application logs with Flatkey Usage Logs. The counts should make sense together. If one application operation maps to several gateway requests, your retry instrumentation should explain why.

Step 7: define rollout and rollback thresholds

An OpenAI client migration should have numerical stop conditions before the first canary starts.

Example thresholds:

• Roll back if the final error rate rises by more than an agreed percentage point.
• Pause if attempts per operation exceed the expected retry budget.
• Pause if p95 latency exceeds the product timeout budget.
• Pause if token usage per successful operation changes unexpectedly.
• Expand traffic only after streaming and non-streaming paths both pass.

Avoid comparing only raw 429 counts. A good queue may reduce final errors while temporarily increasing delayed requests. Track both attempt-level and operation-level outcomes.

Migration pull-request checklist

Copy this checklist into the implementation PR.

• Base URL and key come from environment variables.
• The canary uses an exact, verified model identifier.
• One layer owns retries.
• SDK retry defaults are documented in the PR.
• 429, timeout, and 5xx behavior have bounded attempts.
• Backoff includes jitter and honors Retry-After when present.
• RPM and TPM ceilings are estimated from observed traffic.
• Streaming has a separate failure test.
• Each attempt shares one logical operation_id.
• Usage logs and application logs are compared.
• Rollout and rollback thresholds are written before launch.
• The previous endpoint can be restored without another code change.

Common migration mistakes

Keeping SDK retries and application retries without calculating the total

This is the most important review finding. Defaults are still behavior, even when they are not visible in the local function.

Testing only a tiny prompt

A one-line request proves credentials and response compatibility. It says almost nothing about TPM pressure, output limits, long streams, or p95 latency.

Retrying authentication and validation errors

Backoff cannot repair an invalid key, unsupported parameter, or misspelled model. Retrying unchanged payloads wastes capacity and hides the real defect.

Treating a timeout as proof that no request ran

The client may stop waiting after the upstream accepted the call. Design retries and accounting with that ambiguity in mind.

Changing endpoint, models, prompts, and retry policy in one release

That makes failures hard to attribute. Migrate one known request shape first, then expand model choice after the route is observable.

A safer definition of “OpenAI-compatible”

For migration planning, “OpenAI-compatible” should mean the interaction pattern is familiar enough to reduce code changes. It should not be interpreted as a promise that every provider shares identical quotas, token counting, error semantics, latency, streaming behavior, or parameter support.

That distinction makes an OpenAI client migration easier to review. Keep the stable interface where it helps, but test the operational contract where providers and routes can differ.

Flatkey centralizes access and billing behind one OpenAI-compatible base URL, which can simplify the client diff and later model expansion. The engineering work is still to make retries, throughput, and observability explicit before production traffic moves.

That is the standard a production OpenAI client migration should meet: a small interface change backed by explicit operational evidence.

Review the Flatkey pricing page when selecting the models for your canary, then approve the migration only after the checklist passes in code review and the route behavior is visible in logs.

Frequently asked questions

Should I disable OpenAI SDK retries during migration?

Disable them if your application or queue already owns retries. If no other layer retries, bounded SDK retries can be reasonable. The important rule is to avoid multiple independent retry owners.

What is the difference between RPM and TPM during a migration?

RPM limits request frequency, while TPM limits token throughput. Small high-frequency calls can hit RPM first; fewer large prompts or outputs can hit TPM first. Test both workload shapes.

Should a 429 always be retried?

Only within a capped retry and latency budget. Honor Retry-After when available, otherwise use exponential backoff with jitter. Stop if the operation can no longer meet the product's latency objective.

Can I retry a timed-out generation safely?

Not with certainty. The upstream request may have executed even though the client timed out. Treat the retry as a possible duplicate request and log the relationship between attempts.

What is the minimum safe canary?

Use one pinned model, one request shape, explicit retry ownership, a concurrency cap, and tests for 429, timeout, streaming interruption, and rollback. Compare client-side attempts with gateway usage logs before expanding traffic.

Sources

• Flatkey quickstart
• OpenAI Python SDK: retries, timeouts, and base URL configuration
• OpenAI Node SDK: retries and timeouts
• OpenAI Cookbook: how to handle rate limits