AI Gateway ArchitectureJuly 30, 2026Flatkey Team

AI Model Evaluation Before Switching API Providers: A Workflow Checklist

Use a decision-grade AI model evaluation workflow to compare providers on task success, compatibility, reliability, latency, effective cost, and rollout risk.

Switching AI API providers is not a model leaderboard decision. It is a production change that can alter output quality, JSON validity, tool calls, latency, rate-limit behavior, error handling, and total cost at the same time.

The safest approach is to turn the proposed switch into a repeatable acceptance test. Run the incumbent and candidate on the same production-shaped cases, score complete workflow outcomes, define hard migration gates before seeing results, and expose the candidate gradually behind a rollback-ready route.

This guide gives you that process, including a scorecard, dataset design, compatibility matrix, paired test method, cost formula, rollout stages, and decision memo.

The short answer: use a provider-switch acceptance test

Before moving production traffic, require the candidate route to pass five checks:

Workflow quality: It completes the user task at an acceptable rate on representative inputs.
Contract compatibility: Structured outputs, tool calls, streaming, errors, and finish states work with your application.
Operational reliability: Latency, timeouts, rate limits, retries, and concurrency remain within your service objectives.
Economic value: The effective cost per accepted task improves or stays inside an approved tradeoff.
Safe rollout: Shadow traffic and a staged canary show that offline results survive production conditions.

Do not approve a switch because the candidate wins a public benchmark, produces a few impressive answers, or has a lower advertised token price. Those signals can help create a shortlist. They do not prove that the provider can run your workflow.

Start with the migration decision, not the model list

Write a one-sentence decision before building the evaluation:

Replace route A with route B for workflow X if B is non-inferior on task success, passes every contract gate, meets the production latency and reliability budget, and lowers effective cost per accepted task by the required amount.

That sentence forces the team to define scope. A provider may be suitable for extraction but not agentic tool use, or for batch enrichment but not an interactive assistant. Avoid a universal "best model" conclusion when the real decision concerns one route, one workload, and one operating envelope.

Record these inputs in the evaluation plan:

Field	What to specify
Workflow	The exact feature, automation, or agent route under consideration
Incumbent	Current provider, model, version or alias, region, and settings
Candidate	Proposed provider, model, version or alias, region, and settings
Traffic shape	Requests per minute, tokens per minute, concurrency, prompt sizes, and output sizes
Required capabilities	JSON Schema, tools, streaming, images, long context, caching, or other dependencies
Hard gates	Conditions that automatically block migration
Tradeoff limits	Maximum acceptable regression for quality, latency, reliability, or cost
Rollback owner	Person or team authorized to stop the rollout

If the integration change itself is still uncertain, review the OpenAI-compatible API gateway migration checklist before testing models. A shared interface reduces code changes, but it does not make model behavior identical.

Define the evaluation unit as a complete trace

The evaluation unit should match the thing your customer experiences. For a single-turn classifier, that may be one request and response. For an agent, it may be an entire trace containing several model calls, tool invocations, retries, and a final answer.

A useful trace record includes:

{
  "case_id": "support-refund-042",
  "segment": "refund-policy",
  "input": {},
  "expected_contract": {},
  "route": "candidate-b",
  "attempts": 1,
  "latency_ms": 1840,
  "input_tokens": 3120,
  "output_tokens": 486,
  "provider_cost_usd": 0.0124,
  "schema_valid": true,
  "tool_sequence_valid": true,
  "task_success": true,
  "failure_class": null
}

This prevents a common measurement error: scoring only the final prose while ignoring malformed arguments, repeated tools, hidden retries, or a latency spike that made the workflow unusable.

Build a production-shaped test set

The evaluation set should represent the distribution and failure modes of the route you plan to migrate. Randomly selecting a small pile of "typical" prompts usually hides the cases that cause incidents.

Use six case groups:

Frequent cases: The inputs responsible for most normal traffic.
High-value cases: Tasks where a wrong answer causes a costly human correction or lost conversion.
Long-tail cases: Rare languages, formats, domains, or user intents.
Contract cases: Inputs that stress schemas, enums, nested objects, tools, and streaming assembly.
Adversarial cases: Ambiguous instructions, conflicting evidence, prompt injection, and unsupported requests.
Operational cases: Large contexts, long outputs, concurrent bursts, timeouts, and provider-side errors.

Stratify the dataset so every important segment has enough examples to inspect separately. A candidate can look acceptable in the aggregate while failing one language, one tool, or one customer tier.

Keep three dataset layers:

Development set: Visible cases used to improve prompts and validators.
Decision set: Held-out cases used to approve or reject the migration.
Production audit set: Fresh cases sampled after rollout to detect drift.

Do not repeatedly tune against the decision set. Once the team has seen its failures and changed the system, those cases have effectively become development data.

Freeze the test conditions

Paired comparisons are only useful when the incumbent and candidate receive equivalent work. Freeze or record:

System and developer instructions.
User input and attachments.
Tool definitions and JSON Schemas.
Temperature, maximum output, seed where supported, and reasoning settings.
Retrieval results and document ordering.
Region, API version, model identifier, and provider route.
Retry policy, timeout, and concurrency.
Evaluation timestamp and pricing source.

If one route uses a different prompt because the provider requires it, version both prompts and treat that difference as part of the migration package. The business decision is about the new system, not an abstract model isolated from its integration.

Run every case through both routes. Randomize or blind presentation when humans score subjective outputs, so reviewers are not influenced by provider names.

Set hard gates before weighted scores

A weighted score is useful for tradeoffs, but it should not allow a cheap model to compensate for a critical contract failure.

Define non-negotiable gates first. Example gates might include:

No unauthorized tool execution.
No secrets or restricted data in outputs.
Required JSON parses and validates against the production schema.
Required languages stay above their minimum task-success threshold.
Timeout and server-error rates remain inside the approved budget.
The client correctly handles streaming termination and provider errors.
A rollback route is tested before production exposure.

Thresholds must come from your product risk and current baseline. The numbers below are an illustrative scorecard, not universal recommendations:

Dimension	Weight	Example measure	Example migration rule
Task success	35%	Accepted outcomes / total traces	Candidate is no worse than baseline beyond the approved margin
Contract compliance	20%	Valid schema, tools, and stream completion	Pass all hard gates
Reliability	15%	Successful traces after bounded retries	Stay inside service budget
Latency	10%	p50, p95, and p99 end-to-end trace time	p95 stays below route objective
Effective cost	15%	Total route cost / accepted outcomes	Meet savings or value target
Operability	5%	Observability, debugging, quotas, and support	No unresolved launch blocker

Publish the weights and gates before the final run. Changing them after results arrive turns evaluation into justification.

Score objective checks before using model judges

Use deterministic validators wherever possible:

JSON parsing and JSON Schema validation.
Exact or normalized field matching.
Numeric tolerance checks.
Citation and URL validation.
Allowed-tool and argument validation.
Tool sequence and maximum-step checks.
Code compilation, unit tests, and sandboxed execution.
Policy rules and prohibited-content detectors.
Retrieval evidence coverage.

Use human review or a model-based grader for criteria that cannot be reduced to a deterministic check, such as clarity, tone, synthesis, or whether an answer follows nuanced instructions.

When using a model grader:

Give it a narrow rubric with observable pass conditions.
Calibrate it against a human-scored sample.
Hide provider identity where possible.
Preserve grader prompts, model version, and raw rationale.
Send disagreements and borderline cases to human review.

OpenAI's evaluation guidance recommends task-specific evals and continuous evaluation, while Anthropic similarly recommends defining observable success criteria and building evaluations around them. The practical implication is simple: your rubric should describe the workflow outcome you need, not generic intelligence.

Compare paired outcomes and uncertainty

An average score alone can hide instability. Because both routes process the same cases, compare them case by case.

For binary task success, create a paired table:

Outcome	Meaning
Both pass	The switch does not change this case
Incumbent passes, candidate fails	Candidate regression
Incumbent fails, candidate passes	Candidate improvement
Both fail	Shared product or evaluation gap

The two disagreement groups are especially useful. Review them manually and classify the root cause before approving the switch.

For a decision-grade result, report a confidence interval around the difference in task success, cost, and latency. A bootstrap over case IDs is a practical method because it can preserve the paired structure without assuming that every metric follows a normal distribution.

Use a non-inferiority rule when the candidate offers a clear benefit, such as lower cost or better regional availability, and the product can tolerate a small bounded quality difference. Define the permitted margin before the test. Approve only when the confidence interval does not cross the unacceptable-regression boundary.

Also inspect results by segment. An overall pass should not hide a failed contract case, language, tool, or high-value workflow.

Use a trace-level failure taxonomy

Every failed case should receive one primary failure class. A consistent taxonomy turns the evaluation into engineering work instead of a debate over anecdotes.

Failure class	Example
`quality`	Answer is incorrect, incomplete, or unsupported
`schema`	Output is not valid JSON or violates the schema
`tool_selection`	Wrong tool chosen or required tool omitted
`tool_arguments`	Tool arguments are missing, malformed, or unsafe
`looping`	Agent repeats actions or exceeds the step budget
`streaming`	Partial output cannot be assembled or finish state is wrong
`rate_limit`	Request fails after the approved queue and retry policy
`timeout`	End-to-end trace exceeds the route timeout
`provider_error`	Upstream 5xx or unavailable route
`client_compatibility`	SDK, parameter, or error-shape mismatch
`policy`	Output or action violates a required policy

Track both the first failure and the final trace outcome. A retry that recovers a request still consumes time and money, and repeated recovery can become a production capacity problem. The LLM rate limits guide explains how to separate RPM, TPM, queueing, retries, and fallback behavior.

Test provider compatibility as a matrix

An OpenAI-compatible endpoint can reduce migration work, but compatibility is not binary. Test the exact features your application uses.

Surface	What to verify
Model names	Stable identifiers, aliases, version pinning, and retirement behavior
Request parameters	Accepted fields, ignored fields, defaults, and validation errors
Structured output	Supported schema subset, refusal shape, truncation, and invalid-output handling
Tool calling	Tool-choice behavior, parallel calls, argument encoding, and call IDs
Streaming	Event format, usage fields, tool deltas, finish reasons, and disconnect recovery
Multimodal input	File types, size limits, URL handling, and token accounting
Errors	HTTP status, provider codes, retry hints, and request IDs
Usage	Input, output, cached, reasoning, image, audio, or video units where applicable
Limits	RPM, TPM, concurrency, daily quotas, burst rules, and tier changes
Data controls	Retention, training policy, regional processing, and logging options

Google's structured-output documentation, for example, notes that schema support is based on a subset of JSON Schema. That is why you should test your actual schema rather than assuming that a schema accepted by one provider will behave identically everywhere.

Calculate effective cost per accepted task

Token price is only one component of migration economics. Measure the full cost of obtaining a usable outcome:

effective cost per accepted task =
  (model usage
   + retries
   + fallback usage
   + tool and retrieval costs
   + evaluation or moderation calls
   + incremental infrastructure cost)
  / accepted tasks

Also estimate the human-review cost created by low-confidence or malformed results. A candidate with cheaper tokens can be more expensive if it increases retries, tool loops, review queues, or rejected outputs.

For current route prices, use the AI API pricing comparison as a starting point, then confirm the exact model and price at decision time. Save the pricing timestamp with your results because prices and model availability can change.

Report cost by segment as well as overall. Long-context cases, multilingual tasks, image inputs, and agent traces may produce a different winner from short text requests.

Load-test the candidate under the real retry policy

Offline quality tests usually run slowly and sequentially. Production does not.

Repeat a representative subset at expected and peak concurrency. Measure:

End-to-end trace latency at p50, p95, and p99.
Time to first token and time to complete where streaming matters.
Queue time versus provider time.
Rate-limit responses and Retry-After behavior.
Timeouts, connection failures, and upstream 5xx errors.
Retry count and retry recovery rate.
Duplicate side effects caused by retried tool actions.
Fallback frequency and final successful route.

Use the same bounded retry behavior planned for production. Unlimited retries can make the success rate look good while violating latency and cost limits. If the candidate requires materially different retry or queue settings, include that operational change in the migration decision.

Run shadow traffic before a canary

Shadow testing sends a copy of eligible production inputs to the candidate while the incumbent still serves the user. It reveals realistic prompt distributions and provider behavior without allowing candidate output to affect the customer.

Protect the shadow path:

Exclude sensitive traffic unless the candidate's data controls are approved.
Disable side-effecting tools or route them to mocks.
Redact or tokenize restricted fields where required.
Cap shadow volume and cost.
Keep incumbent and candidate trace IDs linked for paired analysis.
Do not let shadow retries consume the primary route's capacity budget.

Shadow results should be evaluated with the same validators and failure taxonomy as the offline decision set.

Canary the provider switch in stages

After offline and shadow gates pass, expose a small, observable slice of traffic. A practical sequence is:

Internal and synthetic traffic.
Low-risk users or workflows.
A small percentage of eligible production traffic.
Gradual increases with a fixed observation window at each stage.
Full rollout only after the candidate remains inside every guardrail.

Define automatic rollback triggers before starting. Examples include a task-success regression, schema-failure spike, p95 latency breach, elevated fallback rate, cost overrun, or a critical policy failure.

Use stable routing so the same conversation, agent run, or customer stays on one route when switching mid-session would corrupt state. Keep the incumbent warm until the rollback window closes.

For multi-provider reliability patterns, see the LLM API fallback routing playbook.

Keep routing policy after the evaluation

The result does not need to be "move everything." Many teams get a better outcome by routing deliberately:

A quality-focused route for complex or high-value tasks.
A low-cost route for bounded extraction and classification.
A low-latency route for interactive suggestions.
A regional route for data-location or availability requirements.
A fallback route for rate limits and outages.

This makes the evaluation reusable. Each route has a contract, a dataset, and an operating budget. New candidates compete for a defined job instead of becoming another platform-wide migration project.

Flatkey provides one OpenAI-compatible access layer across multiple model providers, which can simplify side-by-side testing and routing. It does not remove the need for evaluation; it reduces the integration work required to run the evaluation and retain a rollback option. Compare current routes on Flatkey pricing.

Provider-switch decision memo template

End the evaluation with a short, signed decision record:

Section	Required evidence
Decision	Approve, reject, or approve for limited routes
Scope	Workflow, users, regions, and traffic included
Baseline	Incumbent model, prompt, settings, and measurement window
Candidate	Provider, model, prompt, settings, and measurement window
Hard gates	Pass/fail result for every gate
Quality	Paired task-success difference and confidence interval
Compatibility	Schema, tools, streaming, errors, usage, and limits
Operations	Latency, reliability, retries, queueing, and fallback
Economics	Effective cost per accepted task and projected volume
Exceptions	Segments excluded or routed differently
Rollout	Shadow and canary stages, owners, and observation windows
Rollback	Trigger, route, owner, and maximum recovery time
Recheck date	When price, model version, or workload changes require reevaluation

Attach the case-level results, runner version, prompts, validators, raw responses, and pricing snapshot. A future reviewer should be able to reproduce why the switch was approved.

Final AI model evaluation checklist

Before switching AI API providers, confirm that you have:

Defined the exact workflow and candidate route.
Recorded the incumbent baseline and operating envelope.
Built development, held-out decision, and production audit sets.
Included common, high-value, long-tail, adversarial, contract, and operational cases.
Frozen prompts, tools, settings, retrieval inputs, and retry policy.
Run paired incumbent-versus-candidate tests.
Applied deterministic validators before subjective grading.
Calibrated any model-based grader against humans.
Defined hard gates, weights, and non-inferiority margins in advance.
Reported confidence intervals and segment-level outcomes.
Classified trace failures consistently.
Tested schemas, tools, streaming, errors, usage, and rate limits.
Calculated effective cost per accepted task.
Load-tested expected and peak concurrency.
Completed privacy, retention, regional, and access-control review.
Passed shadow traffic and staged canary checks.
Tested automatic rollback and kept the incumbent available.
Signed and stored the provider-switch decision memo.

Frequently asked questions

How many test cases are enough for an AI model evaluation?

There is no universal count. Use enough cases to cover every material segment and narrow uncertainty around the migration decision. High-risk or low-frequency segments may need deliberate oversampling. Report confidence intervals instead of treating a sample size as proof by itself.

Should I use public benchmarks to choose an API provider?

Use benchmarks to create a shortlist or understand broad capabilities. Do not use them as the migration gate. Your prompts, tools, schemas, latency budget, retry policy, data controls, and traffic mix determine production fit.

Can one OpenAI-compatible base URL make providers interchangeable?

It can centralize authentication and reduce client changes. It cannot guarantee identical parameter support, structured outputs, tool behavior, streaming events, limits, or model quality. Test the compatibility matrix for every route you use.

What is the most important provider-switch metric?

For most automation systems, start with successful end-to-end workflow completion. Then explain that outcome using quality, contract validity, latency, reliability, retries, fallback, and effective cost.

When should we rerun the evaluation?

Rerun it when the model version, provider route, prompt, tools, retrieval system, pricing, workload distribution, risk policy, or traffic envelope changes materially. Keep a smaller continuous audit running after launch so regressions appear before the next planned migration.

Make every provider switch reproducible

The durable asset is not the winning model. It is the evaluation system: versioned cases, trace capture, validators, graders, scorecards, load tests, rollout controls, and a decision record.

With that system in place, a new provider becomes a bounded experiment rather than a risky rewrite. You can measure the candidate against the same production contract, expose it gradually, and reverse the change quickly if reality disagrees with the lab.