Prompt caching can reduce the cost and latency of repeated LLM requests, but only when the workflow produces stable prefixes, enough reuse, and acceptable cache-hit behavior. Turning the feature on is not the same as proving return on investment.
This guide gives engineering and FinOps teams a practical prompt caching workflow: find eligible traffic, shape prompts for reuse, instrument cache metrics, calculate net savings, and roll out without hiding quality or reliability regressions.
What is prompt caching?
Prompt caching lets an LLM provider reuse computation for prompt content it has recently processed. Instead of charging and processing every repeated input token at the normal rate, the provider can apply a lower cached-input rate or a separate cache-read price to the reusable portion.
The reusable content is usually a stable prompt prefix. Common examples include:
- A long system prompt and policy block.
- Tool definitions shared by every agent turn.
- A large document, repository map, or product catalog queried repeatedly.
- Few-shot examples reused across a classification or extraction job.
- A conversation history shared by several possible next actions.
Provider implementations differ. OpenAI documents automatic caching for qualifying prompt prefixes and exposes cached-token details in API usage. Anthropic supports explicit cache breakpoints and multiple time-to-live options. Google Gemini supports explicit context caches with storage charges, while DeepSeek documents automatic disk-based context caching with separate cache-hit and cache-miss input rates. Always confirm current model support and pricing in the provider's official documentation before committing savings to a forecast.
The ROI mistake: measuring the discount instead of the workflow
A cached-token discount is not the same as net savings. The workflow may also create cache-write charges, storage charges, extra requests, operational complexity, or quality regressions when teams optimize prompt structure too aggressively.
Measure the unit that matters:
Net prompt caching ROI = avoided uncached input cost - cache write/storage cost - implementation and operating cost
For production decisions, connect that result to an accepted outcome:
Cost per accepted task = total request cost / validated successful tasks
This prevents a misleading result where token spend falls but retries, rejected outputs, or human review increase. It also aligns prompt caching with a broader AI API cost optimization program instead of treating caching as an isolated billing trick.
A six-step prompt caching workflow
1. Find workloads with real prefix reuse
Start with request traces, not intuition. Group traffic by workflow and estimate how many input tokens remain identical from the beginning of one request to the next.
Good candidates usually have four properties:
- Large repeated input: the reusable prefix is material relative to the dynamic suffix.
- Frequent reuse: several requests reference the same prefix within the provider's effective cache lifetime.
- Stable ordering: system instructions, tools, examples, and reference material appear in the same order.
- Low cardinality: the application reuses a manageable number of prompt variants rather than creating a unique prefix for every user.
Typical high-potential workflows include coding agents with stable tool schemas, support assistants grounded in a shared knowledge package, document Q&A sessions, batch extraction with repeated examples, and multi-turn research agents.
Poor candidates include one-off short prompts, highly personalized prefixes, requests that change tool definitions on every call, and low-volume jobs that rarely reuse a cache entry.
Build a baseline table for each workflow:
| Metric | Why it matters |
|---|---|
| Requests per day | Determines reuse volume |
| Average input tokens | Establishes total input cost |
| Reusable prefix tokens | Defines the cacheable surface |
| Prefix variants | Reveals fragmentation |
| Reuse interval | Tests whether entries remain useful |
| Accepted-task rate | Protects quality and business value |
| P50/P95 latency | Measures performance impact |
2. Put static content before dynamic content
Prompt caching usually depends on matching the prompt from its beginning. A small difference near the front can prevent reuse for everything that follows.
Use this order where the provider and SDK permit it:
1. Stable system instructions
2. Stable policy and safety rules
3. Stable tool definitions
4. Stable reference material or examples
5. Semi-stable conversation context
6. Dynamic user input and runtime values
Do not place timestamps, request IDs, user-specific labels, randomly ordered JSON, or frequently changing feature flags near the beginning of the prompt. Normalize tool schemas and serialize structured content deterministically.
This is not permission to combine unrelated data into an oversized prefix. Keep tenant boundaries, authorization rules, and data-retention requirements intact. A cheaper prompt is not worth a privacy or isolation failure.
3. Define a cache identity and invalidation policy
Your application needs an explicit way to reason about prompt versions, even when the provider manages the cache automatically.
A practical cache identity can include:
workflow + prompt_version + tool_schema_version + knowledge_version + model_family
Track this identity in telemetry. When instructions, tool contracts, or reference data change, increment the relevant version. That makes cost changes explainable and prevents teams from confusing expected invalidation with a provider outage.
Set a reuse window based on workload behavior and provider support. A short-lived interactive agent may benefit from minutes of reuse. A recurring research workflow may justify a longer explicit cache if storage cost remains lower than repeated input processing.
4. Instrument hits, misses, writes, and accepted outcomes
At minimum, log these fields for every attempt:
- Provider, model, and workflow.
- Prompt version and cache identity.
- Total input, cached/read, cache-write, and output tokens when exposed.
- Cache hit or inferred hit status.
- Input, cache, output, and total estimated cost.
- Latency, status, retry number, and fallback path.
- Validated success or accepted-task result.
Use provider-reported usage fields as the billing source of truth when available. If a provider does not return a clean cache-hit flag, infer carefully from cached-token counts or billing records and label the metric as inferred.
Prompt caching telemetry belongs in the same trace as retries and model fallback. Otherwise a retry storm can look like a successful cache optimization. The AI observability implementation checklist shows how to connect per-attempt cost with application outcomes.
5. Calculate savings and break-even volume
Use a model that matches the provider's charging structure.
For automatic caching with a discounted read rate:
gross_savings = cache_read_tokens × (uncached_input_rate - cached_input_rate)
net_savings = gross_savings - incremental_operating_cost
For explicit caching with write and storage charges:
net_savings = avoided_uncached_cost
- cache_write_cost
- cache_storage_cost
- incremental_operating_cost
You can estimate the break-even number of reuses for one cached prefix:
break_even_reuses =
(cache_write_cost + storage_cost + implementation_cost_per_entry)
/ savings_per_cache_read
Round up to the next whole reuse. Then add a margin for misses, invalidations, and traffic variability.
Worked ROI example
Assume a workflow has:
- 40,000 requests per month.
- 18,000 input tokens per request.
- A 12,000-token stable prefix.
- A 70% effective cache-hit rate.
- An uncached input rate of $3 per million tokens.
- A cached input rate of $0.30 per million tokens.
- $350 per month in amortized engineering and monitoring cost.
Monthly cached tokens:
40,000 × 12,000 × 70% = 336,000,000 cached input tokens
Gross savings:
336 million × ($3.00 - $0.30) / 1 million = $907.20
Net monthly savings:
$907.20 - $350 = $557.20
If the workflow produces 32,000 accepted tasks, caching contributes about $0.017 in savings per accepted task. That may be meaningful at scale, but the result is far more modest than simply quoting a 90% cached-input discount.
The rates above are illustrative, not a current provider quote. Replace them with your contracted or published rates and include cache writes, storage, regional pricing, service tiers, and gateway charges where applicable.
6. Roll out with a controlled experiment
Run caching as an engineering change with a measurable control group.
- Select one high-reuse workflow.
- Freeze the evaluation set and acceptance criteria.
- Establish uncached cost, latency, and quality baselines.
- Restructure only the stable prefix.
- Send a small share of production traffic through the cached path.
- Compare hit rate, cost per accepted task, P95 latency, errors, and fallbacks.
- Expand only when savings remain positive after operating cost.
Keep retry and fallback policies separate from cache logic. A failed request may be safe to retry, unsafe to replay after partial streaming, or better served by an equivalent model. Use a defined model fallback strategy rather than treating every cache miss or timeout as the same failure.
Prompt caching KPI dashboard
Track the following metrics by workflow and prompt version:
| KPI | Formula or definition | Decision signal |
|---|---|---|
| Cacheable token ratio | Reusable prefix tokens / total input tokens | Is the optimization surface large enough? |
| Cache-hit rate | Cache-read requests / eligible requests | Is reuse actually occurring? |
| Cached-token ratio | Cached input tokens / total input tokens | How much input receives the lower rate? |
| Savings per request | Uncached baseline cost - observed cost | Is each request cheaper? |
| Net savings | Gross savings - writes, storage, and operating cost | Is the project financially positive? |
| Cost per accepted task | Total cost / accepted tasks | Did quality-adjusted economics improve? |
| P95 latency delta | Cached P95 - baseline P95 | Did user-visible performance improve? |
| Miss reason rate | Misses by version, order, TTL, or provider | What should engineering fix next? |
A high hit rate with weak savings can occur when the repeated prefix is small. A low hit rate with a large prefix may still identify a valuable opportunity if prompt fragmentation can be fixed. Read the metrics together.
Common prompt caching failure modes
Dynamic values at the beginning
Timestamps, IDs, and per-user metadata near the front fragment the cache. Move them after the stable prefix where possible.
Tool schemas change between requests
Agents often rebuild or reorder tool definitions dynamically. Normalize order, remove irrelevant tools, and version the schema deliberately.
Cache entries are written but rarely reused
Explicit cache creation can cost more than it saves when traffic is sparse or the TTL is too long. Measure reuse per cache identity before extending retention.
Teams optimize tokens but ignore outputs
Output tokens, retries, and human review can dominate total cost. Continue measuring the full request and the accepted outcome.
Provider behavior is assumed to be portable
Automatic prefix caching, explicit breakpoints, storage billing, minimum prompt lengths, eligible models, and usage fields vary by provider. Build a provider adapter and keep the business metric provider-neutral.
Fallback destroys cache locality
Switching providers or model families can eliminate reuse because caches are not portable. This does not mean fallback should be disabled. It means reliability and cost need a shared policy: fail over when required, then attribute the miss and incremental cost correctly.
Provider implementation checklist
Before enabling prompt caching for a model, confirm:
- Is caching automatic, explicit, or both?
- Which models and API endpoints support it?
- What minimum prompt length applies?
- How is a matching prefix defined?
- What TTL or retention options exist?
- Are cache writes, reads, and storage priced separately?
- Which response fields expose cached tokens or cache creation?
- Do service tier, region, data residency, or zero-retention settings change behavior?
- Are caches isolated by project, account, organization, or another boundary?
- What happens when the request falls back to another model or provider?
Use the official OpenAI prompt caching guide, Anthropic prompt caching documentation, Google Gemini context caching guide, and DeepSeek context caching guide for current implementation details. Pricing and model eligibility can change, so recheck these sources during every material cost review.
Where an AI gateway fits
A unified AI gateway does not make provider caches portable. Each provider still controls its own caching semantics and billing. A gateway can, however, give teams one place to normalize model identifiers, route eligible workloads, record provider-specific usage, compare cost per accepted task, and enforce fallback or budget policies.
Flatkey provides one OpenAI-compatible endpoint and a unified balance for access to multiple model families. That makes it easier to benchmark cached and uncached workflows without rebuilding every integration. Confirm the selected model's current caching support and provider behavior before treating a route as cache-enabled.
If you are consolidating existing clients first, use the OpenAI-compatible API gateway migration checklist and review Flatkey's current model access and pricing.
Frequently asked questions
How much can prompt caching save?
Savings depend on the reusable prefix, hit rate, provider pricing, write or storage charges, and implementation cost. Calculate net savings from observed cached tokens rather than applying the headline discount to all input tokens.
What cache-hit rate is good?
There is no universal target. A useful hit rate is one that produces positive net savings and improves or preserves cost per accepted task. Large prefixes can justify lower hit rates; small prefixes may need very high reuse.
Does prompt caching improve latency?
It can reduce input-processing latency for cache hits, but the effect depends on the provider, model, prompt size, network path, and workload. Track P50 and P95 latency rather than assuming a fixed improvement.
Should I cache the entire conversation?
Usually you should maximize a stable prefix, not blindly cache everything. Conversation turns grow and change. Keep stable instructions, tools, and reference content early, then append changing history and user input.
Can cached prompts be shared across providers?
No. Provider-side prompt caches are provider-specific. If routing changes the provider or model, treat the request as a likely cache miss unless the provider explicitly documents compatible reuse.
Is prompt caching safe for sensitive data?
Review the provider's data handling, cache isolation, retention, residency, and zero-retention terms for your account and model. Do not use cost optimization to bypass security, privacy, or tenant-isolation requirements.
Start with one repeated prefix
The best prompt caching workflow is deliberately narrow: choose one expensive, high-reuse workload; move stable content to the front; version it; measure hits, misses, latency, quality, and cost; then calculate net ROI.
When the result improves cost per accepted task, expand the pattern to the next workflow. When it does not, the telemetry will tell you whether the problem is prompt fragmentation, insufficient volume, short retention, provider pricing, or a workload that was never a good caching candidate.
