Self-Hosted LLM Cost per Million Tokens: A Reproducible GPU Cost Model
The reliable way to calculate self-hosted LLM cost per million tokens is to divide fully loaded serving cost by the tokens your workload actually accepts—not by a GPU's advertised peak throughput.
That distinction matters. A GPU can benchmark well, yet be costly in production when it spends paid time idle, processes unusually long prompts, retries failed requests, or generates output that your product rejects. This guide provides a repeatable model for self-hosted LLM cost per million tokens, a worksheet, a worked example, and a fair way to compare a self-managed endpoint with a managed API.
Updated October 16, 2026. All dollar amounts and throughputs below are illustrative inputs, not provider quotes or performance claims.
The formula for self-hosted LLM cost per million tokens
For an output-token view, use:
cost per 1M accepted output tokens =
fully loaded hourly serving cost
/ (active-window output tokens/second × 3,600 × paid-time utilization × success rate × acceptance rate)
× 1,000,000
Where:
| Input | Definition | How to use it |
|---|---|---|
| Fully loaded hourly serving cost | Every cost required to keep the capacity available for one paid hour | Include GPU nodes, CPU/RAM, storage, network, control plane, observability, support, and an allocated share of operations work. |
| Active-window output tokens/second | Generated output throughput while the endpoint serves a production-shaped workload | Measure it with your actual model, precision, context-length distribution, concurrency, and latency target. |
| Paid-time utilization | Useful serving seconds divided by paid capacity seconds | This is not the same as a GPU's instantaneous SM-utilization reading. |
| Success rate | Completed requests divided by requests started | Count timeouts, OOMs, 5xx responses, and failed tool/schema execution consistently. |
| Acceptance rate | Output that passes the product's quality, safety, schema, and human-review tests | This converts generated output into useful output. |
The denominator is the key. Self-hosted LLM cost per million tokens should be a cost per useful unit, not a cost per GPU benchmark.
The no-double-counting rule
Use one of these two measurement paths:
- Measure active-window throughput, then multiply by paid-time utilization.
- Measure accepted tokens across the entire paid interval, then set utilization to
1because idle time is already in the observation.
Do not do both. Applying a utilization factor to wall-clock throughput that already includes idle time makes the LLM inference cost per token look worse than it is. Leaving utilization out of an active-window benchmark makes it look better than it is.
A copyable GPU cost per million tokens worksheet
Put one row in your spreadsheet for every model, hardware topology, region, and workload shape. Do not mix a short-chat benchmark with a long-context production workload.
The worksheet makes the self-hosted LLM cost per million tokens calculation auditable: another operator should be able to replace the inputs, rerun the measurement window, and reproduce the result.
| Field | Illustrative value | Collection note |
|---|---|---|
| Model and precision | your-model / FP8 |
Record the exact weight format and serving engine version. |
| GPU topology | 1 GPU node |
Capture SKU, form factor, GPU count, and interconnect. |
| Fully loaded hourly cost | $4.07 |
Sum every paid cost for the capacity used in the row. |
| Active-window output tokens/sec | 900 |
Measure under the approved p95 latency target. |
| Paid-time utilization | 55% |
Useful serving time divided by paid capacity time. |
| Request success rate | 98% |
Requests that finish without infrastructure failure. |
| Output acceptance rate | 92% |
Output that passes your agreed evaluation rule. |
| Accepted output tokens/hour | 1,606,176 |
900 × 3,600 × 0.55 × 0.98 × 0.92 |
| Cost per 1M accepted output tokens | $2.53 |
$4.07 ÷ 1,606,176 × 1,000,000 |
This is an illustrative calculation, not a price sheet for any GPU. Replacing the five inputs with measured values produces your own GPU cost per million tokens.
Worked example: why utilization changes the answer
Hold the hypothetical endpoint's $4.07 fully loaded hourly cost and 900 active-window output tokens/second constant. Small changes in the inputs change the unit cost sharply.
| Scenario | Paid-time utilization | Success × acceptance | Cost per 1M accepted output tokens |
|---|---|---|---|
| Baseline | 55% | 90.2% | $2.53 |
| Same capacity, steadier demand | 75% | 90.2% | $1.86 |
| Same capacity, bursty demand | 25% | 90.2% | $5.57 |
| Same capacity, lower acceptance | 55% | 68.6% | $3.33 |
| Faster illustrative configuration: $5.35/hour, 1,300 tok/s, 60% utilization | 60% | 90.2% | $2.11 |
The lesson is not to buy a particular accelerator. The lesson is that self-hosted LLM pricing is an operations-and-throughput model. A higher hourly node cost can yield a lower cost per accepted output if it has enough memory, batching headroom, and demand to sustain more useful work. A lower hourly rate can lose once idle capacity and retries enter the denominator.
Separate prefill from decode before benchmarking
LLM serving is not one homogeneous workload. Inference has a context-processing (prefill) phase and a generation (decode) phase; NVIDIA's TensorRT-LLM guide explicitly treats them as separate phases and emphasizes workload-specific tuning. Review the TensorRT-LLM performance tuning guide before treating any headline tokens-per-second figure as representative.
- Long prompts, retrieval context, and large attachments put pressure on prefill and KV cache.
- Long answers, code generation, and agent reasoning put pressure on decode capacity and time per output token.
- High concurrency makes scheduler behavior, batching, cache residency, and memory headroom part of the price model.
Record prompt and output distributions separately. For a retrieval-heavy assistant with 20,000 input tokens and 300 output tokens, output-only economics can miss the real bottleneck. For a coding workflow with 2,000 input tokens and 4,000 output tokens, decode may dominate.
For a rigorous self-hosted LLM cost per million tokens comparison, maintain at least these three views:
| View | Best use | Important caveat |
|---|---|---|
| Cost per 1M input tokens | Context-heavy workloads, embeddings, reranking | It does not express decode capacity. |
| Cost per 1M output tokens | Chat, coding, summarization, generation | It can hide expensive prompt processing. |
| Cost per accepted response or task | Product and FinOps decision-making | It requires a stable acceptance rule. |
Measure the denominator from production telemetry
Do not infer accepted throughput from a hardware specification. The serving stack should export the inputs that make the calculation reproducible.
For example, vLLM documents Prometheus metrics for prompt tokens, generated tokens, request completions, admission rejections, queue time, prefill time, decode time, KV-cache usage, time to first token, and per-output-token latency. Those are enough to build a measurement record instead of guessing from a benchmark. vLLM's metrics reference lists the current metric names and meanings.
Capture this record for every run:
measurement_window_start/end
model_id and weights_precision
serving_engine and version
gpu_sku, gpu_count, interconnect, region
input_tokens_p50/p95 and output_tokens_p50/p95
concurrency and request-rate distribution
prompt_tokens_total and generation_tokens_total
request_success_total and admission_rejections_total
time_to_first_token_p50/p95 and time_per_output_token_p50/p95
KV-cache usage and OOM/timeout counts
paid capacity hours and fully loaded hourly cost
acceptance-rule version and accepted-output count
Hardware telemetry is useful, but it is a diagnostic—not a unit-cost denominator. NVIDIA's DCGM profiling documentation notes that activity is an interval average and should be correlated with representative workload throughput. Use DCGM profiling metrics to investigate whether compute, memory bandwidth, or interconnect behavior is constraining the run; do not substitute a single GPU-utilization percentage for paid-time utilization.
Build the fully loaded numerator
GPU rental price is an input, not the answer. To calculate LLM serving cost fairly, include the capacity and operations costs required for the same measurement window.
| Cost area | Include | Common omission |
|---|---|---|
| Compute capacity | GPU, CPU, RAM, host/VM, reserved warm replicas | Counting only one accelerator when failover capacity is paid. |
| Data and network | Model artifacts, block/object storage, egress, load balancer, private networking | Startup, cross-zone, or log-transport costs. |
| Platform | Kubernetes, Slurm, Ray, queueing, secrets, autoscaling, image registry | Shared platform charges that appear on a different invoice. |
| Observability and security | Metrics, traces, logs, GPU telemetry, alerting, patching, audit controls | The tools needed to operate a customer-facing endpoint. |
| People and support | On-call, incident response, benchmarking, deployment maintenance, provider support | Treating engineering time as zero after the proof of concept. |
For a pilot, report two figures:
GPU-only cost per 1M tokens
Fully loaded cost per 1M accepted tokens
The gap is not a mistake; it is the operating cost of the self-hosted service. It makes the self-hosted LLM pricing decision auditable.
Include retries and rejected output
Generated tokens are not automatically useful tokens. A request can consume prefill and partial decode capacity before it times out, produces invalid JSON, fails a tool call, breaks a policy rule, or misses a quality threshold.
Use an acceptance adjustment that your product team can reproduce:
accepted output tokens =
generated output tokens
× execution-success rate
× product-acceptance rate
If acceptance is measured at the request level, record the exact rule: evaluator version, sampling set, schema validator, human-review protocol, or task-success event. Changing the rule without versioning it makes a cost trend meaningless.
This is also why a raw LLM inference cost per token can be misleading. A configuration that gains 30% throughput but lowers accepted-task success may cost more at the product layer.
Compare self-hosting with a managed API on the same unit
Self-hosting purchases capacity. A managed API purchases usage. They only become comparable when the numerator and denominator match.
Use this break-even throughput check when the API benchmark is expressed in the same unit:
required active-window output tokens/second =
fully loaded hourly serving cost × 1,000,000
/ (managed-API cost per 1M accepted output tokens × 3,600 × utilization × success rate × acceptance rate)
Before deciding, align these controls across both paths:
- Same model or an explicitly evaluated quality-equivalent model.
- Same prompt and output-length distribution.
- Same latency SLO and concurrency shape.
- Same retry, fallback, and acceptance rules.
- Same treatment of cache hits, tool failures, and cancelled work.
If direct GPU control, custom weights, hard placement, or a specialized kernel stack is a requirement, self-hosting may remain the right answer even above the managed baseline. If traffic is low or bursty, models change frequently, or the team does not want to run inference infrastructure, a usage-priced API can absorb utilization risk.
For a broader buying comparison, see Flatkey's GPU cloud vs. model API guide and GPU rental prices guide. When direct GPU control is not required, a managed route is a useful benchmark: Flatkey provides a unified model and tool API with one balance, so teams can compare a managed path against their self-hosted calculation without maintaining separate provider keys. Start a Flatkey API baseline with the same test set and acceptance rule.
A seven-step procurement test
Use this self-hosted LLM cost per million tokens framework as a procurement artifact, not just an engineering benchmark. It should show exactly which workload, capacity, and quality assumptions changed the outcome.
- Define the unit: input token, output token, blended token, accepted response, or accepted task.
- Measure active-window throughput under production-shaped traffic, not a single synthetic prompt.
- Measure paid-time utilization across every paid replica and warm-reserve node.
- Add all operating costs to the hourly numerator.
- Version the success and acceptance rules, then calculate accepted output.
- Run the same workload through a managed API baseline.
- Review cost, p95 latency, reliability, quality, and engineering effort together before reserving capacity.
Common errors in a self-hosted LLM cost model
Using peak benchmark throughput. Peak is useful for comparing a controlled setup. It is not the expected throughput of your production mix.
Calling GPU utilization capacity utilization. A busy GPU can still be attached to an endpoint that is oversized for paid demand. Use paid-time utilization for unit economics and use hardware telemetry to diagnose the reason.
Applying utilization twice. Choose active-window throughput plus utilization, or wall-clock throughput that already includes idle time.
Ignoring prompt tokens. Long context raises prefill and KV-cache pressure even when the product mainly bills for output.
Treating every generated token as accepted. Retries, invalid tool arguments, schema failures, and quality rejects are paid work.
Comparing a GPU invoice with API list price only. Engineering, support, storage, network, observability, and warm capacity belong in the same calculation.
Frequently asked questions
How do I calculate self-hosted LLM cost per million tokens?
Divide fully loaded hourly serving cost by accepted tokens per paid hour, then multiply by 1,000,000. Use measured production-shaped throughput, paid-time utilization, success rate, and acceptance rate—not theoretical GPU peak.
Is GPU cost per million tokens the same as LLM inference cost per token?
GPU cost per million tokens can be a GPU-only infrastructure metric. LLM inference cost per token should include the relevant operational numerator and a clearly defined denominator. For a product decision, cost per accepted response or task is often the safer companion metric.
Should I use output tokens or total tokens?
Use both when prompt and output costs have different bottlenecks. Output tokens are easy to compare for generative workloads; total or weighted tokens better represent long-context retrieval workloads. Keep the unit identical when comparing self-hosting with an API.
When does self-hosting save money?
It is more likely to save money when demand is stable, capacity stays well utilized, the model and deployment are stable, and direct control adds value. It is less likely to save money when traffic is bursty, model selection changes often, or operational overhead outweighs capacity savings.
Conclusion
The defensible self-hosted LLM cost per million tokens number is fully loaded paid capacity divided by accepted throughput. Measure prefill and decode separately, account for idle time once, version your acceptance rule, and compare a managed baseline on the same unit. That turns a GPU quote into a decision-ready cost model.
