Self-Hosted LLM Cost: Calculate a Workload-Specific Break-Even

A self-hosted LLM is cheaper only when its complete operating cost, measured capacity and task quality compare favorably with the API alternative for your workload. There is no universal token-volume threshold or GPU configuration that establishes the answer. Start with observed traffic and a common acceptance test, then calculate both alternatives using dated prices.

The numbers below are deliberately hypothetical teaching inputs. They are not current cloud quotes, model benchmarks, salary estimates or a recommendation to buy hardware. Replace every price, throughput and quality assumption with evidence before making a purchasing decision.

Define the work before comparing token prices

Take a fictional support tool that receives 100,000 requests in a month. Each request averages 3,000 input tokens and 400 output tokens. That represents 300 million input tokens and 40 million output tokens before retries, tools or other processing.

A token is a billing unit, not necessarily a successful business result. Define acceptance: the answer must cite the correct policy, avoid disclosing another customer's data and resolve the intended question. Test the same questions and rubric for both model configurations.

Record different request types separately. A long-document summary and a short classifier can have different memory, latency and cost characteristics. An average across them can hide the workload that sets your actual capacity requirement.

Calculate an API baseline with all billed components

For illustration only, suppose an API charges $1 per million input tokens and $4 per million output tokens. The fictional monthly baseline is:

Input:  300 million / 1 million x $1 = $300
Output:  40 million / 1 million x $4 = $160
Token subtotal:                         $460

Add the selected provider's actual charges for cache writes or reads, tools, images, audio and other billable work where relevant. Include retries and calls that fail to produce an accepted answer. Account for minimum commitments or dedicated capacity if the commercial arrangement has them; API spend is not always purely variable.

Keep the pricing date, model identifier, region and contract assumptions with the calculation. A comparison using one model's output rate and another model's quality is not a meaningful estimate. The prompt caching guide explains why warm-request savings cannot be applied blindly to every request.

Build a complete self-hosting budget

An illustrative monthly budget might allocate $900 to compute, $100 to storage and transfer, and $1,200 to engineering and operations. Its total is $2,200. These are arbitrary inputs showing the calculation structure, not observed market rates.

Replace them with a scoped inventory:

• Compute capacity and any minimum reservation period.

• Storage for weights, data and operational records.

• Network transfer and supporting application services.

• Model and software licensing obligations.

• Deployment, updates and security maintenance.

• Monitoring, incident response and support coverage.

• Task evaluation and regression testing.

• Standby capacity, recovery testing and other resilience costs.

Avoid treating an engineer's existing salary as proof that maintenance is free. Equally, do not impose a fixed staffing fraction on every deployment. Estimate the actual work and identify the owner. Hosted APIs also require application evaluation, integration maintenance and incident handling, so include those costs on that side of the comparison too.

Calculate break-even only under explicit assumptions

In the simplified example, the API token subtotal is $0.0046 per request. If self-hosting costs a fixed $2,200, has no additional variable cost and achieves equivalent quality, the arithmetic crossing point is approximately:

$2,200 / $0.0046 = 478,261 requests per month

This is not a recommendation or a feasible-capacity result. It excludes several real costs listed above. The hardware may be unable to serve that workload at the required latency; adding capacity changes the fixed cost and therefore changes the crossing point.

For a more complete model, compare total cost over a range of workloads using each alternative's actual fixed and variable charges. If self-hosted variable cost equals or exceeds the API's equivalent variable cost, increasing volume alone may not create the expected crossing point.

Measure capacity with the actual request mix

Load-test the chosen model, precision, serving stack and hardware. Include prompt lengths, output lengths, concurrency and arrival bursts representative of production. Distinguish aggregate throughput from each user's latency and measure waiting time as well as generation.

Define the acceptance condition before tuning the server. A configuration that maximizes tokens per second by delaying requests may violate the application's response requirement. A configuration that looks fast on short prompts may run out of memory on the longest supported document.

Capture cold starts, warm behavior and recovery from a failed worker. Autoscaling suitability depends on your startup time and whether work can wait. There is no fixed cold-start duration that applies to every model, storage setup and runtime.

Size memory with runtime evidence

A useful first approximation separates model weights, attention caches, runtime buffers and temporary working memory. For an idealized seven-billion-parameter model, two bytes per parameter gives 14 billion bytes of raw weights. Four-bit storage gives a theoretical 3.5 billion bytes before metadata and other overhead. Neither figure is the memory required by the complete running service.

Context length, concurrency, model architecture and cache precision can change working memory substantially. Check the model and serving implementation rather than relying on a generic model-size-to-GPU table. Hardware specifications such as NVIDIA's L40S page describe the device; they do not establish your model's throughput or safe concurrency.

The vLLM quantization documentation lists implementation and hardware compatibility considerations. Quantization may reduce memory, but its impact on speed and task quality must be tested for the selected method. Do not assume a universal percentage loss or automatic suitability for extraction, reasoning or a particular language.

Normalize the result by accepted work

Suppose, again illustratively, that one configuration accepts 95,000 of the month's requests while another accepts 90,000. The cost of correcting or escalating the remaining requests belongs in the comparison. A cheaper raw generation can become an expensive workflow when it produces more rejected results.

Calculate total operating cost divided by accepted tasks under a consistent rubric. Show the rejection rate, latency and human review effort beside that value. Our RAG evaluation framework provides an example of separating evidence retrieval from answer quality rather than scoring fluent output alone.

Evaluate control and resilience independently

Self-hosting can provide more control over infrastructure and model deployment, but confidentiality still depends on access controls, logs, network paths, backups and operating practice. A managed private deployment may also satisfy some requirements. Have applicable residency and contractual obligations assessed for the actual design.

Self-hosted capacity can queue, saturate or fail just as hosted capacity can. Model weights remaining available does not remove runtime patching, licensing or hardware maintenance. Define recovery objectives and test the required capacity after a failure rather than assuming a second GPU provides every resilience property.

What hardware should we buy first?

Rent or otherwise test an appropriate candidate configuration before a long commitment. Select it from the model, memory estimate, measured workload and operational requirements, not a universal recommendation for a particular card.

Does an API-compatible endpoint make migration automatic?

No. Compare authentication, streaming, tool calls, structured output, error behavior and model quality. Similar request syntax does not guarantee equivalent application behavior.

What should an approval brief contain?

Dated prices, benchmark configuration, acceptance results, peak-capacity evidence, complete operating costs and a recovery plan. Discuss an inference architecture review with those inputs so the decision is based on the workload you actually need to serve.