On-premises vs hosted LLM: how to choose

Teams often open this decision by comparing model benchmarks. That is the wrong end of the problem. In the engagements we run, the answer is fixed long before anyone compares output quality, because data residency rules, procurement terms, and existing processing agreements have already ruled one option out.

Start with what the data allows

Three things decide most cases: whether the material includes personal data and on what legal basis it is held, whether an existing contract or sector rule restricts where processing may happen, and whether adding a subprocessor would trigger an assessment nobody has budget or appetite for.

If any answer blocks sending content to a third party, the hosting decision is made. What remains is engineering, not strategy.

What hosted inference actually gives you

A hosted API removes an entire category of work. You get current frontier models the week they ship, no accelerator capacity to plan, no inference stack to patch, and elastic cost that starts near zero. For most general knowledge work this is the sensible default, and arguing otherwise usually means paying for control nobody asked for.

The costs are real but specific. Your content leaves your boundary. Model versions change under you, sometimes with behaviour changes you did not schedule. Rate limits belong to the provider. Cost scales with use, which is comfortable at pilot volume and less so when a workflow runs ten thousand times a day.

What running your own actually costs

Self-hosting is often presented as the safe option. It is safer in exactly one dimension, and more expensive in several others.

• Accelerator capacity has to be bought or reserved ahead of demand, and it sits idle between peaks
• Someone owns the serving stack: version upgrades, quantisation choices, batching, and the incident at two in the morning
• Open weight models trail the frontier, so quality is set by what you can host rather than what exists
• Evaluation becomes your job, because no provider is silently improving the model behind your endpoint

None of that is a reason to avoid it. It is a reason to be honest that self-hosting moves work onto your platform team rather than removing work from the organisation.

The middle position is usually the right one

Most organisations do not face one decision. They face a portfolio: some material can go to a hosted provider, some cannot, and the boundary moves as legal reviews complete and as new use cases appear.

That is the argument for putting a gateway between your applications and whatever serves the model. Applications call one internal interface. Routing, logging, access control, and redaction live in that layer. Where a request goes is then a policy decision rather than a code change, and moving a workload from a hosted API to local inference stops being a migration project.

The decision worth defending is whether you can change your mind later without rewriting the applications.

What we recommend

Use hosted inference by default. Run models yourself for the specific material that legal or contractual constraints exclude from third party processing. Put a gateway in front of both from the first production workload, before you have applications with a provider SDK compiled into them.

The exception is organisations with no platform capacity to speak of. If nobody will own an inference stack after launch, self-hosting will decay into an unpatched service holding your most sensitive content, which is worse than the risk it was meant to avoid.

The organisations that get this choice wrong are rarely wrong about residency or contracts. They are wrong about who will still own the inference stack a year after launch, which is a platform engineering question rather than a procurement one. Ask it before the decision, not after the stack has already started to decay.

Is on-premises deployment more secure than a hosted model API?

It is more contained, which is not the same thing. On-premises keeps content inside your boundary and removes a subprocessor from the picture. It also makes you responsible for patching, access control, and monitoring of the inference stack. A well governed hosted deployment beats a neglected local one.

Do open weight models perform well enough for production?

For retrieval grounded question answering, extraction, classification, and summarisation over your own content, current open weight models are generally sufficient. The gap to frontier models is widest on long multi step reasoning. Test on your own material rather than on public benchmarks, which rarely resemble the work.

What does an AI gateway do?

It gives applications one internal interface and holds routing, authentication, logging, quotas, and redaction in a single place. That keeps the hosting decision reversible and gives you one location to answer audit questions about which system sent what, where, and when.

How should we estimate the cost of running models ourselves?

Count the accelerator capacity at your peak rather than your average, then add the platform engineering time to run it as a supported service. Teams that compare only per token pricing against hardware rental usually underestimate by the cost of the people who keep it running.