When should a company use private AI models?

“Can we run this ourselves?” comes up in nearly every enterprise conversation, usually early, usually framed as a security question. It is a reasonable question with a narrower answer than either side of the debate tends to admit.

Self-hosting an open-weight model is genuinely the right call in some situations. In most, it trades a manageable contractual problem for a permanent engineering one.

The three constraints that actually justify it

Data that cannot leave a boundary. Not “sensitive data” — most sensitive data is adequately handled by a commercial provider under a zero-retention agreement. This is about data under a legal or contractual regime that prohibits processing outside a defined jurisdiction or network. Classified material, certain health and defence contexts, some regulated financial records. Here the boundary is the requirement, and self-hosting is how you meet it.

Unit economics at high, stable volume. Per-token pricing is excellent value at unpredictable volume and poor value at very high steady volume. If you are running millions of similar, short, well-understood requests a day, a fine-tuned smaller model on your own infrastructure can cost a fraction of an API call — and the workload is predictable enough to keep the hardware busy, which is the part people forget.

Behaviour you cannot get any other way. Deep domain fine-tuning, specialised output formats, unusual languages, or latency requirements that a network round trip cannot meet. This is rarer than it sounds; prompting, retrieval and tool use cover most of what looks initially like a fine-tuning problem.

If none of these three apply, self-hosting is a preference rather than a requirement, and it should be evaluated as one.

What self-hosting actually costs

The model weights are free. Almost nothing else is.

You take on capacity planning and GPU procurement, an inference stack to keep patched, evaluation infrastructure you now own entirely, and an upgrade treadmill: the frontier moves every few months, and a model you deployed a year ago is a decision you have to defend or redo. You also lose the thing commercial providers are quietly best at — the constant, invisible improvement in capability and safety behaviour that arrives without a migration project.

In our experience the recurring engineering cost of a self-hosted stack is routinely underestimated by a factor of two to three, because it is compared against API spend rather than against API spend plus the team time the API was absorbing.

The middle ground most companies actually want

The framing “our infrastructure or their cloud” is usually false. There are several arrangements between them, and one of them is normally the answer:

Commercial models in your own cloud tenancy. Major providers can be consumed inside your cloud account with data staying in your region and subject to your controls. For most enterprise data-residency requirements this is sufficient, and it is far cheaper than running a stack.

Zero-retention commercial APIs. A contractual answer to a contractual concern. Inputs are not retained and not used for training. This addresses the great majority of what gets raised as a security objection.

Mixed routing. Sensitive paths to a private model, general paths to a commercial one, chosen per request. This is the arrangement we build most often, and it works because the sensitive path is usually a minority of traffic and a narrower task — which is exactly where a smaller private model performs well.

How to make the decision

Start from the constraint, not the preference. Write down what specifically cannot happen — which data, under which rule, crossing which boundary — and check whether a zero-retention agreement or in-tenancy deployment already satisfies it. Very often it does, and the conversation ends there.

Where it does not, scope the private model to the narrow path that needs it rather than to the whole system. And decide up front who owns the upgrade cycle, because a private model is not a one-time build; it is a component your team now maintains.

The right question is never “open or commercial.” It is: which constraint are we solving, and what is the cheapest architecture that genuinely solves it?