The phrase is everywhere now. Private AI. On-premise. Your data never leaves your servers. It has become the default reassurance in every AI pitch deck, and it is almost never defined. That vagueness matters, because the difference between a claim that survives an audit and one that collapses under a single question is not marketing — it is architecture.
Here is the honest version of the claim, the parts vendors leave out, and the questions worth asking anyone who makes it.
There is no single “private” — there is a spectrum
Data residency is not a switch. It is a series of decisions about where inference happens, where documents rest, where logs accumulate, and what the system is allowed to reach out to while it works. A deployment can be private in one of those dimensions and wide open in another.
Most businesses that say they need on-premise actually need the fourth row. The fifth is expensive, slow to change, and genuinely necessary for a narrow set of defence, healthcare and government work. Buying it when you needed row four is one of the more costly mistakes in this space.
The part almost every claim gets wrong
Running the model locally makes inference private. It does not make the system private. An agent is not just a model — it is a model plus tools, and the tools are where data escapes.
- A web search tool sends your query — which may quote a client document — to a search provider
- A remote tool server processes whatever the agent passes to it
- Hosted memory or vector search puts your embeddings on someone else's disk
- Speech, transcription and image generation are usually separate hosted APIs
- A cloud fallback configured “for when the local model struggles” silently reverses the entire guarantee at exactly the moment the query was hardest
How to state the claim so it survives scrutiny
We write data-handling statements per system, not per company, and they name each boundary explicitly. A defensible version reads closer to this:
Model inference runs on infrastructure in your account. Documents, embeddings and conversation history are stored in your database. The agent has network access to two named internal APIs and nothing else. There is no external model fallback — if the local model is unavailable, the request fails and is queued.
That is longer than “your data never leaves.” It is also true, checkable, and something a compliance reviewer can approve without a follow-up call. Precision is not a weaker claim — in front of a buyer who has been pitched three times this quarter, it is the stronger one.
Five questions that separate real from rehearsed
- Which exact component performs inference, and what account does it run in?
- List every outbound network call the system can make, including tools and telemetry.
- What happens when the local model is unavailable or the answer confidence is low?
- Where do prompts, retrieved chunks and outputs get logged, and for how long?
- If we terminated tomorrow, what would we still hold — weights, adapters, index, source, or nothing?
The last one is the tell. A vendor who cannot answer it is selling access, not a system. Everything we build in this space is designed so the answer is: the weights, the adapters, the index, the source and the infrastructure definitions — all of it, in your accounts, on full payment.
When private AI is worth the extra work
It is not always. Self-hosting adds GPU cost, an upgrade treadmill, and an on-call responsibility you did not have before. The honest test is whether one of these is true:
- A regulator, contract or client questionnaire explicitly restricts where the data may be processed
- The corpus is the business — case files, patient records, source code, deal documents, salary data
- Volume is high and sustained enough that per-token pricing has become a real line item
- You need behaviour that is stable and versioned rather than changing under you when a vendor ships a new model
The interesting shift of the last year is that the choice is no longer a compromise. Open-weight models under permissive licences now handle document work, classification, extraction and tool-calling at a quality that would have been frontier-only not long ago. For the workloads most businesses actually have, privacy no longer costs you capability. It costs you engineering — which is a bill you can plan for.