What Is Private Inference and When Does an Enterprise Need It?
Private inference is the practice of running language models on infrastructure the customer controls, so that prompts, retrieved documents and generated output never leave their security perimeter. It is used when regulation, data residency rules or internal policy prevent an organization from sending operational data to a commercial model API. In practice it means open-weight models served on the customer's own GPUs, with the surrounding application, governance and audit layer unchanged.
When an enterprise actually needs it
Most AI workloads do not need private inference. Commercial model APIs are cheaper per token, stronger at most tasks and require no infrastructure work. Private inference earns its cost in a narrow set of cases.
The first is a legal or contractual bar. Some regulated institutions are not permitted to transfer operational data outside a defined jurisdiction or outside their own tenancy, regardless of the provider's certifications.
The second is data classification. An organization may be free to use a commercial API for general work while one specific data class, such as customer records or internal risk material, cannot be included in any external call.
The third is supplier concentration. Some institutions require a demonstrated fallback for critical processes, so that one vendor's outage or policy change cannot stop an operational workflow.
If none of these apply, private inference adds cost and operational burden without adding value.
Sizing comes from concurrency, not from user counts
The most common sizing error is estimating GPU requirements from the number of people who will use the system. That number says almost nothing about the load.
What determines capacity is how many requests are served at the same time and how long each request's context is. A retrieval-heavy workload with long contexts behaves very differently from short chat turns, even at identical user counts.
In a feasibility study we ran for a Slovak telecommunications operator, a retrieval workload at 32K context reached roughly 33 seconds to first token for a single user. At five concurrent users the same workload ranged from about 101 to 169 seconds. Raising the context to 100K reduced effective vLLM concurrency to about 1.64.
Those numbers are specific to that model, that hardware and that workload. The transferable part is the method: size from measured peak concurrency and the real distribution of context lengths, using the organization's own documents.
Configuration errors look exactly like model failures
The same study produced a result worth repeating, because it is easy to get wrong in a way that costs real money.
An initial round of testing concluded that 7B and 8B models could not produce acceptable Slovak, and that 70B was the practical minimum for the language. That conclusion was wrong.
On re-testing with deterministic settings, at least three runs per case, verified chat templates and objective detection of foreign scripts, the degraded output turned out to be a configuration artifact. The vLLM chat template was wrong or missing. With the correct template the same 8B model produced no foreign-script output at all and became usable, with caveats.
The consequence is financial. A wrong conclusion at this point moves the hardware requirement from a 24GB card to an 80GB one, which is roughly a fivefold difference in monthly cost. Evaluation discipline matters more than parameter count.
Model behaviour is also task and language specific in ways published benchmarks do not show. In the same tests one 72B model emitted Chinese characters in Slovak output, and one reasoning model placed its reasoning trace, in English, inside the answer.
What the measured tiers looked like
Three tiers came out of that testing. A primary Slovak-language tier running EuroLLM-22B in BF16 on an A100 80GB. A budget tier running Llama 3.1 8B in BF16 on 24GB cards such as L4, A10G or RTX 4090. A long-context retrieval tier running Llama 3.3 70B AWQ on H100 NVL.
Each tier was pinned to specific vLLM and driver versions. A driver or runtime upgrade can change both output and latency, so version pinning belongs in the deployment contract rather than in a runbook.
What does not change when the model moves
The model is one component of an enterprise AI deployment, and usually not the one carrying the governance burden.
Routing decides which model and which infrastructure handles a given task. Permissions decide what an agent is allowed to do and which data it may read. Verification checks output against a quality bar before it is used. The audit trail records what happened and on what basis.
Those four functions are where auditability and control live. When they sit in a platform layer instead of inside the model integration, moving a workload from a commercial API to a self-hosted model becomes a configuration change rather than a rebuild. That portability is what makes a sovereign deployment variant economically viable at all.
The limits of what we measured
Stating limits is part of the method, so these are ours.
The study was a feasibility assessment, not a production deployment, and it used synthetic prompts rather than the client's real production traffic.
The models tested were the 2025 generation. It is a Slovak-language deep dive, not a statement about the current state of open-weight models.
Quantized EuroLLM on 24GB cards was not verified, because one quantization test failed and was not repeated. A clean latency figure at 100K context was extrapolated rather than measured.
Anyone sizing a deployment from published figures, ours included, should treat them as a starting hypothesis and run a short paid pilot on their own prompts and their own documents before committing to hardware.
How we work with NVIDIA
Elevon.io is a member of the NVIDIA Inception program. For this part of our work the relevant benefits are access to developer tools, technical resources, preferred hardware pricing and GPU cloud credits, which lower the cost of testing a configuration before a customer commits capital expenditure.
That matters because the honest sequence for private inference is to prove the requirement on rented GPUs first and buy hardware last. Programme access makes the proving step cheaper, which makes it more likely to actually happen.
Where to start
If your organization has been told that a particular AI use case is impossible because the data cannot leave your environment, that is a constraint on where inference runs. It is not a constraint on whether the use case is feasible.
The first step is not procurement. It is a short measurement exercise on your own data, at your own context lengths and at your expected concurrency, which tells you what the deployment would actually cost.