Can part of the AI work a company does run on a normal laptop, with data that never leaves the company? That is the question behind our current research. One thing is already clear: a small model alone is not enough. What decides whether local AI is usable is the architecture around it.
Why local AI matters
Sending every document to a cloud model is not always possible. Contracts, client data and internal reports are often exactly the documents a company would rather keep on its own devices. If part of the work can run locally and reliably, the data stays where it is, and the cloud is used only where it really adds something.
How we test it
Answering the question means testing many combinations of small models, quantisations and prompts on the same data. We run this work on the Discoverer supercomputer (PetaSC, Sofia) through the EuroHPC AI Factories Playground. Its H200 GPUs let us run dozens of configurations in parallel and measure them under identical conditions. We work only with synthetic data.
Four things that decide whether a small model is usable
A deterministic layer. The model returns structured output defined by a JSON schema. Every output is validated and every citation is checked against its source. What the model produces is not trusted until it passes these checks.
Short, closed questions. Instead of one large prompt, the work is broken into small, well-defined steps. Each step asks one question with a clear answer.
Calibrated decision models. Where a decision is sensitive, it is made by a model calibrated for that decision, not by a general one.
Our own evaluation. We measure on our own test harness with synthetic Slovak, Czech and English data, not on public leaderboards. What matters is how a model performs on the kind of documents and languages it will actually handle.
Local where it fits, cloud where it makes sense
The goal is simple: AI that runs locally and reliably where it should, and cloud where it makes sense. The model can change. The architecture around it is what makes the result something a company can rely on.
If you are working on local models or evaluation, get in touch. We are happy to compare notes.
Sources
- EuroHPC JU: AI Factories Access Modes (eurohpc-ju.europa.eu/ai-factories/ai-factories-access-modes_en)
