On-premises AI: more control, but at a cost

Using AI is now standard in the workplace. The efficiency gains are significant—but so are the AI costs per employee. However, AI delivers its full value only when it can work with a business’s own data. This is precisely where many companies struggle, as the established providers are now exclusively U.S. or Chinese companies.

This raises a question: Can medium-sized and large enterprises operate their AI infrastructure themselves? After all, they already have an IT department and a server infrastructure in place. This article explains what running AI on premises involves. Cloud hosting is another option that companies should also consider when making this decision.

Some open models can now compete with the leading models from Anthropic and OpenAI. The following chart shows that the capabilities of open models are only a few months behind those of commercial models.

The intelligence and size of these models vary considerably. For language models, intelligence depends on size, architecture, and training data. So far, each new generation of models has been significantly larger than the previous generation. This has worked well, but it comes with a price: greater hardware demands. The next step is to optimize the model so that it requires fewer resources while maintaining the same level of intelligence. The following chart shows how this works in practice.

The leading open models differ only slightly in intelligence, but their sizes vary considerably. For example, GLM-5.3-Flash, with 320 billion parameters, performed better in tests than DeepSeek V4 Pro, which has 1.6 trillion parameters.

The leading open language models come from Chinese or U.S. labs—European models are far behind. The Swiss model Apertus (V1) scores just 5 points on the Artificial Analysis Intelligence Index. By contrast, Chinese models score 40 points (DeepSeek V4.1 Flash) to 45 points (GLM-5.3).
 

Kimi K3 is one of the most capable open models and trails only slightly behind Fable 5 and GPT-5.6 Sol. With 2.8 trillion parameters, however, it is also very large and requires 1.5 TB of RAM for the model weights alone.

A model’s memory requirements depend on its parameter count and precision: A 30B model with 8-bit precision requires 30 GB of RAM. Reducing its precision—for example, to 4-bit or 2-bit—conserves resources. We call this process “quantization.” It significantly reduces the amount of memory required—but with the disadvantage of lower quality. Even so, this is standard practice when user demand is high because it enables all requests to be handled. With NVFP4, NVIDIA offers 4-bit quantization for its GPUs. For very large models, quality is only slightly lower than with 8-bit. This makes it an attractive option for hosting. By contrast, quality does decline for smaller dense models (under 30B). For these models, Dynamic NVFP4 is recommended, as it stores only the less sensitive parts of the model in 4-bit format.

For most tasks, small- to medium-sized models are sufficient; for more complex tasks, large models are essential. As a guide: A small model has around 30 billion parameters, a medium-sized model around 300 billion, and a large model 1–3 trillion.
 

The type of use significantly affects the resources needed. Chat requires relatively few resources. We assume a context window of 32k tokens. By comparison, developer agents require far more resources because their requests also include code and documentation. Here, we assume a context window of 300,000–500,000 tokens. One word corresponds to roughly 2 tokens.

For each new request, the model has to load the entire representation of the conversation (KV cache) into GPU memory. For long conversations, this can amount to tens or hundreds of gigabytes—and it can even exceed the memory that the model itself uses. This is why high memory bandwidth is so important. 

All this means that suitable hardware requires a large amount of fast memory—which is also expensive.
 

Today’s enterprise solutions are based on NVIDIA GPUs from the Blackwell series or AMD GPUs. The state of the art is NVIDIA’s B300. One option here is the DGX B300 (a 10U server with 8 GPUs). But an even more efficient alternative is the GB300 NVL72 in a liquid-cooled rack system with 72 GPUs and 36 CPUs.

Many companies are still using older GPUs from the Hopper series, such as the H100 and H200. However, they do not support NVFP4 and are less energy-efficient by a factor of 2–4 (8 GPUs) to 50 (72 GPUs). NVIDIA has announced even more efficient GPUs in the Vera Rubin series, but they are not yet available.

In its marketing materials, NVIDIA uses tokens per megawatt to express energy efficiency. The company’s main target group consists of large data centers.
 

The following examples show how a company might use a DGX B300 server. The figures are calculated estimates, not measured values.

Scenario 1: Day-to-day operations 
Chat with a 32k-token context window, at speeds comparable to Copilot. Breakdown:

• 2 GPUs: Qwen3.8-27B for simple tasks (comparable to Claude Sonnet)
• 6 GPUs: GLM-5.3-Flash for more complex tasks (comparable to Claude Opus)

This is enough to handle 50–100 requests at the same time or 500–1,200 users per day.

Scenario 2: Development team
Developer sessions have an average context window of 300k tokens. The output speed should be 50–60 tokens per second. This setup uses GLM-5.3-Flash as the model. Using 2 GPUs per model instance improves performance and results in a total of 4 instances. This is enough for 8–16 coding agents running at the same time or 10–25 developers.

Older hardware 
The situation is different with the H100 GPUs, which many organizations are still using: For Qwen3.8-27B (80–120 tokens per second), a single GPU is fully utilized with 1–2 sessions. Optimizing the setup—for example, with multi-token prediction—enables 2–4 sessions. A single GLM-5.3-Flash session requires two H100 GPUs to achieve 50–70 tokens per second.
 

Self-hosting protects company data from unauthorized access by the model provider. However, some risks remain:

• Political factors influence a model’s behavior. Most models—whether American or Chinese—have a certain degree of bias.
• Several Chinese coding models produced code with more vulnerabilities when the prompt appeared to come from an individual working for the U.S. government. What’s more, these vulnerabilities were well hidden. This tendency was particularly noticeable with Qwen. (Source: Booz Allen, 2026)
• Models can be vulnerable to prompt injection. This means they may carry out tasks that originate from an outside attacker rather than the user. Without an additional application layer, users lack protection.

Under an open-weights license, the model provider assumes no liability—the user bears the risks of using the model.
 

The model itself is only the “brain.” A separate platform is also needed to deliver the model. At a minimum, it has to handle access management, billing, security checks, and load balancing. Commercial open-source tools such as Bifrost or LiteLLM can provide this functionality.

Companies can expand the tech stack for their AI platform to suit their needs. Features such as vector search, evaluations, MCP registry, and coding sandboxes are available as standard packages from hyperscalers. For on-premises deployments, however, companies have to provide and maintain them in-house.
 

Running large language models considerably increases the demands on hardware, power, and cooling compared to conventional applications. A 14-kW server with 5.6 kW of cooling (PUE 1.4) consumes 172 MWh per year on average; a rack server with 72 GPUs consumes around 1.5 GWh per year. At an average electricity price of CHF 0.20/kWh, that amounts to CHF 34,400 and CHF 300,000 in energy costs per server per year, respectively.

The global expansion of AI data centers has led to supply shortages for GPUs and memory: RAM prices have increased tenfold over the past 18 months, while GPU prices have tripled. There are only two GPU vendors, and lead times for AI servers are several months.

Depending on the supplier, an NVIDIA DGX B300 with 8 GPUs costs between CHF 400,000 and CHF 700,000. For comparison, the previous model, the DGX B200, costs between CHF 300,000 and CHF 400,000. Both require 14 kW. In addition, the heat generated must be dissipated. This means an additional 3–6 kW is needed for cooling. In practice, expenditure on cooling infrastructure typically amounts to around 30–50% of the hardware costs. Companies also need to budget for electricity costs, which vary by location.

AI models are evolving rapidly: New, improved, or more efficient versions are released several times a year. Organizations should roll them out promptly. In addition, the tooling needed for AI hosting also requires ongoing maintenance. Altogether, this means that a company would need 2–3 people at a minimum to operate an in-house AI infrastructure.
 

Anyone buying new AI hardware today is paying a high price due to strong demand and limited production capacity. With enough investment, they can build an infrastructure that rivals the quality of the offerings from the major providers. However, they will need to continually maintain and update their AI infrastructure themselves.

In-house hosting keeps company data outside the jurisdiction of U.S. or Chinese authorities. But it doesn’t protect it against the political bias that is already embedded in the model weights.

Are you planning to host your own AI infrastructure? Our experts are on hand to assist you.