Sixteen GPUs and two racks: what a sovereign AI platform needs on-premises

Summary: An infrastructure lead who has run five-kilowatt racks for a decade is told the AI platform is going on-premises. The GPUs turn out to be the easy part. This is what it actually takes to house sixteen H200s: the power, the cooling, the six environments on two nodes, and the disaster-recovery decision nobody wants to make.

For ten years the racks had drawn four or five kilowatts each. The cooling had never been questioned. Hardware arrived in days. Then the infrastructure lead at a federal entity in the UAE got the brief: the new AI platform goes on-premises, the data never leaves the building, and the spec says sixteen NVIDIA H200 GPUs.

His first instinct was the same as everyone’s. Order the GPUs.

The accelerators are the easy part

They’re easy to specify, anyway. NVIDIA’s own figures for the H200 are 141 GB of memory per GPU at 4.8 TB/s, with a configurable draw of up to 700 W each in the SXM form factor. Sixteen of them across four HGX nodes with NVLink between the GPUs in each node. Two nodes for production, one for user acceptance testing that doubles as pre-production, one for development. Eight worker nodes underneath carrying the application, database, search, analytics and object-storage tiers. Seven Kubernetes control-plane nodes. A 25/100 GbE fabric tying it together.

That’s a bill of materials. It fits on one page. And it’s where most conversations stop, which is why most sovereign AI builds arrive at a facility that can’t take them.

Because the honest cost of an accelerator isn’t the invoice. One GPU server at this class runs to the high hundreds of thousands of dollars, and the lead time from order is 10 to 16 weeks. That’s the number that reorganises a project plan. Nothing in Wave 1 can depend on hardware that’s four months out, which is a story we’ve told separately about starting in the cloud first. This story is about what happens when the trucks finally arrive.

The facility conversation

We came in at the point where the infrastructure lead had a BOM and no building. The first thing we did was throw out the rule of thumb and size the rack elevation from component draw.

Here’s what that produced. Rack A carries the two production GPU nodes, five production workers and the network fabric. Twenty-one rack units used, roughly 13.8 kW typical and 15.8 kW at peak. Rack B carries the UAT and development GPU nodes, three workers, the seven control-plane nodes and the backup repository. Twenty-three units, about 12.2 kW typical and 14.3 kW peak.

Read those numbers against the five-kilowatt racks he’d been running. A single rack was about to draw three times what the whole row used to. The facility request became: 32 kW of usable IT power across two racks, delivered on separate A and B feeds with N+1 redundancy, from UPS and generator-backed supply. Air cooling, front-to-rear airflow, with hot and cold aisle containment strongly recommended at that density. No liquid cooling, no facility water, which matters, because both would have turned an equipment install into a construction project.

That was the threshold moment. Not the purchase order. The realisation that a sovereign AI platform is a building problem before it’s a computing problem, and that the electrical contractor is on the critical path.

Six environments on two nodes

The next trial was one the infrastructure lead hadn’t seen coming. The entity needed six segregated environments: development, system integration testing, UAT, pre-production, an agent sandbox and production. On a conventional platform that’s six sets of hardware, or a lot of VMs. On a GPU platform it’s a budget conversation nobody wins.

The answer was Multi-Instance GPU. NVIDIA’s MIG partitions a single Hopper GPU into as many as seven instances, each with its own memory, cache and compute cores, and each fault-isolated so a crash in one doesn’t touch the others. The two non-production nodes were carved into per-environment quotas, giving five logically separate environments on two physical nodes, with production kept whole on its own pair. When a pre-production performance rehearsal needed a full node, the partitions could be collapsed for the window and restored afterwards.

That single design decision took two GPU nodes off the bill of materials and kept the environment count the auditors wanted. It’s the kind of thing that only shows up if someone has built one of these before.

The disaster-recovery decision

Every one of these programmes reaches the same ordeal, and it’s not technical. It’s the moment somebody prices a duplicated estate for disaster recovery and the number doubles.

For a single-site government deployment, a mirrored second estate is usually the wrong answer. It doubles the sovereign footprint, doubles the power and cooling problem we’d just solved, and protects against a scenario that’s real but rare. The design we recommended was immutable backup and controlled restore. A hardened repository, roughly 484 TB raw, whose backups can’t be deleted or encrypted by any account, including an administrator’s and including a compromised one. That last clause is the ransomware defence. Restore tests quarterly, a full DR rehearsal annually, both with written results.

The cost of that choice is honest and stated: a recovery time objective measured in days, not minutes, for total site loss. The baseline was 72 hours. If the entity needed faster, a warm secondary site or a standing cloud recovery environment was priced as a separate, removable line. Nobody was forced to pay for resilience they hadn’t chosen.

What the build actually looks like

The return

The platform that came out of this has no single point of failure in its services, and a documented degraded mode for the one component that’s genuinely hard to make redundant. If a reasoning-tier GPU node is lost, the model gateway reroutes leadership queries to the surviving tier and flags the response as reduced-capability. The platform slows. It doesn’t stop.

And there’s a line in the design for Year 3: one additional GPU node and a storage expansion, priced separately and removable. The infrastructure lead knows where the next node goes, which rack has the units and the power for it, and what it’ll cost. Three years ago he’d have called that a computing problem. Now he knows it’s a floor plan.

That’s the elixir he brought back: a data centre that had quietly become an AI facility, and the fluency to run it. If you’ve got a bill of materials and no building yet, talk to 10ⁿ Tech before the trucks are booked.

Related resources

• Cloud first, sovereign second: starting an AI platform before the hardware arrives
• Sovereign AI in the GCC: what in-country actually requires
• Self-hosted on-prem AI
• On-prem IT infrastructure

Photo credits: Winston Chen and Tyler on Unsplash.