Key Takeaways:

• Infrastructure sets the ceiling on model performance: compute budget, memory bandwidth, networking, and power decide what a model can learn and how fast it can answer.
• Chinchilla scaling laws show that for a fixed compute budget, performance depends on balancing model size against training data — roughly a 20:1 token-to-parameter ratio at the compute-optimal point.
• Scaling has hard floors: irreducible loss means returns diminish sharply, and training scaling of about 4x per year faces limits in power, chips, and data by 2030.
• In training, the interconnect between GPUs (InfiniBand, RoCE, or high-speed Ethernet) becomes the primary bottleneck once you move past a single node.
• In inference, the memory wall dominates: token generation is limited by how fast GPUs can move weights and KV-cache data, not by raw compute.
• HBM has become the critical scarce component — Micron and SK Hynix sold out 2026 capacity, and HBM4 targets roughly 2 TB/s of bandwidth.
• Inference is taking over: it is projected to reach about two-thirds of AI compute in 2026 and 80–90% of a production system’s lifetime cost.
• Software efficiency — quantization, pruning, distillation, and KV-cache compression — increasingly closes the gap between hardware a model needs and hardware available.

AI infrastructure plays a decisive role in model performance — arguably the decisive role once the architecture is fixed. The model’s design determines what is theoretically possible, but the hardware around it, the compute budget, memory bandwidth, networking fabric, and power supply, determines what is actually achievable. A frontier architecture starved of compute or bottlenecked on memory will underperform a simpler model running on a well-built system. Infrastructure is not a supporting cast member here; it sets the ceiling.

The clearest way to see this is to split performance into two phases. During training, infrastructure decides how large a model you can build and how much data you can feed it, which directly shapes how capable the finished model becomes. During inference — the model actually answering queries in production — infrastructure decides how fast and how cheaply it responds, which shapes whether the model is usable at all. Both phases are governed by physical limits in chips, memory, and interconnect, and those limits are now the binding constraint across the industry.

Training: compute budget sets the performance ceiling

The relationship between infrastructure and model quality during training was formalized by scaling laws. The 2022 Chinchilla work from DeepMind showed that for a fixed computational budget, measured in floating-point operations (FLOPs), there is a zero-sum trade-off between model size and training data. A bigger model costs more compute per pass; more data costs more passes. Spend too much on parameters and you starve the model of data; spend too much on data and the model is too small to use it.

The compute-optimal point sits at roughly a 20:1 ratio of training tokens to parameters. Earlier giants like GPT-3 were badly undertrained by this standard — Chinchilla implied they needed many times more data than they were given. By 2025, the industry had swung the other way: models like LLaMA-3 8B were deliberately “over-trained” at ratios above 1,800:1, because when you serve billions of inference requests, it pays to spend extra compute up front to get a smaller, cheaper-to-run model. The point is that the compute budget — an infrastructure question — directly determines the optimal model configuration and the loss it can reach.

There is a hard floor underneath all of this. Scaling curves do not head to zero; an irreducible loss term remains, so the marginal return on each additional unit of compute shrinks as you approach it. And the macro trend of scaling training compute roughly fourfold per year runs into four real-world ceilings by 2030: power availability, chip manufacturing capacity, data scarcity, and the latency wall of sequential computation. Infrastructure does not just enable scaling — it is where scaling eventually stops.

Infrastructure factor	Effect on training	Effect on inference
Compute (FLOPs)	Sets model size and data volume; defines the loss ceiling	Matters less per query; reasoning models raise demand
Memory bandwidth (HBM)	Feeds compute cores; underutilized memory wastes GPUs	Primary bottleneck — sets token generation speed
Networking (interconnect)	Critical past one node; slow links idle the GPUs	Governs multi-region and cross-cloud data movement
Power and cooling	Caps cluster size; grid access now the hard limit	Dominates lifetime cost since inference runs continuously

The interconnect bottleneck no one sees coming

Raw GPU count is the headline number, but it is rarely the real constraint. Once a training run spreads beyond a single node — moving from 8 GPUs to 64, 128, or thousands — the speed of the link between GPUs becomes the primary bottleneck. Without high-bandwidth, low-latency interconnect such as InfiniBand, RoCE, or NVIDIA’s Spectrum-X Ethernet fabrics, GPUs spend more time waiting for data from their peers than doing matrix multiplications. A cluster with world-class chips and a mediocre network can run at a fraction of its theoretical throughput.

This is why purpose-built AI systems treat networking as a first-class design decision, not an afterthought. Interconnects like NVLink handle GPU-to-GPU traffic inside a node, while InfiniBand and high-speed Ethernet move data between nodes. The same logic now extends across regions and clouds: by 2026, some analysts argue the binding constraint on large-scale AI is no longer GPU availability but how intelligently data moves between compute nodes, across regions, and between cloud and edge. The companies winning are not always the ones with the most GPUs — they are the ones that orchestrate data without creating bottlenecks.

The memory wall: why more GPUs don’t fix inference

Inference flips the bottleneck from networking to memory. When a model generates text token by token, the limiting factor is how fast the GPU can move weights and key-value cache data from memory to the compute units. This is the memory wall: adding more FLOPs does not help, because the chip is waiting on memory, not arithmetic. Latency per token is set by the slowest memory read in each step.

The numbers make this concrete. As of 2026, the AMD MI300X offers about 5.3 TB/s of HBM bandwidth, the NVIDIA H200 about 4.8 TB/s, and the B200 roughly 8 TB/s. For latency-sensitive serving, that bandwidth gap maps almost directly to how fast a model answers. Spreading a model across more GPUs reduces per-GPU memory pressure but adds communication overhead, and the per-token step stays memory-bound unless bandwidth scales with it. Engineers now diagnose this with memory-bandwidth utilization: when it sits above 80–90% while compute utilization lags, the system is memory-bound and needs better memory, not more GPUs.

HBM: the scarce component shaping the whole industry

That dependence on bandwidth has turned high-bandwidth memory into the critical bottleneck for AI hardware. HBM stacks DRAM chips vertically using through-silicon vias and advanced packaging, giving a 1024-bit interface that vastly outpaces conventional memory. The catch is supply. Micron and SK Hynix reported their entire 2026 HBM production sold out, and the memory crunch has been severe enough to force cuts in gaming GPU production while memory makers post record margins.

The trajectory is steep: NVIDIA’s Blackwell B200 carries 192 GB of HBM3E, a 140% jump from the H100’s 80 GB, and HBM4 entering production in 2026 targets roughly 2 TB/s of total bandwidth with a 2048-bit interface. Because moving parameters between memory and compute consumes more time and energy than the math itself, HBM availability now effectively gates how capable and how fast deployed models can be. The bottleneck has moved off the GPU die and onto the memory stacked beside it.

Inference is taking over the cost equation

The center of gravity in AI compute has shifted. Training happens once; inference runs continuously, every query consuming compute and power. Inference workloads are projected to reach about two-thirds of all AI compute in 2026, up from roughly half in 2025 and a third in 2023. More striking, inference can account for 80–90% of the lifetime cost of a production AI system. An organization that optimizes its infrastructure purely for training can find itself badly positioned once the model goes live and inference becomes the dominant, ongoing expense.

This reframes the infrastructure question for most companies. It is not “what is the biggest model we can train” but “what does it cost to serve, every second, at the latency our users tolerate?” That is an inference-economics question, and it is decided by memory bandwidth, GPU utilization, and how efficiently the serving stack is built. For a fuller picture of what a production-grade system actually requires across compute, networking, cooling, and orchestration, see our guide to premium AI infrastructure system requirements.

Software efficiency narrows the hardware gap

Infrastructure is not only about buying more hardware. A growing toolkit of optimization techniques lets a fixed system do more, partly decoupling performance from raw capacity. Quantization lowers numerical precision with minimal accuracy loss; pruning removes non-essential parameters; distillation produces smaller models that mimic larger ones; and KV-cache compression lets large models run on more modest GPUs. Newer architectures, such as hybrid designs that blend different layer types, push compute efficiency further while preserving accuracy.

The lesson is that efficiency now matters as much as raw capacity. Throwing more hardware at a problem is not a strategy when the real issue is poor GPU utilization, an undersized memory tier, or an interconnect that idles the chips. The organizations getting the most from their models are the ones treating compute, memory, networking, and software as a single coordinated system rather than a stack of separate purchases.

The bottom line

Model architecture defines the upper bound of what is possible, but infrastructure decides how much of that bound you actually reach. In training, the compute budget and the interconnect set how capable a model can become. In inference, memory bandwidth and HBM supply set how fast and how affordably it runs. Power and grid access increasingly cap the whole enterprise. As scaling approaches its physical limits and inference takes over the cost equation, the role of infrastructure in model performance is not shrinking — it is becoming the main event. The model is the engine; bandwidth, networking, and power are the drivetrain, and a drivetrain that cannot scale stops the whole machine.

If you are interested in this topic, we suggest you check our articles:

• What Is Required for a Premium AI Infrastructure System?
• How AI Factory Engineering Works and Top Sectors to Benefit
• How Much Electrical Power Does AI Require?
• European Countries With the Most Data Centers 2026
• OpenAI’s Infrastructure Push: Becoming Its Own Cloud Provider

Sources: Bessemer Venture Partners, Introl (Inference vs Training), Introl (Memory Supercycle), Spheron, TrendForce, EnkiAI, Brenndoerfer (Chinchilla), AIMultiple, Lyceum Technology

Written by Alius Noreika