Microsoft Scales Up AI Infrastructure with Custom Clea Maia 300 Accelerators to Tackle the Global Inference Bottleneck

As the generative AI revolution matures, the industry has pivoted from the initial scramble for raw training power to a more complex and economically pressing challenge: the inference problem. Even with the current generation of state-of-the-art GPU and XPU accelerators, the cost of generating tokens remains high, creating a significant barrier to the widespread, sustainable deployment of large language models. For Microsoft, which operates as both a primary host for OpenAI’s massive ChatGPT models and a provider of AI-infused productivity tools for millions of enterprise users, this inference problem is perhaps the most acute in the world.

The scale of Microsoft’s internal demand is staggering. Beyond the massive compute requirements of training and running OpenAI’s foundation models, Microsoft is rapidly integrating AI routines into every corner of its worker productivity and back-office applications. This integration is driving usage up a "hockey stick" growth curve, necessitating a hardware strategy that moves faster than the traditional commercial silicon cycle. To meet this demand, Microsoft is accelerating the development of its homegrown Maia family of custom XPU accelerators. At this week’s Open Compute Project (OCP) Summit 2026, the company provided a detailed look at the "Clea" Maia 300, a third-generation accelerator designed to dramatically expand the scale and efficiency of AI inference on the Azure cloud.

The Maia 300 represents a significant architectural leap over its predecessors, specifically tailored to handle the emerging complexity of "agentic" workloads. These workloads often involve dozens or even hundreds of models working in concert—whether through a Mixture of Experts (MoE) foundation model architecture, physically distinct independent models, or a hybrid of both. For these systems to function with low latency, they require a massive, shared network and memory space, a requirement that the Maia 300 is built to address directly.

Microsoft’s journey into custom silicon has been a rapid evolution. The company first entered the space with the "Athena" Maia 100 in November 2023, alongside its custom Arm-based Cobalt 100 server CPU. While the Maia 100 served as a foundational step, its successor, the "Braga" Maia 200—unveiled only in January of this year—marked a more serious entry into production. The Maia 200 has been ramping up within Azure data centers throughout 2026, already handling inference for Copilot and supporting the development of new models. According to Scott Guthrie, Executive Vice President of Cloud and AI at Microsoft, the Azure Maia program has become a cornerstone of the company’s infrastructure strategy. By the end of this year, GitHub Copilot and the Microsoft Foundry are expected to transition their workloads entirely to Maia 200 systems, which also support various open-source and proprietary OpenAI models.

The strategic ambition behind the Maia program is reflected in its scale. Industry reports suggest Microsoft aims to deploy at least one million Maia sockets in the field. Guthrie has noted that Maia will eventually support "multi-gigawatt" deployments on Azure. To put that in perspective, a single gigawatt of Maia 300 capacity would encompass roughly 575,000 devices, considering the power draw of the accelerator alongside its associated host, storage, and networking components. Microsoft has disclosed that a single rack of Maia 300s, containing 96 interlinked XPUs, consumes approximately 167 kilowatts.

The transition from the Maia 200 to the Maia 300 involves a fundamental shift in both manufacturing and architecture. While the Maia 100 and 200 were monolithic designs produced on TSMC’s 5-nanometer and 3-nanometer processes respectively, the Clea Maia 300 moves to a sophisticated chiplet architecture utilizing TSMC’s 2-nanometer N2P process. The Clea package is less a single chip and more a "system in a socket," comprising seven distinct chips and eight High Bandwidth Memory (HBM) stacks on an interposer. By breaking out I/O and HBM controllers from the compute and SRAM cache elements, Microsoft can optimize each component individually. For instance, while the compute tiles leverage the cutting-edge 2nm process, the I/O and memory controller dies likely utilize a more mature 5nm process to maintain yield and performance without unnecessary cost.

The internal organization of the Maia 300 consists of four tile clusters, with each cluster containing twelve compute tiles. These tiles feature a Tile Tensor Unit (TTU) and a Tile Vector Processor (TVP), supported by a Tile Data Movement Engine (TDMA) and a Tile Control Processor (TCP). Microsoft has also integrated two specific elements first introduced in the Maia 200: a reshaper and a tile synchronizer. These components are critical for managing dataflow across the entire system, ensuring that information moves efficiently from the network down to the individual compute tiles.

Memory architecture is another area where the Maia 300 pushes the envelope. The device features 288 GB of HBM4, likely sourced from SK Hynix, providing roughly 18 TB/sec of bandwidth. This is balanced by a massive aggregate SRAM bandwidth of approximately 166.2 TB/sec across the complex. Interestingly, Microsoft has adopted a "NUMA-ish" memory approach for Clea. Instead of sharing all HBM memory across all compute tiles globally, pairs of compute tiles are tied to specific HBM4 memory banks. This localized memory access is designed to accelerate specific routines essential for generative AI inference, a design choice also mirrored in other high-end AI accelerators like OpenAI’s rumored "Jalapeno" chip.

The networking capabilities of the Maia 300 are equally ambitious. Microsoft has introduced the second generation of its custom AI Transport Layer (ATL 2.0), a custom RoCE (RDMA over Converged Ethernet) implementation. The networking strategy for Maia has evolved with every generation. The original Maia 100 used 400 Gb/sec links to create a coherency domain of 2,304 XPUs. The Maia 200 expanded this to 6,144 devices. With the Maia 300, Microsoft is leveraging a more efficient 1U form factor and advanced liquid cooling to double the density of its racks.

A standard Maia 300 rack now houses 96 XPUs across two dozen servers, compared to the 48 XPUs found in Maia 200 racks. By lashing 128 of these racks together, Microsoft can create a single coherency domain of 12,288 accelerators. This is currently the largest coherency domain reported by any major cloud provider, significantly exceeding the 72 or 144 GPU domains offered by Nvidia and AMD in their rack-scale systems. Even Google’s forthcoming "Sunfish" TPU 8t training chip is expected to top out at 9,600 units in a single domain. This massive scale is intended to allow the Maia 300 system to behave as a single, giant computer, capable of handling the most complex agentic AI workflows with minimal latency.

The performance gains from these architectural improvements are substantial. According to Microsoft’s hardware leadership, including Rani Borkar and Saurabh Dighe, a rack of Maia 300s can deliver up to seven times the token throughput of a Maia 200 rack. While a portion of this gain comes from doubling the number of XPUs per rack and a 36 percent increase in raw floating-point performance per chip, the majority of the improvement is attributed to the refined architecture and improved dataflow.

Crucially, Microsoft is achieving these performance leaps while focusing on power efficiency. The company reports a 2.5-fold improvement in performance-per-watt when moving from the Maia 200 to the Maia 300. In an era where data center power availability is becoming a primary constraint for AI expansion, the ability to extract significantly more work from the same energy footprint is as important as the raw speed of the chips themselves.

The development of the Maia line also highlights the shifting alliances in the semiconductor industry. While Global UniChip (GUC) of Taiwan acted as the "chip shepherd" for the Maia 100, rumors have swirled about the involvement of other major players like Broadcom and Marvell for subsequent generations. Microsoft’s decision to pull the host server—presumably powered by its 132-core Cobalt 200 series Arm CPUs—directly into the Maia 300 node suggests a move toward deeper system-level integration.

By engineering its own silicon, networking, and cooling systems, Microsoft is attempting to bypass the supply constraints and high margins of the commercial GPU market. The Clea Maia 300 is not merely a play for "speeds and feeds" dominance; it is a calculated effort to industrialize AI inference at a scale that can support the next generation of agentic software. As these systems move into multi-gigawatt deployments, the efficiency and architectural advantages of custom silicon will likely become the deciding factor in which cloud platforms can afford to lead the AI era.