Microsoft challenges Nvidia dependence with new in-house Maia 300 AI chip; All details

Microsoft announced its third-generation Maia AI accelerator, Maia 300, on October 12 to reduce dependence on Nvidia GPUs. The chip is designed to run trained AI models, and the company plans to use the chip to support AI features and services across its ecosystem, including cloud-based AI workloads.

Maia 300 AI chip is said to be optimised for inference, a process through which an AI model responds to prompts, generates text, writes code, or performs other tasks. Microsoft plans to gain greater control over its computing infrastructure and reduce its dependence on third-party GPU suppliers such as Nvidia.

Must read: Microsoft President calls for greater safety oversight, says ‘even milk has safety inspection’

Microsoft CEO Satya Nadella shared an X post announcing the Maia 300 and claimed that it can generate 40% more AI tokens for the same amount of electricity as leading GPU-based systems. 

“Starting next calendar year, we will scale our Maia family to multi-gigawatt deployments, making it the primary inference system for our own products and services,” Nadella said.

“And we’re introducing Maia 300, which delivers 40% more tokens per watt compared to leading GPU-based systems.” He further added that Microsoft has started deploying its Maia AI chips in data centres across the US, Europe and Australia. “Our goal is to build on the full-stack efficiency gains we’ve already seen, while continuing to bring the latest innovations from across the ecosystem into our fleet,” he said.

Maia 300 AI chip: Key capabilities

Microsoft has designed its Maia 300 AI chip primarily to run AI models efficiently that can manage large amounts of information and generate lengthy responses. With the chip, Microsoft is aiming to balance “compute, memory bandwidth, networking, and power efficiency.”

Its capabilities include reducing the time and energy spent moving data between computing components and moving data between memory and processing units at high speeds while managing power consumption. Its eight HBM4 stacks provide peak bandwidth of 18 TB/s, supporting demanding AI workloads involving large amounts of data.

Must read: Microsoft’s Nvidia-powered Surface Ultra laptop put a price on its hybrid AI ambitions: All details

Maia 300 uses a multi-chiplet architecture that separates computing, memory control and input/output functions. Its computing chiplets are manufactured using TSMC's N2P process, with a focus on improving efficiency and scalability.

Maia 300 is designed to help multiple AI accelerators communicate and share workloads efficiently across data centres. Microsoft said its high-speed interconnects and custom Ethernet-based transport system allow deployments to scale from individual racks to larger data-centre systems.

Microsoft's Maia 300 rack systems pack 96 accelerators into a 167-kilowatt power envelope, with 27.6 TB of combined HBM capacity and 1.7 PB/s of aggregate HBM bandwidth. The company said the configuration is designed to run more AI workloads simultaneously and increase computing capacity within each rack.

While Microsoft is shifting to Maia infrastructure, it will continue to use Nvidia and AMD chips for its resources. However, the new AI chip could potentially compete with third-party GPU suppliers.