Competitors Close in on Nvidia as AI Inference Market Heats Up with New Power-Efficient Chips

Competitors Close in on Nvidia as AI Inference Market Heats Up with New Power-Efficient Chips | Nvidia's Blackwell chip was submitted to MLPerf for the first time, showing the best per GPU performance in the LLM Q&A benchmark.
Image credit: Nvidia

At the recent IEEE Hot Chips conference, Cerebras and Furiosa announced their latest AI inference chips. Cerebras introduced an upgraded version of its CS3 platform, which it claims outperforms Nvidia's H100 by 7x in LLM tokens generated per second. 

Meanwhile, Furiosa unveiled its second-generation chip, RNGD, featuring a unique Tensor Contraction Processor (TCP) architecture. Although neither company participated in the latest MLPerf Inference v4.1 competition, they highlighted their chips' capabilities in-house, with Furiosa's RNGD reportedly matching Nvidia's L40S chip in performance while using significantly less power.

MLPerf Inference v4.1 Results: Nvidia Maintains Lead, Competitors Show Promise

ML Commons recently released the results of the MLPerf Inference v4.1 competition, which revealed that Nvidia continues to dominate the AI inference landscape with its new Blackwell chip. However, competitors are gaining ground, especially in terms of power efficiency. Submissions included entries from AMD, Google, and Untether AI, with Nvidia's Blackwell outperforming all previous iterations on a per-accelerator basis in the LLM Q&A task. 

Untether AI’s speedAI240 Preview chip also performed impressively, nearly matching Nvidia’s H200 in image recognition.

Untether AI's Innovative Approach to Power Efficiency

Untether AI emerged as a strong contender in the competition, particularly in the power efficiency category. The startup's chips, designed with an "at-memory computing" approach, significantly reduce the energy required for AI workloads by minimizing data movement between memory and processing units. 

This method allowed Untether AI to outperform Nvidia's L40S chip in latency and throughput for image recognition tasks. Untether AI's speedAI240 Preview chip also demonstrated a substantial reduction in power consumption, using only 150 Watts compared to Nvidia's 350 Watts, resulting in a 2.3x power reduction with improved performance.

The Power of Nvidia's Blackwell Chip

Nvidia's new Blackwell chip showcased its impressive capabilities in the latest MLPerf Inference competition. The chip's success is attributed to its ability to operate using 4-bit floating-point precision, a first for MLPerf benchmarks. 

Nvidia GB2800 Grace Blackwell Superchip
Image credit: Nvidia

This innovation, combined with doubled memory bandwidth of 8 terabytes/second, allowed Blackwell to excel in the LLM Q&A task. Nvidia's submission of the Blackwell chip was in the preview category, indicating that it is not yet available for sale but is expected to be released within the next six months.

Growing Competition in the AI Inference Market

As AI inference continues to gain importance, the competition among chipmakers is intensifying. Nvidia remains the leader, but companies like Untether AI, Cerebras, and Furiosa are closing in with innovative approaches to power efficiency and performance. 

The AI inference chip market is set to become even more dynamic as these competitors continue to push the boundaries of technology, offering new options for various AI applications.