Nvidia Pushes for 100M IOPS SSDs to Eliminate AI GPU Bottlenecks

SMI CEO claims Nvidia wants SSDs with 100M IOPS -- up to 33X performance uplift could eliminate AI GPU bottlenecks

Now that the AI industry has exceptionally high-performance GPUs with high-bandwidth memory (HBM), one of the bottlenecks that AI training and inference systems face is storage performance. To that end, Nvidia is working with partners to build SSDs that can hit random read performance of 100 million input/output operations per second (IOPS) in small-block workloads, according to Wallace C. Kuo, who spoke with Tom's Hardware in an exclusive interview. "Right now, they are aiming for 100 million IOPS -- which is huge," Kuo told Tom's Hardware. Modern AI accelerators, such as Nvidia's B200, feature HBM3E memory bandwidth of around 8 TB/s, which significantly exceeds the capabilities of modern storage subsystems in both overall throughput and latency. Modern PCIe 5.0 x4 SSDs top at around 14.5 GB/s and deliver 2 - 3 million IOPS for both 4K and 512B random reads. Although 4K blocks are better suited for bandwidth, AI models typically perform small, random fetches, which makes 512B blocks a better fit for their latency-sensitive patterns. However, increasing the number of I/O operations per second by 33 times is hard, given the limitations of both SSD controllers and NAND memory. In fact, Kioxia is already working on an 'AI SSD' based on its XL-Flash memory designed to surpass 10 million 512K IOPS. The company currently plans to release this drive during the second half of next year, possibly to coincide with the rollout of Nvidia's Vera Rubin platform. To get to 100 million IOPS, one might use multiple 'AI SSDs.' However, the head of SMI believes that achieving 100 million IOPS on a single drive featuring conventional NAND with decent cost and power consumption will be extremely hard, so a new type of memory might be needed. "I believe they are looking for a media change," said Kuo. "Optane was supposed to be the ideal solution, but it is gone now. Kioxia is trying to bring XL-NAND and improve its performance. SanDisk is trying to introduce High Bandwidth Flash (HBF), but honestly, I don't really believe in it. Right now, everyone is promoting their own technology, but the industry really needs something fundamentally new. Otherwise, it will be very hard to achieve 100 million IOPS and still be cost-effective." Currently, many companies, including Micron and SanDisk, are developing new types of non-volatile memory. However, when these new types of memory will be commercially viable is something that even the head of Silicon Motion is not sure about.

Kioxia introduces high-IOPS SSDs and long-term flash strategies for emerging AI and enterprise storage demands

New drive uses XL-Flash, a type of SLC NAND, and a new in-house controller Kioxia has unveiled plans for a new SSD it says could hit an impressive 10 million IOPS, a level of performance aimed squarely at the demands of AI-driven systems. The SSD will use XL-Flash, a type of single-level cell (SLC) NAND, combined with a new in-house controller. A Kioxia spokesperson told TechPowerUp, "We're taking our ultra-fast XL-Flash memory chips, which use single-level cells, and pairing them with a completely new controller... We're targeting over 10 million IOPS, and we plan to have samples ready by the second half of 2026." IOPS, or input/output operations per second, measures how quickly a storage device can handle small, random requests, particularly important in AI and server applications where fast access to small files is key. This is different from GBps, which refers to the actual data transfer speed and is used to measure how fast large files can be read or written. A drive with high GBps might excel in video editing or large file transfers, but for machine learning tasks where thousands of small data packets are read or written constantly, high IOPS matters more. Kioxia's approach to next-gen storage includes not just one-off projects but a wider effort to meet varied use cases. Its CM9 series, which is sampling to customers now, focuses on speed and reliability to match high-end GPUs used in AI, while the LC9 series delivers massive 122TB capacities for large databases. Behind these products is the 8th generation BiCS FLASH, which introduces CBA tech to boost performance and efficiency. Kioxia is also preparing future flash memory generations using two methods. The first will add more layers for capacity, while the second blends new CMOS designs with older cell structures to keep investment costs in check.