AMD's Ryzen AI 300 Series Outperforms Intel in LLM Performance Benchmarks

AMD Ryzen AI 9 HX 375 outperforms Intel's Core Ultra 7 258V in LLM performance -- Team Red provided benchmarks show a strong lead of up to 27% in LM Studio

In some applications, Strix Point can deliver up to 3.5X lower latency than Lunar Lake. AMD claims that its Ryzen AI 300 (codenamed Strix Point) offerings can easily beat Intel's latest Core Ultra 200V (codenamed Lunar Lake) CPUs in consumer LLM workloads. Team Red has showcased several charts - comparing the performance of processors from both lineups - with AMD in front by upwards of 27%. Since the AI boom, tech giants -- both on the hardware and software fronts -- have entered an arms race to outpace each other in the AI landscape. While much of this is purely for the quote-unquote "AI hype," benefits for mainstream customers are starting to materialize. Microsoft demands a minimum of 40 TOPS for any system to be considered a "Copilot+ PC." Almost every CPU manufacturer now reserves valuable silicon space for a Neural Processing Unit (NPU). On that note, AMD says that its Strix Point APUs seemingly outperform Intel's Lunar Lake when running LLMs locally. LM Studio's typical consumer LLM is run based on the llama.cpp framework. The Ryzen AI 9 HX 375 leads the Core Ultra 7 258V by up to 27% in the CPU department. The latency benchmarks are an absolute bloodbath for Intel - with AMD reportedly delivering 3.5X lower latency in the Mistral Nemo 2407 12b Instruct model. When switching to the integrated graphics solution, the Ryzen AI 9 HX 375's Radeon 890M iGPU is, at best, 23% faster than Intel's Arc 140V. Interestingly, while the HX 375 is the fastest Strix Point APU, the Core Ultra 7 258V is not. The flagship Core Ultra 9 288V offers a full 30W TDP, which aligns it relatively better with Strix Point since the latter can be configured as high as 54W. That aside, AMD then tested both laptops in Intel's own and first-party AI Playground - where we see AMD in front by 13.1% using the Mistral 7b Instruct v0.3 model. The Microsoft Phi 3.1 Mini Instruct test is slightly less exciting as the lead drops to 8.7%. All in all, it is not every day that you fire up your laptop to run a local LLM, but who knows what the future holds? It would've been more exciting to see the kingpin Core Ultra 9 288V in action, but don't expect drastically different results. AMD says that it is actively working to make LLMs, currently gated behind many technical barriers, more accessible to everyone, which highlights the importance of projects like LM Studio.

AMD Crushes Intel Lunar Lake By 27% Performance in LLM Applications As Ryzen AI Showcases Its NPU & iGPU Bruteness

AMD's Strix Point APUs showcase a strong performance advantage in AI LLM workloads against Intel's Lunar Lake offerings. AMD Strix Point APUs show dominance in AI LLMs while reducing overall latency against competing Intel Lunar Lake SoCs The demand for higher performance in AI workloads has not only forced many companies to bring their own specialized hardware to the market but also made the competition more fierce. Since LLMs(large language models) have evolved significantly, the need for faster hardware is also increasing. To tackle this, AMD introduced its own AI-oriented processors for mobile platforms, known as Strix Point, a while back. In the latest blog post, the company claims that its Strix Point APUs can have a big lead over its rivals while decreasing the latency for quicker output. According to AMD, the Ryzen AI 300 processors can deliver higher Tokens per second than Intel's Lunar Lake chips, which are Intel's special mobile chips for AI workloads. As per the comparison, the Ryzen AI 9 HX 375 offers up to 27% higher performance in consumer LLM applications in LM Studio than the Intel Core Ultra 7 258V. The latter isn't the fastest in the Lunar Lake lineup, but it's surely close to the higher-end Lunar Lake CPUs since the core/thread count remains the same except for the core clocks. The LM Studio is AMD's consumer-friendly tool built on the llama.cpp that doesn't require its users to learn the technical side of the LLMs. Llama.cpp is a framework that is optimized for x86 CPUs and uses AVX2 instructions. While the framework doesn't need a GPU to run LLMs, it can surely be accelerated using a GPU. In the latency department, the Ryzen AI 9 HX 375 can deliver up to 3.5x lower latency than its rival and can achieve up to 50.7 tk/s vs 39.9 tk/s by Core Ultra 7 258V in Meta Llama 3.2 1b Instruct. As both Intel Lunar Lake and Strix Point APUs come with powerful integrated graphics, the LM Studio can offload the tasks to the iGPU to boost LLM performance using Vulkan API. Strix Point APUs bring powerful Radeon graphics based on the RDNA 3.5 architecture and can offer up to a 31% boost in performance for Llama 3.2. Furthermore, using the VGM(Variable Graphics Memory) Ryzen AI 300 processors can allow memory reallocation for iGPU-oriented tasks, enhancing power efficiency, and resulting in a solid 60% higher performance combined with the GPU acceleration. AMD said that to make the comparison fair, it also tested both CPUs in Intel AI Playground with the same settings and found that the Ryzen AI 9 HX 375 was up to 8.7% faster than Core Ultra 7 258V on Microsoft Phi 3.1 and up to 13% faster on Mistral 7b Instruct 0.3 model. Nonetheless, it would have been interesting to see the Ryzen AI 9 HX 375 go against the flagship Core Ultra 9 288V processor as the HX 375 is itself the fastest Strix Point CPU. Currently, AMD is focusing on making LLMs accessible to most users who don't possess technical skills and this can only be achieved using the LM Studio, based on the llama.cpp framework.

Ryzen AI 300 takes big wins over Intel in LLM AI performance -- up to 27% faster token generation than Lunar Lake in LM Studio

Strix Point runs the tables when it comes to local LLM performance AMD's Ryzen AI 300 series of mobile processors beats Intel's mobile competition handily at local large language model (LLM) performance, according to recent in-house testing by AMD. A new blog post [at the time of publication, this link was not active] from the company's community blog outlines the tests AMD performed to beat Team Blue in AI performance, and how to make the most of the popular LLM program LM Studio for any interested users. Most of AMD's tests were performed in LM Studio, a desktop app for downloading and hosting LLMs locally. The software, built on the llama.cpp code library, allows for CPU and/or GPU acceleration to power LLMs, and offers other control over the functionality of the models. Using the 1b and 3b variants of Meta's Llama 3.2, Microsoft Phi 3.1 4k Mini Instruct 3b, Google's Gemma 2 9b, and Mistral's Nemo 2407 12b models, AMD tested laptops powered by AMD's flagship Ryzen AI 9 HX 375 against Intel's midrange Core Ultra 7 258V. The pair of laptops were tested against each other measuring speed in tokens per second and acceleration in the time it took to generate the first token, which roughly match to words printed on-screen per second and the buffer time between when a prompt is submitted and when the LLM begins output. As seen in the graphs above, the Ryzen AI 9 HX 375 shows off better performance than the Core Ultra 7 258V across all five tested LLMs, in both speed and time to start outputting text. At its most dominant, AMD's chip represents 27% better speeds than Intel's. It is unknown what laptops were used for the above tests, but AMD was quick to mention that the tested AMD laptop was running slower RAM than the Intel machine -- 7500 MT/s vs. 8533 MT/s -- when faster RAM typically corresponds to better LLM performance. It should be noted that Intel's Ultra 7 258V processor is not exactly on a fair playing field against the HX 375; the 258V sits in the middle of Intel's 200-series SKUs, with a max turbo speed of 4.8 GHz versus the HX 375's 5.1 GHz. AMD's choice to pit its flagship Strix Point chip against Intel's medium-spec chip reads as a bit unfair, so take the 27% improvement claims with that in mind. AMD also showed off LM Studio's GPU acceleration features in tests showing off the HX 375 against itself. While the dedicated NPU in Ryzen AI 300-series laptops is meant to be the driving force in AI tasks, on-demand program-level AI tasks are more prone to use the iGPU. AMD's tests with GPU acceleration using the Vulkan API in LM Studio so heavily favored the HX 375 that AMD did not include Intel's performance numbers with GPU acceleration turned on. With GPU acceleration on, the Ryzen AI 9 HX 375 saw up to 20% faster tk/s than when it ran tasks without GPU acceleration. With so much current press around computers based on AI performance, vendors are eager to prove that AI matters to the end user. Apps like LM Studio or Intel's AI Playground do their best to offer a user-friendly and foolproof way to harness the latest 1 billion+ iteration LLMs for personal use. Whether large language models and getting the best out of your computer for LLM use matters to most users is another story.

Ryzen AI 300 outperforms Intel in AI workloads: AMD

With AI becoming integrated into everyday use, many people are exploring the possibility of running Large Language Models (LLMs) locally on their laptops or desktops. For this purpose, many are using LM Studio, a popular software, based on the llama.cpp project, which has no dependencies and can be accelerated using only the CPU. Capitalizing on the popularity of LM Studio, AMD showcased performance gains for LLMs using its latest Ryzen AI processors. With the new AMD Ryzen AI 300 Series processors, the company promises state-of-the-art workload acceleration and industry-leading performance in llama.cpp-based applications like LM Studio for x86 laptops. For the tokens per second (tk/s) metric, which denotes how quickly an LLM can output tokens (response), Intel Core Ultra 7 and Ryzen AI 9 HX 375 processors were compared. AMD achieves up to 27% faster performance in this comparison. Further, AMD also outperformed the competitor by 3.5x time in time to the first token benchmark. AMD Ryzen AI processors utilize AMD XDNA 2 architecture-based NPUs for persistent AI tasks like Copilot+ workloads, while relying on the iGPU for on-demand AI tasks. The Ryzen AI 300 Series processors also include a feature called Variable Graphics Memory (VGM), which will utilize the 512 MB block of dedicated allocation for an iGPU plus the second block of memory that is housed in the "shared" portion of system RAM. This combination of iGPU acceleration and VGM offers a significant average performance increase on consumer LLMs based on vendor-agnostic Vulkan API. After turning on VGM (16GB), we saw a further 22% average uplift in performance in Meta Llama 3.2 1b Instruct for a net total of 60% average faster speeds, compared to the CPU, using iGPU acceleration when combined with VGM, said the company. AMD also outshined Intel's iGPU performance which used the first-party Intel AI Playground application, showing 8.7% faster performance with Microsoft Phi 3.1 Instruct and 13% faster in Mistral 7b Instruct 0.3. Regarding the matter, AMD says,

AMD AI 9 HX 375 processor generates tokens 27% faster than Intel Core Ultra 7 258V in LM Studio

AMD has announced significant advancements in its Ryzen AI 300 series mobile processors, specifically highlighting the Ryzen AI 9 HX 375. In performance tests conducted using LM Studio -- a desktop application designed for downloading and hosting large language models (LLMs) based on llama.cpp and utilizing CPU acceleration through AVX2 instructions -- the Ryzen AI 9 HX 375 demonstrated a 27% increase in token generation speed compared to Intel's Core Ultra 7 258V processor. The benchmarks employed various models, including Meta Llama 3.2 (1b and 3b), Microsoft Phi 3.1 4k Mini Instruct 3b, Google Gemma 2 9b, and Mistral Nemo 2407 12b. It's noteworthy that the Intel platform utilized memory operating at 8533 MT/s, whereas the AMD platform operated at 7500 MT/s, potentially influencing the comparative results. The performance evaluation revealed that the Ryzen AI 9 HX 375 consistently outperformed the Intel Core Ultra 7 258V across all tested large models. This superiority was evident not only in the speed of token generation but also in the latency before the commencement of text output, with the Ryzen processor maintaining a maximum advantage of 27%. However, it's important to contextualize these results by considering the product tiers of the processors involved. The Intel Core Ultra 7 258V is positioned as a mid-to-high-end offering, whereas the Ryzen AI 9 HX 375 represents a flagship model within AMD's lineup. This distinction suggests that the performance disparity may partially stem from the differing market segments these processors target. Additionally, LM Studio supports GPU acceleration through the Vulkan API, which, when enabled, further enhances performance. Specifically, enabling GPU acceleration resulted in a 20% performance boost for the Ryzen AI 9 HX 375 compared to its CPU-only configuration. This capability underscores the processor's versatility and potential for optimized performance in environments that leverage both CPU and GPU resources. Overall, the Ryzen AI 9 HX 375's performance in LM Studio benchmarks positions it as a robust option for applications requiring efficient handling of large language models, though comparisons should account for the differing specifications and market positioning of the processors involved.