The Intensifying Competition in LLM Model Size: A Shift Towards Smaller, More Efficient Models

'LLM Model Size Competition is Intensifying, Backwards'

The era of small language models has just begun. "LLM model size competition is intensifying... backwards!" quipped OpenAI cofounder Andrej Karpathy, reflecting on the recent announcements about GPT-4o mini and other small language models (SLMs). This week was notable for the release of several SLMs. OpenAI released GPT-4o mini, a highly cost-efficient model designed to expand AI applications by making intelligence more affordable. Priced at $0.15 per million input tokens and $0.6 per million output tokens, GPT-4o mini is 30x cheaper than GPT-40 and 60% cheaper than GPT-3.5 Turbo. On the other hand, Paris-based AI startup Mistral AI, in collaboration with NVIDIA, released Mistral NeMo, a 12 billion parameter model with a 128k token context length. Benchmarked against Gemma 2 9B and Llama 3 8B, it excels in reasoning, world knowledge, and coding accuracy. Mistral NeMo is particularly effective in English, French, German, Spanish, Italian, Portuguese, Chinese, Japanese, Korean, Arabic, and Hindi. Not only that, Mistral also launched MathΣtral, a specialised 7B model for advanced mathematical reasoning and scientific exploration, A few days ago, Hugging Face also released a new series of compact language models called SmolLM, available in three sizes: 130M, 350M, and 1.7B parameters. These models are perfect for use on local devices such as laptops and phones, eliminating the need for cloud-based resources and significantly reducing energy consumption. Similarly, H2O.ai introduced H2O-Danube3, a new series of SLMs to bring AI capabilities to mobile devices. The series includes two models: H2O-Danube3-4B, trained on 6 trillion tokens, and H2O-Danube3-500M, trained on 4 trillion tokens. Apple too entered the game, releasing a 7B model called DCLM-BASELINE 7B, along with its weights, training code, and dataset. Trained on 2.5 trillion tokens from open datasets, the model primarily uses English data and features a 2048-token context window. In China, Alibaba released Qwen's Base and Instruct models in five sizes, including Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B, and Qwen2-72B, trained on data in 27 additional languages besides English and Chinese. SLMs or specialised models are going to be the future, alongside generalised models like GPT-4 or Claude 3.5 Sonnet. "For everyday use, an 8B or maybe 70B LLM will suffice. If you don't want to test a model to the max, you don't need a SOTA model. For everyday questions, which now appear in all training data, smaller models are sufficient," posted a user on X. "Both specialised and generic models will coexist, one is not a replacement for the other. It's the wrong dream to believe we only need one API such as OpenAI. We will need both mega models aiming for AGI and specialised micro models that can integrate into today's workflows," said Pramod Varma, chief architect of Aadhaar, in a recent interaction with AIM. In India, Gnani.ai recently launched a series of voice-first SLMs, which are trained on millions of audio hours of proprietary audio datasets and billions of Indic language conversations, capturing the rich diversity of dialects, accents, and linguistic nuances prevalent across the country. Abacus.AI chief Bindu Reddy has predicted that in the next five years, smaller models will become more efficient, LLMs will continue to become cheaper to train, and LLM inference will become widespread. "We should expect to see several Sonnet 3.5 class models that are 100x smaller and cheaper in the next one to two years." Earlier this year, Microsoft unveiled Phi-3-Mini, a 3.8 billion parameter language model trained on an extensive dataset of 3.3 trillion tokens. Despite its compact size, the Phi-3-Mini boasts performance levels that rival larger models such as Mixtral 8x7B and GPT-3.5. On similar lines, Karpathy explained that the reason current models are so large is that the training process is still quite wasteful: "We're asking them to memorise the internet, and, remarkably, they do and can, e.g., recite SHA hashes of common numbers or recall really esoteric facts." "But imagine if you were going to be tested, closed book, on reciting arbitrary passages of the internet given the first few words. This is the standard (pre)training objective for models today," he added. He noted that the ability to demonstrate thinking (e.g., reasoning) is closely tied to the amount of knowledge the model has learned from the training data. He further explained that in the future, as larger models help refine and optimise the training process, smaller models will emerge. "The models have to first get larger before they can get smaller because we need their (automated) help to refactor and mould the training data into ideal, synthetic formats." The constant use of LLMs also has its drawbacks, including high inference costs and energy consumption. SLMs, such as Gemma or LLaMA 8B, are very cost-effective as they require less compute and their inferences consume less energy. OpenAI used to spend around $700,000 daily to run ChatGPT. With GPT-4o mini, that cost may decrease to some extent. Moreover, since SLMs can run locally on devices without requiring the cloud, they don't need extensive AI infrastructure and GPUs. Compared to Claude 3.5 Sonnet and Google's Gemini 1.5 Pro, OpenAI's GPT-4o mini is significantly cheaper. According to Hugging Face, GPT-4o mini is the least expensive, with a combined input and output cost of just $0.75 per million tokens. Ola Krutrim chief Bhavish Aggarwal recently pointed out that a single H100 NVIDIA GPU consumes 30 times more electricity in a year than an Ola S1 scooter. "An H100 GPU requires around 8.7 MWh of energy per year, whereas an S1 requires 0.25 MWh/year. Need a lot of electricity in the future!" he said. However that is now changing as Karpathy recently demonstrated that today one can train GPT-2 on their own for ~$672, running on one 8XH100 GPU node for 24 hours. "Incredibly, the costs have come down dramatically over the last five years due to improvements in compute hardware (H100 GPUs), software (CUDA, cuBLAS, cuDNN, FlashAttention) and data quality (e.g. the FineWeb-Edu dataset)," he said. On the other hand, TWO AI chief Pranav Mistry claimed that the future is in 1-bit LLMs as they have multiple advantages, meaning that they can fit on smaller chips, require less memory, and have faster processing. One question haunts enterprises: the balance between the impressive capabilities of SLMs and their potential drawbacks compared to private instances of large models, and whether small models represent the future of generative AI. "Technology is rapidly evolving how we operate and train. Three months ago, using an SLM isolated in a customer's private database might have seemed like a drawback compared to a private instance of a large public model," said Upstage's Kasey Roh, on the sidelines of the recent AWS Summit in Washington DC. The South Korean company has built an SLM called Solar, one of the top-ranked models on the HuggingFace Open LLM leaderboard, and a finetune of Llama 2. Further, Roh said that to address the challenges of SLMs, the company has now entered into a subscription model with its customers, allowing them to continuously train and fine-tune models with the latest data points. She believes that this modular and standardised approach significantly mitigates the drawbacks of using small private models. "Once we have a more complete set of the tools and programmes available, I think that the drawback of having a small model that's private can be largely addressable," concluded Roh.

New GPT-4o Mini vs Claude 3 AI model performance tested

OpenAI's release of GPT-4o Mini this week marks a significant milestone in the AI industry. This new model is not only cost-effective but also features impressive performance metrics, making it a catalyst for various applications. Priced at just 15 cents per input token and 60 cents per million output tokens, GPT-4o Mini is a staggering 60% cheaper than its predecessor, GPT 3.5 Turbo. But how does it compare to other high-performance AI models? In this quick overview we will compare GPT-4o Mini vs Claude 3. This remarkable affordability opens up exciting new possibilities for businesses and developers looking to integrate advanced AI capabilities into their products and services without the burden of exorbitant costs. We will also take a look at what else OpenAI is currently developing to push artificial intelligence (AI) forward in the near future in the ultimate goal of Artificial General Intelligence (AGI). GPT-4o Mini's performance is nothing short of exceptional. It scores an impressive 82% on the MMLU benchmark, surpassing even the highly regarded GPT 4 in chat preferences. This high score is a testament to its ability to handle a wide range of tasks with remarkable efficiency and accuracy. Whether it's natural language processing, text generation, or complex problem-solving, GPT-4o Mini consistently delivers top-notch results. One of the standout features of GPT-4o Mini is its multimodal support. The model currently supports both text and vision in the API, with plans to expand its capabilities to include image, video, and audio inputs and outputs in the near future. This versatility makes it an ideal choice for a wealth of applications, such as: The possibilities are truly endless, and as GPT-4o Mini continues to evolve, it will undoubtedly unlock even more innovative use cases across various domains. When stacked up against other prominent AI models like Gemini Flash, Claude Haiku, and GPT 3.5 Turbo, GPT-4o Mini truly shines. It not only matches but often surpasses the capabilities of these models across various benchmarks and performance metrics. This exceptional performance, coupled with its unbeatable cost-effectiveness, positions GPT-4o Mini as the go-to choice for organizations seeking to leverage the power of AI without breaking the bank. Moreover, GPT-4o Mini's ability to chain or parallelize multiple calls significantly reduces latency, making it an ideal solution for real-time applications that demand swift and seamless responses. This feature further enhances its utility and practicality in a wide range of scenarios, from interactive virtual assistants to dynamic content generation systems. As the AI landscape continues to evolve at a breakneck pace, GPT-4o Mini sets a new standard for cost-effective, high-performance AI models. Its introduction is poised to reshape the industry, making advanced AI capabilities more accessible and affordable than ever before. With GPT-4o Mini leading the charge, the future of AI looks brighter and more promising than ever. The introduction of GPT-4o Mini is not just a milestone for OpenAI; it has far-reaching implications for the entire AI industry. Its affordability and exceptional performance are likely to accelerate the adoption of AI technologies across various sectors, from healthcare and finance to manufacturing and entertainment. As more businesses recognize the immense potential of integrating cost-effective AI solutions like GPT-4o Mini into their operations, we can expect to see a surge in AI investments and a rapid acceleration of development timelines. Experts predict that the advancements driven by models like GPT-4o Mini could lead to significant breakthroughs in AI capabilities by 2027-2028. This accelerated progress has the potential to transform industries, streamline processes, and unlock new frontiers of innovation. As AI becomes more deeply integrated into the fabric of our society, it will undoubtedly shape the way we live, work, and interact with technology. Here are a selection of other articles from our extensive library of content you may find of interest on the subject of OpenAI's new ChatGPT-4o Omni However, the implications of GPT-4o Mini and other advanced AI models extend beyond just technological progress. There are also important geopolitical considerations at play. Governments around the world, particularly in the United States, are grappling with the need to develop comprehensive AI strategies and policies. Discussions about potential AI executive orders and initiatives are gaining momentum, with a focus on harnessing AI for national security and maintaining technological leadership on the global stage. For instance, reports suggest that allies of former U.S. President Donald Trump are drafting plans for AI military technology development. These initiatives underscore the growing recognition of AI's strategic importance and its potential to reshape the balance of power in the international arena. As nations compete to gain an edge in AI capabilities, the geopolitical landscape is likely to undergo significant shifts, with far-reaching consequences for global security and technological dominance. OpenAI, the company behind GPT-4o Mini, is not content to rest on its laurels. Even as it celebrates the success of this groundbreaking model, the company is already looking ahead to the next frontier of AI innovation. One of the key areas of focus for OpenAI is the development of specialized AI chips that can optimize the performance of its models while further reducing costs. Reports indicate that OpenAI is actively exploring collaborations with leading technology companies like Broadcom to create custom AI chips tailored to its specific needs. By designing hardware that is optimized for running AI models, OpenAI aims to push the boundaries of what is possible with AI, delivering even more impressive performance gains and cost savings. In addition to its hardware initiatives, OpenAI has also been strategically hiring top talent from industry giants like Google. The company has brought on board former members of Google's AI chip development team, signaling its serious commitment to advancing its own AI hardware capabilities. With a team of experienced professionals at the helm, OpenAI is well-positioned to make significant strides in the development of innovative AI chips that can power the next generation of AI models. As OpenAI continues to innovate and push the envelope, it is clear that the company is not content with merely incremental improvements. Instead, it is setting its sights on transformative breakthroughs that have the potential to redefine the AI landscape. With GPT-4o Mini as a testament to its capabilities and ambition, OpenAI is poised to play a leading role in shaping the future of AI technology. The rapid advancements exemplified by GPT-4o Mini are part of a broader trend of AI's growing influence and impact on various aspects of society. From healthcare and finance to transportation and manufacturing, AI is becoming an indispensable tool for driving efficiency, innovation, and growth. As AI capabilities continue to expand and mature, we can expect to see even more profound changes in the way we live and work. One of the key trends shaping the future of AI is the increasing collaboration and competition between the public and private sectors. Governments around the world are recognizing the strategic importance of AI and are investing heavily in research and development initiatives. At the same time, private companies like OpenAI are pushing the boundaries of what is possible with AI, driving innovation and setting new standards for performance and affordability. This dynamic interplay between public and private sector efforts is likely to accelerate the pace of AI advancements, leading to breakthroughs that have the potential to transform entire industries and reshape the global economy. As AI becomes more deeply integrated into the fabric of society, it will also raise important questions about the ethical and societal implications of these technologies. Issues such as job displacement, privacy concerns, and the potential for AI to perpetuate biases and inequalities will need to be carefully addressed as we navigate this new era of AI-driven change. Policymakers, industry leaders, and civil society organizations will need to work together to develop responsible and inclusive approaches to AI development and deployment, ensuring that the benefits of these technologies are widely shared and that potential risks are mitigated. One of the most exciting and transformative applications of AI models like GPT-4o Mini is in the realm of creative industries. From entertainment and media to art and design, AI is opening up new possibilities for content creation and expression. With its ability to generate high-quality text, images, and potentially even video and audio, GPT-4o Mini is poised to become a powerful tool in the hands of creative professionals and enthusiasts alike. In the entertainment industry, AI-generated content is already making waves, allowing solo creators and small teams to produce compelling and engaging media that rivals the output of larger studios. By leveraging the power of AI, these creators can streamline their workflows, generate new ideas, and bring their visions to life in ways that were previously unimaginable. The democratization of content creation enabled by AI tools like GPT-4o Mini has the potential to reshape the entertainment landscape, fostering a more diverse and vibrant ecosystem of creative voices. As AI continues to advance, we can expect to see even more innovative and boundary-pushing content emerge, blurring the lines between human and machine creativity. Beyond the entertainment industry, AI is also transforming the world of art and design. GPT-4o Mini's multimodal capabilities, including its support for image and potentially video generation, open up new avenues for artistic expression and experimentation. Artists and designers can use AI as a collaborative tool, generating new ideas, iterating on concepts, and pushing the boundaries of what is possible in their respective mediums. As AI becomes more deeply integrated into the creative process, it will undoubtedly raise questions about the nature of creativity, authorship, and the role of technology in artistic expression. These are complex and nuanced issues that will require ongoing dialogue and exploration as we navigate this new frontier of AI-enabled creativity. The release of GPT-4o Mini by OpenAI represents a major milestone in the evolution of AI technology. Its exceptional performance, cost-effectiveness, and multimodal capabilities make it a catalyst for a wide range of industries and applications. As businesses and developers embrace the potential of this powerful model, we can expect to see a wave of innovation and transformation across various sectors. However, the implications of GPT-4o Mini and other advanced AI models extend far beyond just technological progress. They raise important questions about the future of work, the nature of creativity, and the ethical and societal implications of AI. As we navigate this new era of AI-driven change, it will be crucial to engage in ongoing dialogue and collaboration to ensure that the benefits of these technologies are widely shared and that potential risks are carefully managed. With GPT-4o Mini leading the charge, the future of AI looks brighter and more promising than ever. As we stand on the cusp of this exciting new frontier, it is up to us to shape the direction of AI development and deployment in ways that promote innovation, inclusivity, and the greater good of society as a whole. The journey ahead may be complex and challenging, but the potential rewards are truly transformative.