Google's Titans and Sakana's Transformer Squared: Revolutionizing AI Architectures Beyond Transformers

Google's New Titans AI Architecture Could Give LLMs 'Long-Term Memory'

Titans AI is said to memorise and forget context during test time Google researchers unveiled a new artificial intelligence (AI) architecture last week that can enable large language models (LLMs) to remember the long-term context of events and topics. A paper was published by the Mountain View-based tech giant on the topic, and the researchers claim that AI models trained using this architecture displayed a more "human-like" memory retention capability. Notably, Google ditched the traditional Transformer and Recurrent Neural Network (RNN) architectures to develop a new method to teach AI models how to remember contextual information. The lead researcher of the project, Ali Behrouz, posted about the new architecture on X (formerly known as Twitter). He claimed that the new architecture provides a meta in-context memory with attention that teaches AI models how to remember the information at test-time compute. According to Google's paper, which has been published in the pre-print online journal arXiv, the Titans architecture can scale the context window of AI models to larger than two million tokens. Memory has been a tricky problem to solve for AI developers. Humans remember information and events with context. If someone asked a person about what he wore last weekend, they would be able to remember additional contextual information, such as attending a birthday party of a person who they have known for the last 12 years.This way, when asked a follow-up question about why they wore a brown jacket and denim jeans last weekend, the person would be able to contextualise it with all these short-term and long-term information. AI models, on the other hand, typically use retrieval-augmented generation (RAG) systems, modified for Transformer and RNN architectures. It uses information as neural nodes. So, when an AI model has been asked a question, it accesses the particular node that contains the main information, as well as the nearby nodes that might contain additional or related information. However, once a query is solved, the information is removed from the system to save processing power. However, there are two downsides to this. First, an AI model cannot remember information in the long run. If one wanted to ask a follow-up question after a session was over, one would have to provide the full context again (unlike how humans function). Second, AI models do a poor job of retrieving information involving long-term context. With Titans AI, Behrouz and other Google researchers sought to build an architecture which enables AI models to develop a long-term memory that can be continually run, while forgetting information so that it be computationally optimised. To this end, the researchers designed an architecture that encodes history into the parameters of a neural network. Three variants were used -- Memory as Context (MAC), Memory as Gating (MAG), and Memory as a Layer (MAL). Each of these variants is suited for particular tasks. Additionally, Titans uses a new surprise-based learning systen, which tells AI models to remember unexpected or key information about a topic. These two changes allow Titans architecture to showcase improved memory function in LLMs. In a separate post, Behrouz claimed that based on internal testing on the BABILong benchmark (needle-in-a-haystack approach), Titans (MAC) models were able to outperform large AI models such as GPT-4, LLama 3 + RAG, and LLama 3 70B.

Meet Titans : Google's New Brain Inspired AI Model That Outshines Transformers

Google has introduced "Titans," a innovative AI architecture designed to address the limitations of the widely-used Transformer model. Since its introduction in 2017, the Transformer model has been a foundational element in artificial intelligence, powering advancements in natural language processing, machine translation, and more. However, despite its success, the model faces challenges in scalability, computational efficiency, and handling increasingly complex tasks. Titans aim to overcome these obstacles by integrating mechanisms inspired by human cognitive processes, such as memory prioritization and adaptive attention. These innovations enhance the model's ability to process vast datasets, scale effectively, and tackle intricate problems with greater precision and efficiency. What makes Titans so exciting isn't just its ability to process more data or solve tougher problems -- it's how it does it. By mimicking human cognitive processes, like prioritizing surprising or novel information and dynamically managing memory, Titans promise to tackle some of AI's biggest limitations. Whether it's extending context windows far beyond what was previously possible or adapting on the fly to new challenges, this architecture could redefine what AI can achieve. But what does this mean for the real world? From scientific breakthroughs to smarter everyday tools, the possibilities are staggering and Wes Roth provides more insight below. Titans represent a significant evolution from the Transformer model, addressing its most pressing limitations. While Transformers have been instrumental in advancing AI, they rely on fixed-length context windows and suffer from computational inefficiencies due to quadratic scaling costs. Titans introduce several key innovations to resolve these issues, making them a more robust and efficient architecture. These advancements position Titans as a more capable and efficient alternative, particularly for tasks that demand extensive data processing and long-term contextual reasoning. A defining feature of Titans is its integration of mechanisms modeled after human cognitive processes. By mimicking how the human brain stores, retrieves, and prioritizes information, Titans achieve a level of adaptability and efficiency that sets them apart from traditional AI architectures. The architecture incorporates both short-term and long-term memory modules, allowing it to manage information dynamically and effectively. Key innovations include: These brain-inspired features enable Titans to manage memory with remarkable efficiency, making sure that the architecture remains relevant and adaptable across a wide range of applications. Enhance your knowledge on AI architecture Titans by exploring a selection of articles and guides on the subject. Titans have demonstrated exceptional performance across a variety of domains, showcasing their versatility and potential to drive advancements in numerous fields. Their ability to process large datasets, adapt to new information, and generalize across tasks makes them a valuable tool for both research and industry. These capabilities highlight Titans' potential to transform industries ranging from healthcare and engineering to scientific discovery and beyond. Effective memory management lies at the core of Titans' success. The architecture strikes a careful balance between memorization and generalization, making sure that it retains essential information without overfitting to specific datasets. This balance is achieved through advanced dynamic memory management techniques, which enable Titans to adapt to evolving requirements and diverse challenges. These features ensure that Titans remain a robust and versatile solution, capable of addressing the demands of both research and practical applications. Titans represent a pivotal advancement in AI development, addressing critical challenges in scalability, memory management, and computational efficiency. By combining human brain-inspired mechanisms with innovative computational techniques, Titans pave the way for more powerful and versatile AI systems capable of tackling increasingly complex problems. Potential impacts include: By addressing longstanding limitations and introducing new innovations, Titans set the stage for a new era in AI development. Their ability to process vast amounts of data, adapt dynamically, and generalize across tasks positions them as a fantastic force in the evolution of artificial intelligence.

Beyond Transformers: New AI Architectures Could Revolutionize Large Language Models - Decrypt

In the past weeks, researchers from Google and Sakana unveiled two cutting-edge neural network designs that could upend the AI industry. These technologies aim to challenge the dominance of transformers -- a type of neural network that connects inputs and outputs based on context -- the technology that has defined AI for the past six years. The new approaches are Google's "Titans," and "Transformers Squared," which was designed by Sakana, a Tokyo AI startup known for using nature as its model for tech solutions. Indeed, both Google and Sakana tackled the transformer problem by studying the human brain. Their transformers basically utilize different stages of memory and activate different expert modules independently, instead of engaging the whole model at once for every problem. The net result makes AI systems smarter, faster, and more versatile than ever before without making them necessarily bigger or more expensive to run. For context, transformer architecture, the technology which gave ChatGPT the 'T' in its name, is designed for sequence-to-sequence tasks such as language modeling, translation, and image processing. Transformers rely on "attention mechanisms," or tools to understand how important a concept is depending on a context, to model dependencies between input tokens, enabling them to process data in parallel rather than sequentially like so-called recurrent neural networks -- the dominant technology in AI before transformers appeared. This technology gave models context understanding and marked a before and after moment in AI development. However, despite their remarkable success, transformers faced significant challenges in scalability and adaptability. For models to be more flexible and versatile, they also need to be more powerful. So once they are trained, they cannot be improved unless developers come up with a new model or users rely on third-party tools. That's why today, in AI, "bigger is better" is a general rule. But this may change soon, thanks to Google and Sakana. Google Research's Titans architecture takes a different approach to improving AI adaptability. Instead of modifying how models process information, Titans focuses on changing how they store and access it. The architecture introduces a neural long-term memory module that learns to memorize at test time, similar to how human memory works. Currently, models read your entire prompt and output, predict a token, read everything again, predict the next token, and so on until they come up with the answer. They have an incredible short-term memory, but they suck at long-term memory. Ask them to remember things outside their context window, or very specific information in a bunch of noise, and they will probably fail. Titans, on the other hand, combines three types of memory systems: short-term memory (similar to traditional transformers), long-term memory (for storing historical context), and persistent memory (for task-specific knowledge). This multi-tiered approach allows the model to handle sequences over 2 million tokens in length, far beyond what current transformers can process efficiently. According to the research paper, Titans shows significant improvements in various tasks, including language modeling, common-sense reasoning, and genomics. The architecture has proven particularly effective at "needle-in-haystack" tasks, where it needs to locate specific information within very long contexts. The system mimics how the human brain activates specific regions for different tasks and dynamically reconfigures its networks based on changing demands. In other words, similar to how different neurons in your brain are specialized for distinct functions and are activated based on the task you're performing, Titans emulate this idea by incorporating interconnected memory systems. These systems (short-term, long-term, and persistent memories) work together to dynamically store, retrieve, and process information based on the task at hand. Just two weeks after Google's paper, a team of researchers from Sakana AI and the Institute of Science Tokyo introduced Transformer Squared, a framework that allows AI models to modify their behavior in real-time based on the task at hand. The system works by selectively adjusting only the singular components of their weight matrices during inference, making it more efficient than traditional fine-tuning methods. Transformer Squared "employs a two-pass mechanism: first, a dispatch system identifies the task properties, and then task-specific 'expert' vectors, trained using reinforcement learning, are dynamically mixed to obtain targeted behavior for the incoming prompt," according to the research paper. It sacrifices inference time (it thinks more) for specialization (knowing which expertise to apply). What makes Transformer Squared particularly innovative is its ability to adapt without requiring extensive retraining. The system uses what the researchers call Singular Value Fine-tuning (SVF), which focuses on modifying only the essential components needed for a specific task. This approach significantly reduces computational demands while maintaining or improving performance compared to current methods. In testing, Sakana's Transformer demonstrated remarkable versatility across different tasks and model architectures. The framework showed particular promise in handling out-of-distribution applications, suggesting it could help AI systems become more flexible and responsive to novel situations. Here's our attempt at an analogy. Your brain forms new neural connections when learning a new skill without having to rewire everything. When you learn to play piano, for instance, your brain doesn't need to rewrite all its knowledge -- it adapts specific neural circuits for that task while maintaining other capabilities. Sakana's idea was that developers don't need to retrain the model's entire network to adapt to new tasks. Instead, the model selectively adjusts specific components (through Singular Value Fine-tuning) to become more efficient at particular tasks while maintaining its general capabilities. Overall, the era of AI companies bragging over the sheer size of their models may soon be a relic of the past. If this new generation of neural networks gains traction, then future models won't need to rely on massive scales to achieve greater versatility and performance. Today, transformers dominate the landscape, often supplemented by external tools like Retrieval-Augmented Generation (RAG) or LoRAs to enhance their capabilities. But in the fast-moving AI industry, it only takes one breakthrough implementation to set the stage for a seismic shift -- and once that happens, the rest of the field is sure to follow.