SK hynix and TetraMem unveil in-memory computing chip that processes AI directly inside memory

SK hynix and TetraMem collaborate on experimental chip to bolster energy efficiency for edge AI devices -- memristor-based in-memory SoC research leaves performance questions up in the air

SK hynix, TetraMem, and researchers from the University of Southern California have developed a memristor-based in-memory computing (IMC) system-on-chip (SoC) for AI edge devices. The device is designed to accelerate neural network inference in lightweight AI models while consuming a fraction of the power that higher-end GPUs or NPUs would. To a large degree, the SoC is a proof-of-concept chip, as its performance would peak at around 2.54 TOPS in a theoretical best-case scenario, which is 16X below Microsoft's Copilot+ requirements. A DWC-optimized IMC architecture Memristor-based in-memory computing (IMC) accelerates neural networks by performing analog computations directly inside memory arrays, which reduces data movement and power consumption. However, depthwise convolution (DWC) -- a core operation in lightweight networks such as MobileNet -- performs independent per-channel filtering with limited data reuse and therefore maps poorly onto conventional crossbar arrays. To address this limitation, researchers from SK hynix, TetraMem, and USC developed an SoC that features both conventional IMC crossbars and a memristor-based IMC architecture specifically optimized for DWC. The jointly developed SoC is based on an embedded RISC-V processor that schedules workloads and features 10 neural processing units (NPUs). One NPU out of 10 is dedicated to depthwise convolution, while the remaining nine execute pointwise and dense operations. Nine out of 10 NPU include a 256 × 256 memristor crossbar that performs the analog vector-matrix multiplication (VMM), 256 8-bit DACs that convert digital activations into analog voltages, 256 8-bit ADCs that convert the analog outputs back into digital values, and additional peripheral circuitry for reading, writing, programming, and controlling the crossbar. The DWC-optimized NPU replaces its conventional array with eight specialized 252 × 28 zig-zag crossbar blocks, but retains DACs and ADCs. SK hynix developed and fabricated the memristor devices and integrated the resistive switching cells on top of the 65 nm CMOS circuitry using its back-end process. That DWC-optimized NPU is the key feature of the whole SoC. To accelerate depthwise convolution, TetraMem replaced the straight selection lines used in conventional 1T1R crossbars with a zig-zag topology. As a result, the NPU contains eight 252 × 28 crossbar blocks whose diagonal selection lines activate 252 memory cells across 28 columns, which enables 28 independent 3 × 3 convolutions to run in parallel while using 100% of the array for weight storage. The remaining nine NPUs retain conventional 1T1R crossbars for 1×1 pointwise and dense layers and preserve the throughput and energy efficiency of traditional in-memory computing. Great efficiency, low performance overall To demonstrate the architecture, the researchers deployed a customized MobileNetV1Small neural network for the Visual Wake Words benchmark. The network contains approximately 36,000 parameters; all depthwise layers were mapped to the dedicated NPU, and pointwise layers were mapped to the remaining NPUs. Because the memristor-based IMC hardware natively performs unsigned analog vector-matrix multiplication, inputs and weights are quantized to unsigned 8-bit values before execution. Since each memristor device can be programmed with only slightly more than 2 bits of effective precision, the design uses a two-subarray compensation technique that boosts effective weight precision to roughly 4 bits. Conceptually, the approach is somewhat analogous to Nvidia's NVFP4 philosophy, in that both seek to achieve higher effective precision from low-precision hardware. However, the implementations are fundamentally different: NVFP4 relies on a digital floating-point representation and scaling factors, whereas the memristor SoC improves precision by compensating for analog programming errors using two programmed subarrays. When it comes to accuracy, the SoC achieved an end-to-end inference accuracy of 80.36%, which matches the corresponding 4-bit software model. As for performance, the SoC delivers a peak throughput of 0.254 TOPS per NPU and reaches an energy efficiency of 21.3 TOPS/W at 100 MHz and 11.9 TOPS/W at 400 MHz. According to the authors, this compares favorably with published SRAM-based compute-in-memory accelerators despite being manufactured on an older 65 nm process. The SoC also exceeds Nvidia's A100 INT8 energy efficiency by an order of magnitude, the joint paper claims. Yet, these claims are largely unsubstantiated. First up, the MobileNet demonstration does not even use all 10 NPUs. It uses one dedicated DWC NPU, five standard NPUs for pointwise layers, and leaves four standard NPUs idle. The demonstration thereby does not reveal total SoC throughput (TOPS), sustained throughput running a real network, and throughput with all 10 NPUs simultaneously saturated. In fact, the paper does not even reveal whether all 10 NPUs can be used at the same time. To that end, the 2.54 TOPS figure we mentioned earlier in the story is highly theoretical. Validated approach SK hynix, TetraMem, and researchers from the University of Southern California have developed a memristor-based IMC SoC featuring a novel depthwise convolution accelerator that improves crossbar utilization for lightweight AI workloads. The partners have managed to fabricate it using an outdated 65nm process technology and make it work, achieving a 21.3 TOPS/W energy efficiency and inference accuracy comparable to a 4-bit software model despite the fact that memristors can be programmed with a circa 2-bit accuracy. While the architecture validates that the approach works, the paper does not disclose the full performance of the SoC, and it is not clear whether the chip's 10 NPUs can be saturated at all. Follow Tom's Hardware on Google News, or add us as a preferred source, to get our latest news, analysis, & reviews in your feeds.

New chip moves computing closer to memory to save power, boost speed

Artificial intelligence chip developers SK hynix and TetraMem have demonstrated a new memory-centric processor that performs AI calculations directly inside memory, a design aimed at cutting energy use and reducing the bottlenecks caused by moving data between processors and memory. The two companies announced the completion of a joint technology collaboration centered on an analog in-memory computing (A-IMC) system-on-chip (SoC). Their work demonstrates how memory can take on part of the computing workload instead of simply storing data. The prototype uses memristor-based in-memory computing to carry out efficient depthwise convolution, a key operation used in many AI inference models. By processing data where AI model weights are stored, the architecture reduces the need to repeatedly transfer information between memory and processors. The approach targets one of the biggest challenges facing modern AI hardware. As AI models grow from billions to trillions of parameters, data movement has become a major source of power consumption, latency, and heat generation inside computing systems. Traditional AI chips continuously move data between compute units and memory, consuming both time and energy. Analog in-memory computing changes that workflow by performing matrix calculations directly within the memory array, reducing unnecessary data transfers. The joint project combines TetraMem's analog in-memory computing platform with SK hynix's expertise in advanced memory technologies. The companies also integrated emerging memory devices, circuit design, AI architecture, software, and system optimization into a single semiconductor platform. "We are honored to celebrate this important milestone together with SK hynix," said Glenn Ge, CEO and Co-Founder of TetraMem. "This achievement demonstrates what can be accomplished through close collaboration across the semiconductor ecosystem." According to the companies, the work goes beyond proving the concept of analog in-memory computing by demonstrating a practical AI system-on-chip that integrates multiple layers of hardware and software engineering. Growing AI workloads have increased pressure on chipmakers to improve energy efficiency without sacrificing performance. Memory-centric computing has emerged as one possible solution because moving data often consumes more energy than the calculations themselves. "We believe memory-centric computing and Analog In-Memory Computing will become increasingly important technologies for addressing future AI energy efficiency and thermal challenges, and we look forward to continuing our collaboration with SK hynix," Ge said. The project represents a strategic move for SK hynix beyond traditional memory manufacturing into advanced computing architectures. While the company is a major producer of dynamic random-access memory (DRAM) and high-bandwidth memory (HBM) used in standard AI systems, this prototype shifts toward a neuromorphic approach. "We are pleased to see the successful outcome of this collaboration and the recognition from Advanced Intelligent Systems," said Soo Gil Kim, Vice President of SK hynix. "This project demonstrates the value of exploring innovative memory technologies and new computing architectures for future AI systems." The research paper was also selected as the cover feature of the journal, highlighting its technical contribution to next-generation AI hardware. The companies said they plan to continue working together on memory technologies, computing architectures and system integration for future AI infrastructure. The study, "A Memristor-based In-Memory Computing SoC with Efficient Depthwise Convolution," was published in Advanced Intelligent Systems.

New in-memory computing chip promises faster processing with lower energy use

SK hynix and TetraMem have developed a new analog in-memory computing chip that processes AI workloads directly inside memory, dramatically cutting energy use and latency, and if scalable, could be the answer to one of the biggest challenges the AI industry is facing today. A new partnership between SK Hynix and TetraMem has combined SK Hynix's advanced memory expertise with TetraMem's analog computing, creating a prototype device that uses memristor-based in-memory computing to perform efficient depthwise convolution, a key operation carried out by AI inference models. By processing data where it's stored, the system bypasses the energy-heavy data transfers between memory and compute units. The companies claim this architecture addresses a major bottleneck in modern AI hardware as models scale to trillions of parameters. The device demonstrates that it is possible to process data where AI model weights are stored, and with this new architecture, it dramatically reduces the need to transfer between memory and processors, reducing overall power consumption, latency, and generated heat within the system. "We believe memory-centric computing and Analog In-Memory Computing will become increasingly important technologies for addressing future AI energy efficiency and thermal challenges, and we look forward to continuing our collaboration with SK hynix," said Glenn Ge, CEO and Co-Founder of TetraMem Traditional AI chips involve constant data movement, which consumes both time and energy. Analog in-memory computing shifts the workflow by performing matrix calculations directly within the memory array. The joint project combined emerging memory devices, circuit design, AI architecture, and software to create the new chip. The result is a practical AI system-on-chip that demonstrates the viability of memory-centric computing. "We are honored to celebrate this important milestone together with SK Hynix. This achievement demonstrates what can be accomplished through close collaboration across the semiconductor ecosystem," said Ge The collaboration marks SK Hynix's move toward advanced computing architectures beyond traditional memory manufacturing. The study, published in Advanced Intelligent Systems, highlights the potential of in-memory computing for future AI systems, and given the success of this collaboration, we will undoubtedly see future iterations of this technology in the years to come.