Gimlet Labs Raises $300M to Solve AI's Growing Power Crisis with Multi-Silicon Inference Cloud

Investing in Gimlet | Andreessen Horowitz

We are running out of watts. AI inference is one of the fastest-growing markets in the history of capitalism, and we are running out of nearly every physical input required to serve it: powered land, turbines, transformers, data centers, GPUs, advanced-node wafers, and high-bandwidth memory. GPU capacity is being leased at record prices, while OpenAI's and Anthropic's growth is constrained by how quickly they can expand their compute fleets. You feel that scarcity firsthand: rate limits tighten and responses slow as providers trade latency for throughput to produce more tokens from the same watts. This is why the largest capital investment cycle in contemporary history is underway. Seven of the ten most valuable companies in the world are hyperscalers or supply them. Five U.S. hyperscalers alone are expected to spend $1 trillion in capex next year. NVIDIA has gone a step further, working with capital partners to mobilize more than $500 billion for AI factories. But expansion alone will not be enough. Inference demand compounds at software speed; power plants, data centers, and semiconductor fabs do not. Serving the most consequential workload of our lifetime requires squeezing more intelligence from every watt. That starts with recognizing that inference is not one workload. The End of One-Size-Fits-All Compute Without the GPU, today's AI would not exist. Its flexibility enabled the early experiments with transformers. Then enormous clusters built around a single accelerator architecture powered the training runs that proved scaling laws and produced today's frontier models. But the diversity of AI applications is exploding. A voice assistant lives or dies on latency; batch processing maximizes throughput; a research agent may prioritize cost per token; a coding agent must balance all three. Even a single model call splits into compute-intensive prefill and memory-bound decode. Agents multiply the variation by routing among small specialized and larger reasoning models, retrieving files, executing code on CPUs, and calling external tools. Inference is no longer a single workload. At the same time, hardware choices are proliferating. Some architectures optimize arithmetic throughput; others optimize memory capacity, bandwidth, or low-latency connectivity. No single processor can deliver the best combination of cost, throughput, and latency across every application and every step. GPUs will remain foundational, but they will increasingly operate alongside CPUs, memory-optimized systems, and other purpose-built silicon. The optimal system must match each piece of work to the architecture that handles it best, producing more useful work from the same resources. It must be heterogeneous. The challenge is making all of this hardware operate as one system. Large-scale infrastructure is designed around homogeneity with standardized servers, networking, power, cooling, and operating models. Introducing multiple architectures creates complexity across every layer. The software must coordinate processors that were never designed to work together, while the physical facility must accommodate different networking topologies, rack densities, power profiles, and cooling requirements. Cerebras and NVIDIA systems, for example, require different inlet-water temperatures. Sometimes heterogeneous compute is a compiler problem; sometimes it is a plumbing problem. The Multi-Silicon Inference Cloud Making heterogeneous compute work at scale is key to unlocking what we believe will be the world's largest market. We believe Gimlet Labs has built the solution: the first multi-silicon inference cloud, designed to produce more intelligence from every watt. For each workload, Gimlet determines an execution plan that balances latency, throughput, and cost. It can route different models and tools onto different processors, separate prefill from decode, or divide a model at the layer or operation level. Its compiler optimizes each piece for the target hardware, while its runtime coordinates execution across the system. To developers, all of that complexity sits behind a single inference API. Gimlet extends that orchestration into the physical data center. It integrates GPUs, CPUs, and purpose-built accelerators into one pool of capacity, managing the differences in networking, power, and cooling that make heterogeneous infrastructure difficult to operate. This system is already delivering up to 10X gains in throughput and interactivity on frontier models within the same power envelope. In a world where scale matters, Gimlet has done this at the largest scale possible, counting both a frontier lab and a hyperscaler as customers. In a market starved for compute, efficiency is net-new capacity and latency is product differentiation. We believe there is no better team than Zain, Michelle, Natalie, Omid, and James to tackle this problem. They've seen this movie before and are one of the few teams that have the range to follow a problem through every layer of abstraction. When kernels and compilers were not enough, they pushed into high-speed networking. When software was not enough, they moved into power, cooling, and data center construction. Wherever the existing stack ends, the Gimlet team starts. The future of inference is heterogeneous, and Gimlet is building the infrastructure layer that makes it possible. We could not be more excited to partner with the entire Gimlet team.

Gimlet Labs nabs $300M for its disaggregated inference platform

Gimlet Labs nabs $300M for its disaggregated inference platform Gimlet Labs Inc., a startup that helps developers speed up their inference workloads, has raised $300 million in funding at a $3 billion valuation. Andreessen Horowitz led the Series B round. Gimlet stated in a blog post today that the firm was joined by Arm Holdings Inc., Samsung Ventures, Microsoft Corp.'s M12 fund and more than a dozen others. The company's total outside funding now stands at $392 million. A large language model comprises multiple software modules that have vastly different hardware requirements. For example, some modules make heavier use of the underlying graphics cards' memory than others. Gimlet has developed a platform that harnesses that phenomenon to speed up inference. The company's software can automatically break up an LLM into its consistent modules. From there, it deploys each module on the chip architecture that best aligns with its hardware requirements. For example, Gimlet might send a memory-intensive component to an accelerator with a large amount of onboard RAM. There are multiple ways to split up an LLM. The most common approach, PD disaggregation, runs the prefill and decode phases of the inference workflow on separate chips. The prefill and decode phases are processing steps in which an LLM interprets prompts and generates a response, respectively. Gimlet also supports more granular disaggregation approaches. Some developers don't stop at splitting the inference workflow into prefill and decode phases. They take it a step further by diving the decode phase into smaller workflows that can each be assigned to a different chip. In other projects, developers use a lightweight "drafter" model to generate initial prompt responses and a frontier LLM to refine those drafts. Each model runs on a separate chip architecture. Gimlet's platform reduces the amount of work required to implement such disaggregation workflows. Additionally, it optimizes each LLM module for the chip architecture on which it's deployed. The software optimizes model code using a combination of AI agents and a custom compiler. According to Gimlet, its agents find the best way of adapting LLM code to a chip by exploring multiple design approaches. After identifying a promising approach, the AI runs tests to ensure that it's correct. The company's compiler, in turn, applies a mix of generic and chip-specific optimizations to customer models. Gimlet sells its software in a serverless edition and as a managed service that enterprises can deploy on their own infrastructure. The company says that it has received billions of dollars worth of customer orders. In March, Gimlet stated that its customer base includes one of the world's largest cloud providers and a top three AI lab. The funding will help the company grow the infrastructure capacity of its platform's serverless edition. Gimlet plans to add several hundred megawatts of computing power. Additionally, the company intends to expand into the custom hardware market. Gimlet is developing an inference-optimized server that doesn't feature a motherboard and can be used outside data centers.