Microsoft releases Phi-4 multimodal model that knows when thinking wastes time

Microsoft built Phi-4-reasoning-vision-15B to know when to think -- and when thinking is a waste of time

Microsoft on Tuesday released Phi-4-reasoning-vision-15B, a compact open-weight multimodal AI model that the company says matches or exceeds the performance of systems many times its size -- while consuming a fraction of the compute and training data. The release marks the latest and most technically ambitious chapter in the software giant's year-long campaign to prove that carefully engineered small models can compete with, and in key areas outperform, the industry's largest AI systems. The 15-billion-parameter model, available immediately through Microsoft Foundry, HuggingFace, and GitHub under a permissive license, processes both images and text and can reason through complex math and science problems, interpret charts and documents, navigate graphical user interfaces, and handle everyday visual tasks like captioning photos and reading receipts. It arrives at a moment when the AI industry is grappling with a fundamental tension: the biggest models deliver the best raw performance, but their enormous cost, latency, and energy consumption make them impractical for many real-world deployments. "Our goal is to contribute practical insight to the community on building smaller, efficient multimodal reasoning models," the Microsoft Research team wrote in the model's official announcement, "and to share an open-weight model that is competitive with models of similar size at general vision-language tasks, excels at computer use, and excels on scientific and mathematical multimodal reasoning." How Microsoft trained a competitive vision model on one-fifth the data Perhaps the most striking claim in the release is how little training data the model required relative to its competitors. Phi-4-reasoning-vision-15B was trained on approximately 200 billion tokens of multimodal data, built atop the Phi-4-Reasoning language backbone (itself trained on 16 billion tokens) and the foundational Phi-4 model (400 billion unique tokens). By contrast, rival multimodal models from Alibaba's Qwen family (2.5 VL and 3 VL), Moonshot AI's Kimi-VL, SenseTime's InternVL series, and Google's Gemma3 each consumed more than one trillion tokens during training -- roughly five times the total data pipeline Microsoft used. That disparity matters enormously for economics. Training large AI models costs millions of dollars in cloud compute, and the environmental footprint of trillion-token training runs has drawn increasing scrutiny from regulators and investors alike. If Microsoft's claims hold up under independent evaluation, the model represents a significant advance in training efficiency -- one that could reshape how organizations think about the build-versus-buy calculus for AI deployment. The secret, according to the research team, lies not in scale but in meticulous data curation. The team's final dataset drew primarily from three sources: open-source datasets that were "meticulously filtered and improved"; high-quality domain-specific internal data; and targeted data acquisitions. The researchers described a hands-on quality assurance process in which team members manually reviewed samples from each dataset, typically spending five to ten minutes classifying data quality before deciding how to treat each source. For data with incorrect answers, they re-generated responses using GPT-4o and o4-mini. When questions were unsalvageable but images were high quality, they repurposed the images as seeds for new caption or visual question-answering data. They also reported fixing "a surprisingly large number of formatting and logical errors across widely used open-source datasets" -- a finding that raises uncomfortable questions about the quality of training data underpinning many of the industry's most prominent models. Why the model reasons through calculus but stays quiet on captions The model's most technically novel contribution may be its approach to reasoning. In the world of language-only AI, "reasoning models" -- systems that spend extra compute time working through problems step by step -- have become the hottest category in the field, with OpenAI's o-series and DeepSeek's R1 leading the charge. But extending reasoning to multimodal tasks involving images introduces a wrinkle: for many visual tasks like image captioning or optical character recognition, chain-of-thought reasoning is not only unnecessary but can actually degrade performance by introducing unnecessary verbosity and latency. Microsoft's solution was to build what it calls a "mixed reasoning and non-reasoning model." The team started with Phi-4-Reasoning, already a capable reasoning language model, and then trained it on a hybrid data mixture where approximately 20 percent of samples included explicit chain-of-thought reasoning traces (wrapped in <think>...</think> tags) and 80 percent were tagged for direct response (with a <nothink> token). The model learned to invoke structured reasoning for domains like math and science where it helps, while defaulting to fast, direct responses for perception-focused tasks where it does not. This design choice reflects a pragmatic view of reasoning that contrasts with the industry's current enthusiasm for always-on thinking. As the research team explained: "For tasks such as image captioning and optical character recognition (OCR), reasoning is often unnecessary and can even be harmful, while mathematical and scientific problem-solving benefit from multi-step reasoning." Users who want to override the model's default behavior can do so by explicitly prompting with <think> or <nothink> tokens. The team explored four possible training pipelines for multimodal reasoning and chose the one they judged to best balance capability, efficiency, and data requirements. The alternative approaches -- training reasoning and multimodal capabilities simultaneously from a non-reasoning base, learning multimodal skills first and then adding reasoning, or requiring reasoning traces for all training data -- each carried significant drawbacks. Training reasoning from scratch demands enormous multimodal reasoning data. Adding reasoning after multimodal training risks catastrophic forgetting. And forcing reasoning on every query wastes compute on tasks that don't benefit from it. Inside the vision architecture that makes high-resolution screenshots readable Under the hood, Phi-4-reasoning-vision-15B uses a mid-fusion architecture that pairs a SigLIP-2 vision encoder with the Phi-4-Reasoning language backbone. The choice of mid-fusion -- where a pretrained vision encoder converts images into tokens that are then projected into the language model's embedding space -- over early-fusion, where images and text are processed together in a single transformer, reflects the team's resource constraints. Early-fusion yields richer joint representations but demands significantly more compute, memory, and data. The team conducted careful ablation studies on how to handle image resolution, an issue that matters critically for tasks like reading dense screenshots or small UI elements. They tested four approaches -- Dynamic S, Multi-crop, Multi-crop with S, and dynamic resolution using SigLIP-2's Naflex variant -- and found that dynamic resolution encoders performed best, especially on high-resolution data. They selected the SigLIP-2 Naflex variant with up to 3,600 maximum tokens, which corresponds roughly to native 720p resolution and delivered particularly strong results on benchmarks requiring fine-grained visual understanding like ScreenSpot-Pro. This matters for one of the model's headline use cases: powering computer-using agents that navigate desktop, web, and mobile interfaces. With strong high-resolution perception and fine-grained grounding capabilities, the model can identify and localize interactive elements like buttons, menus, and text fields -- a prerequisite for the autonomous software agents that many in the industry view as the next major frontier for AI deployment. The team noted that the model's low inference-time requirements make it particularly well suited "for interactive environments where low latency and compact model size are essential." The benchmarks show a model that trades brute-force accuracy for speed and efficiency The model's benchmark results paint a picture of a system that punches well above its weight class on efficiency while remaining competitive -- though not dominant -- on raw accuracy. On the team's own evaluations across ten benchmarks, Phi-4-reasoning-vision-15B scored 84.8 on AI2D (science diagrams), 83.3 on ChartQA, 75.2 on MathVista, 88.2 on ScreenSpot v2 (UI element grounding), and 54.3 on MMMU (a broad multimodal understanding test). Those numbers generally trail the much larger Qwen3-VL-32B models (which scored 85.0, 84.0, 81.8, 93.9, and 70.6 on the same benchmarks, respectively) but remain competitive with or ahead of similarly-sized systems like Qwen3-VL-8B and Kimi-VL-A3B. The real value proposition, as Figure 1 in the announcement illustrates, emerges when accuracy is plotted against compute time and output token count: Phi-4-reasoning-vision-15B sits at the Pareto frontier of models that are both fast and accurate, delivering competitive results in a fraction of the time required by larger systems. The Microsoft team acknowledged that their benchmark numbers "may be lower than other previously shared numbers" because they ran all evaluations themselves rather than quoting leaderboard claims. They used temperature=0.0, greedy decoding, and a 4,096 maximum output token limit, with no custom prompting or parameter tuning. The team committed to releasing all evaluation logs publicly -- a transparency practice that remains uncommon in the field and should allow independent researchers to verify the results. Still, independent reproduction will be critical: the AI research community has grown increasingly skeptical of self-reported numbers, particularly when evaluation methodologies differ across organizations. From edge devices to humanoid robots, the Phi family keeps expanding Phi-4-reasoning-vision-15B does not exist in isolation. It is the latest entry in a Phi model family that has expanded rapidly over the past year, evolving from a niche research project into a central pillar of Microsoft's AI strategy -- one that now spans language, vision, on-device inference, education, and robotics. The lineage traces back through several milestones. In late 2024, Microsoft released the original Phi-4, a 14-billion-parameter language model that demonstrated the power of synthetic data and careful curation. In April 2025, the company launched Phi-4 mini reasoning (3.8 billion parameters), Phi-4 reasoning (14 billion parameters), and Phi-4 reasoning plus -- with the latter reportedly approaching the performance of DeepSeek's R1, a model with 671 billion parameters, according to TechCrunch's reporting at the time. The family has also extended into specialized domains. Phi Silica, an on-device small language model for Copilot+ PCs, has been used with LoRA fine-tuning to customize generation for specific tasks. In one case study detailed on the Windows Developer Blog, Microsoft's education team used LoRA adapters with Phi Silica to generate Kahoot! quizzes, achieving a 75 percent reduction in rejection rates and a 4.6-times uplift in subjective quality scores. On the hardware side, the Phi-4-mini model has been optimized for MediaTek's NPU platforms, running at over 800 tokens per second for prefill on the Dimensity 9400 -- fast enough for real-time AI on smartphones and tablets. And in what may be the most ambitious extension yet, Microsoft announced Rho-alpha (ρα), described as the company's "first robotics model derived from Microsoft's Phi series." According to Microsoft Research, Rho-alpha translates natural language commands into control signals for robotic systems performing bimanual manipulation tasks, adding tactile sensing to the perception stack and targeting dual-arm setups and humanoid robots. What Phi-4-reasoning-vision signals about the future of enterprise AI The release crystallizes a broader shift in the AI industry's center of gravity. For the past two years, the dominant narrative has held that bigger is better -- that raw scale in parameters, data, and compute is the primary driver of capability. Microsoft's Phi family represents the most visible corporate champion of the counterargument: that careful engineering of data quality, training methodology, and architecture design can substitute for brute-force scale. This thesis has significant implications for enterprise adoption. Organizations deploying AI in latency-sensitive or resource-constrained settings -- edge devices, interactive applications, on-premise servers -- cannot practically run trillion-parameter models. A 15-billion-parameter model that delivers 80 to 90 percent of a frontier model's accuracy at a tenth of the inference cost could unlock deployment scenarios that were previously uneconomical. The model's open-weight release, accompanied by fine-tuning code and benchmark logs, also represents a competitive strategy. By making the model freely available and deeply documented, Microsoft positions Phi as a foundation layer for an ecosystem of downstream applications -- many of which will run on Azure, use Microsoft's development tools, or integrate with its enterprise software stack. Yet the model still trails the largest open-weight competitors on the hardest benchmarks, particularly in mathematical reasoning (where Qwen3-VL-32B-Thinking-40K scores 78.2 on MathVerse compared to 53.1 for Phi-4-reasoning-vision with forced thinking) and general multimodal understanding (MMMU scores of 72.2 versus 55.0). The 20/80 reasoning-to-non-reasoning data split is, by the team's own admission, a heuristic that "may not be optimal for all domains or deployment contexts." And the model's ability to correctly decide when to reason and when to respond directly remains what the researchers called "an open problem." Microsoft is wagering that in the real world, where latency budgets are tight, hardware is finite, and deployment costs compound with every API call, the smartest model is not the biggest one -- it's the one that knows when to think and when to just answer. Whether that bet pays off will depend less on benchmark tables and more on what happens when millions of developers start putting Phi-4-reasoning-vision to work. The model is available now on Microsoft Foundry, HuggingFace, and GitHub. The leaderboard, as always, is open.

Microsoft open-sources multimodal reasoning model with 15B parameters - SiliconANGLE

Microsoft open-sources multimodal reasoning model with 15B parameters Microsoft Corp. today released a hardware-efficient reasoning model, Phi-4-reasoning-vision-15B, that can process multimodal files such as scientific charts. The model is based on two existing algorithms called SigLIP-2 and Phi-4 Reasoning. SigLIP-2 compresses images into a numerical form that neural networks can understand. Phi-4 Reasoning, in turn, is a reasoning model that Microsoft open-sourced last April. The company's researchers combined the two algorithms using an approach known as mid-fusion. An artificial intelligence model comprises collections of artificial neurons called layers. Engineers can equip all of a model's layers with the ability to process multimodal data. In mid-fusion models such as Phi-4-reasoning-vision-15B, only some of the layers support multimodal processing. That arrangement trades off some output quality for a significant reduction in hardware use. According to Microsoft, users can further lower the model's infrastructure footprint by disabling its reasoning feature. The capability can be turned on and off with prompts. The company mainly trained Phi-4-reasoning-vision-15B on open-source data. The data included images and text-based descriptions of the objects depicted in those images. Before it started training the model, Microsoft refined the files through a multi-step process. First, the company identified high-quality datasets that didn't require changes and set them aside. It then searched for file collections that comprised high-quality images with inaccurate captions. Microsoft's researchers generated new captions for those images using GPT-4o and o4-mini. The company enriched the refined open-source files with internally-created training data and "high-quality data from targeted acquisitions." Furthermore, it added in examples of behaviors that the model should avoid. The latter dataset helps Phi-4-reasoning-vision-15B avoid harmful output. Microsoft compared the algorithm to several similarly-sized reasoning models using a set of open-source benchmarks. Phi-4-reasoning-vision-15B scored 17% higher than Google LLC's gemma-3-12b-it on MathVista_Mini, a benchmark that comprises multimodal math questions. The model also achieved higher scores across more than a half dozen other evaluations. "We have competitive performance to much slower models that require ten times or more compute-time and tokens and better accuracy than similarly fast models, particularly when it comes to math and science reasoning," Microsoft researchers wrote in a blog post today. Developers can use Phi-4-reasoning-vision-15B to build AI agents that interact with applications via their user interfaces. The model is capable of deducing the function of different interface elements based on screenshots. "With strong high-resolution perception and fine-grained grounding capabilities, Phi-4-reasoning-vision-15B is a compelling option as a base-model for training agentic models such as ones that navigate desktop, web, and mobile interfaces by identifying and localizing interactive elements such as buttons, menus, and text fields," the researchers explained. The model can also analyze more complicated visual assets such as scientific charts. In a demo shared by Microsoft, a user uploaded a photo of Saturn and asked Phi-4-reasoning-vision-15B why the planet appears tilted. It explained that Saturn's orientation depends on the time of year and the position of the telescope that took the photo.