Microsoft's VibeVoice: AI-Powered Tool Generates Long-Form Conversational Audio

Microsoft's VibeVoice uses AI to create 90-minute podcasts with multiple speakers

TL;DR: Microsoft's open-source VibeVoice AI generates up to 90 minutes of multi-speaker, high-fidelity conversational audio using advanced text-to-speech technology. It leverages a Large Language Model and diffusion framework to maintain speaker consistency and natural dialogue flow, making it ideal for podcasts and long-form audio content. Microsoft's new open-source text-to-voice generative AI tool, VibeVoice, is an interesting one, as it can generate audio of up to 90 minutes in length with four distinct speakers. Naturally, with a script, VibeVoice becomes a viable tool for creating an audio podcast or other "expressive, long-form, multi-speaker conversational audio." With there already being quite a few AI-powered Text-to-Speech (TTS) systems and tools, what separates VibeVoice from the pack is its ability to maintain and preserve audio fidelity, speaker consistency, and "natural turn-taking" over an extended period. "VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details," the official description reads. VibeVoice offers a live demo for you to check out, along with the option to download it. As a pure Text-to-Speech (TTS) tool, VibeVoice requires a script to work, which you'll need to whip up yourself or use another AI tool like ChatGPT to generate. VibeVoice is available in multiple versions: a compact 1.5 billion-parameter model and a more complex 7 billion-parameter model. There's also a 0.5 billion parameter model on the way, designed for real-time audio generation. For those with a modern GPU, the 1.5 billion-parameter version requires approximately 7GB of VRAM, while the larger 7 billion-parameter model requires around 18GB. As for the quality of VibeVoice, the voices and conversation flow, although impressive, still sound very much like those of an AI. For more on VibeVoice, check out its GitHub repository and Hugging Face page.

Microsoft Unveils VibeVoice for Longer Conversational AI Audio | PYMNTS.com

By completing this form, you agree to receive marketing communications from PYMNTS and to the sharing of your information with our sponsor, if applicable, in accordance with our Privacy Policy and Terms and Conditions. But there are notable differences. Microsoft's text-to-speech model can generate four voices and up to 90 minutes of podcast-quality speech. NotebookLM can do two voices. Additionally, VibeVoice reads and organizes text while NotebookLM ingests documents and turns them into two-person podcasts. Users can also query and get document summaries, according to tech firm Hugging Face. That means VibeVoice doesn't try to understand the text but rather performs it audibly, ostensibly to replace a recording studio. VibeVoice is the latest offering in voice AI technology, which has been attracting venture capital funding. In 2024, voice AI startups raised $2.1 billion, up eightfold from the prior year, according to market research firm CB Insights. There's rising interest in voice shopping: A PYMNTS Intelligence report shows that 30.4% of Gen Z consumers already shop by voice every week, followed by millennials. For all ages, the average is 17.9% of consumers using voice to shop. VibeVoice runs on 1.5 billion parameters, relatively small for a model capable of sustaining dialogue across multiple speakers. It was trained using Alibaba's open-source Qwen2.5, a large language model that helps orchestrate natural turn-taking and contextually aware speech patterns during dialogues. Microsoft claims this means VibeVoice can produce fluid conversations among four voices and yet maintain each voice's distinct characteristics, even in longer conversations. See also: How the World Does Digital: A Deep Dive Into Global Digital Engagement Potential research applications of VibeVoice include the following: Recognizing the risks of deepfakes, Microsoft said VibeVoice's safeguards include ensuring every audio file includes both a disclaimer -- such as "This segment was generated by AI" -- and a hidden digital watermark. It bars impersonation, disinformation and live deepfake uses such as real-time voice conversion in calls. It supports only English and Chinese speech for now. The model is available for research, not commercial deployment. Read more: Nobody's Talking: Voice Interfaces Face Hurdles for Wide Adoption