Undergrads Create Open-Source AI Speech Model Rivaling Industry Giants

Undergrad Duo Develops Cutting-Edge AI Speech Model

In a surprising turn of events, two undergraduate students with limited AI expertise have developed an open-source AI speech model that rivals industry giants. Toby Kim and his co-founder, operating under the name Nari Labs, have created Dia, a 1.6 billion parameter text-to-speech (TTS) model designed to produce naturalistic dialogue from text prompts 1

1

2

2

.

Dia's Capabilities and Technical Specifications

Dia offers advanced features that set it apart from existing models:

1. Customizable voices and scripts
2. Insertion of disfluencies, coughs, laughs, and other nonverbal cues
3. Emotional tone control and speaker tagging
4. Voice cloning capabilities

The model runs on PyTorch 2.0 and CUDA 12.0, requiring about 10GB of VRAM. It can generate approximately 40 tokens per second on enterprise-grade GPUs like the NVIDIA A4000 2

2

.

Development Process and Resources

The creators of Dia leveraged Google's TPU Research Cloud program, which provided free access to the company's TPU AI chips for training. This resource was crucial in enabling the undergraduates to compete with well-funded companies in the AI space 1

1

.

Comparison with Industry Leaders

Nari Labs claims that Dia outperforms competing proprietary offerings from ElevenLabs, Google's NotebookLM, and potentially even OpenAI's recent gpt-4-0-mini-tts 2

2

. The company provides side-by-side comparisons on their website, demonstrating Dia's superior handling of:

1. Natural timing and nonverbal expressions
2. Multi-turn conversations with emotional range
3. Nonverbal-only scripts
4. Rhythmically complex content like rap lyrics

Open-Source Nature and Accessibility

Dia is fully open-source, distributed under the Apache 2.0 license, allowing for commercial use. The model is available for download from Hugging Face and GitHub, and can run on most modern PCs with at least 10GB of VRAM 1

1

2

2

.

Potential Applications and Future Plans

The flexibility of Dia opens up various use cases, including:

1. Content creation
2. Assistive technologies
3. Synthetic voiceovers

Nari Labs is developing a consumer version of Dia for casual users interested in remixing or sharing generated conversations. They also plan to release a technical report and expand language support beyond English 1

1

2

2

.

Ethical Considerations and Challenges

While Dia offers impressive capabilities, it also raises concerns about potential misuse. The model currently lacks robust safeguards against the creation of disinformation or scam recordings. Nari Labs discourages abuse but states they are not responsible for misuse 1

1

.

Additionally, questions arise about the data used to train Dia, as it may include copyrighted content. This issue reflects a broader debate in the AI industry about the legality and ethics of training models on copyrighted materials 1

1

.

As Dia enters the market, it represents both the democratization of AI technology and the need for careful consideration of its implications and responsible deployment in the rapidly evolving field of synthetic speech.