Stability AI and Arm Partner to Bring Offline Generative Audio to Mobile Devices

Stability AI and Arm Join Forces for Mobile Audio Generation

Stability AI, the company behind the popular Stable Diffusion text-to-image models, has partnered with global semiconductor giant Arm to bring generative audio AI capabilities to mobile devices. This collaboration aims to run Stability's text-to-audio model, Stable Audio Open, entirely on Arm CPUs, enabling the generation of sound effects, audio samples, and production elements directly on smartphones without an internet connection 1

1

Optimizing Stable Audio Open for Mobile

The partnership faced the challenge of optimizing the Stable Audio Open model for mobile devices. Initial tests on an Arm CPU-equipped device took 240 seconds to generate audio. However, through model distillation and the use of Arm's software stack, including int8 matmul kernels from KleidiAI in ExecuTorch via XNNPack, they achieved a significant breakthrough 1

1

The optimized model can now generate an 11-second audio clip in under 8 seconds on Armv9 CPUs, resulting in a 30x faster response time. This improvement makes on-device audio generation practical for mobile users 2

2

Advantages of On-Device Audio Generation

Unlike cloud-based AI-powered audio generation apps such as Suno and Udio, Stable Audio Open running on Arm chips can operate offline. This capability provides users with greater flexibility and privacy. Additionally, Stability AI claims that Stable Audio Open's training set consists entirely of royalty-free audio and songs, potentially mitigating intellectual property risks associated with some other audio generation models 2

2

Related Stories

Stability AI and Arm Unveil Stable Audio Open Small: A Lightweight, On-Device Audio Generation Model

Meta and Arm Join Forces to Bring AI to Smartphones

Stability AI Enhances Amazon Bedrock with Advanced Text-to-Image Models

Arm's new AI-native chips promise desktop-class gaming on 2027 Android phones

Future Plans and Implications

Stability AI CEO Prem Akkaraju emphasized the importance of making their models and workflows accessible to creators and businesses adopting generative AI in their production pipelines. The company plans to extend this approach to all its models across image, video, and 3D, aiming to transform visual media creation on mobile devices 1

1

While the optimized Stable Audio Open model is not yet available for download, Stability AI and Arm are collaborating to further optimize and fine-tune it for mobile use. The technology will be demonstrated at the Mobile World Congress conference in Barcelona 2

2

Stability AI's Recent Developments

This partnership comes at a crucial time for Stability AI, which has faced challenges in recent months. The company has undergone significant changes, including raising new capital from investors like Eric Schmidt and Sean Parker, hiring a new CEO, and appointing Titanic director James Cameron to its board of directors. These moves are part of efforts to revitalize the company following reported mismanagement issues under its previous leadership 2

2