Imagine watching a dubbed movie, but the characters' lip movements stubbornly stick to the original language, completely breaking the illusion that movies are meant to be. While dubbing, translation, and subtitling may be sorted, if this aspect is ignored, the movie experience is dampened.
Bengaluru-based NeuralGarage was founded in 2021 by Anjan Banerjee, Subhashish Saha, Subhabrata Debnath, and Mandar Natekar, who have known each other for decades. The platform is looking to solve the problem of mismatched facial movements in dubbed content.
NeuralGarage was one of seven Indian AI startups selected for the AWS Global Generative AI Accelerator Program. It received $1 million in AWS promotional credits, among other support initiatives.
Its technology, VisualDub, not only addresses lipsync movements but also ensures the final product looks natural and high-quality. "The challenge is maintaining the video quality when adjusting facial expressions, especially with no limitations on camera angles and lighting," the company stated.
"It cannot look AI-ish," it emphasised, particularly when working with renowned actors whose facial movements are unique.
"For example, if you're watching Money Heist, the professor is still speaking in Spanish while you are listening to him in English. What we do is change the facial movements of the actor so it looks like he has spoken in English," explained Subhabrata Debnath, co-founder and CTO of NeuralGarage.
While dubbed content has become popular, especially on streaming platforms such as Netflix, NeuralGarage sees an opportunity to further enhance the viewing experience. "Once you see it with sync, it is very difficult to go back," they noted.
Prime Video's global head recently acknowledged this challenge, stating that content is still not consumed well because facial expressions and lips don't always match.
NeuralGarage is already being leveraged by major players such as Amazon India, Coca-Cola, and others in advertising, allowing brands to repurpose content with different messages. Debnath explains advertisements shot in December can be updated in February to reflect changing discounts or festivals.
"Amazon India recently started using our solution to repurpose the same video footage with different messages for seasonal campaigns," said Debnath.
In the film industry, NeuralGarage collaborates with movie distributors, such as Europa Movies, which has a 95% market share in India. Through these partnerships, they address challenges related to global distribution, piracy prevention, and dubbing with realistic synchronisation.
NeuralGarage faces significant challenges in perfecting its dubbing and facial synchronisation technology. A primary issue is ensuring that facial adjustments seamlessly match the original high-quality video. The technology also needs to adapt to varying camera angles and lighting conditions, as films are not shot with AI's constraints in mind.
Maintaining a natural, non-AI-generated look is especially crucial for high-profile actors. "Shah Rukh Khan's face is his identity. For movie stars, the expectations are very high," he said.
The scalability of the solution presents another challenge, with films often exceeding 300 GB in size.
However, the challenge of competition persists. Additionally, several AI startups, both globally and in India, are working in the same space. Dubverse.ai and Vitra AI are a couple of names. However, NeuralGarage is looking to go big in 2025.
The platform plans to boost its visibility in 2025 by actively investing in marketing and sales, transitioning from its reliance on organic word-of-mouth within the post-production industry. "This year, we will invest in marketing and sales so that the solution reaches more hands," he said.
After raising $1.45 million from Xfinity Ventures in 2022, the company is preparing for a Series A funding round to scale operations and refine its product. The company's goal is to ensure that more businesses try its solutions, which streamline subtitling and localisation processes.
