AI-Powered 'Visual Dubbing' Brings Swedish Film to US Theaters with Seamless English Lip-Sync

AMC Theatres will screen a Swedish movie 'visually dubbed' with the help of AI

Flawless AI used its technology to alter actors' lip movements in Watch the Skies. On May 9, AMC Theatres will start showing a sci-fi movie that was shot in Swedish but will look like it was made in English instead. Watch the Skies, which was released in its home country as UFO Sweden, had undergone "visual dubbing" with the help of artificial intelligence. An AI company called Flawless used its technology to digitally alter the film's images, making the actors look like they were truly speaking in English. Notably, the original actors recorded their own dialogues in English in a sound booth -- Flawless AI's technology merely altered the movements of their lips in the movie. On its website, Flawless says its TrueSync AI technology "captures every nuance of an actor's performance and generates new lip movements that perfectly map to the new language audio, providing the perfect visual dub." Variety says the tool is compliant with the rules set by SAG-AFTRA, which ended a four-month strike in 2023 after securing a deal with studios that protects members "from the threat of AI." Flawless AI's technology could lower the barrier of entry into foreign films. It could make them more appealing to audiences resistant to watching subtitled movies and could provide a better experience for audiences in countries that normally dub movies in their native language. "Showing our materials to filmmakers, especially over the past year, they realize the potential from going to a local stage to a global stage," the company's co-founder, Scott Mann, told Variety. "It's a huge opportunity to get your work out and it's been invigorating. They are so excited about showing their work in a wider audience, and especially in America." Watch the Skies revolves around a teenager who believes that her missing father wasn't dead but was abducted by aliens. To uncover the truth about her father's disappearance, she teams up with UFO Club to look for him. AMC Theatres has committed to showing the film in 100 locations across America.

AMC Using AI-Powered 'Visual Dubbing' to Premiere Swedish Movie In English

More than simply inserting English voice recordings, actors' faces will be altered to make it appear that they are speaking natively. AMC Theaters, always one to hop on the latest trend, is premiering a Swedish movie using a new form of AI-based dubbing that makes it appear as if the actors are speaking in English. Instead of merely replacing the voice tracks with ones recorded by actors in another language, the movie Watch the Skies will additionally see actors' facial movements altered to make it appear as if they are speaking in English. The original actors in Watch the Skies voice the new English-language track, which is in part why the movie is compliant with SAG-AFTRA rules meant to protect members from the threat of AI. Hollywood talent has pushed back strenuously against the use of AI out of fear that they will someday be replaced entirely. But because the English dialogue was recorded by the actors themselves rather than generated by AI, and with full consent, it does not violate union contracts. The company behind the "visual dubbing," as it is called, is Flawless AI. By digitally altering the film's images to make it appear natively filmed in English, the thinking goes that more moviegoers put off by dubbing will give the film a chance. Translating into another language can sometimes break the lip sync; this new dubbing alleviates that issue by altering the mouth movements themselves. Some may not be happy about this new AI-based dubbing in film, as it is almost tantamount to declaring English is the superior language, erasing other cultures and dialects in the process. It also comes at a time when foreign language films have become increasingly popular in the United States and viewers have become more accustomed to using subtitles. Hollywood already dominates the global entertainment industry, the thinking goes, and automatically dubbing into English could be seen as an encroachment at a time when other countries have only just started to see their local film industries take off globally. Critics might argue that much of the world does not speak English and should not be forced to bend to the will of Americans, but this seems like a minor case of translating a small budget movie intended for theatrical release in English-speaking markets. There is also concern that something could be lost when a movie is not watched in its native tongueâ€"the voice acting is a crucial part of any film, and actors switching to non-native English from Swedish might lose some of their emotional delivery or translate dialogue imprecisely. But voice actors have gotten better at dubbing over the years, and proponents of dubbing say that people should watch films any way they prefer. Companies beyond AMC have been implementing AI-based dubbing in order to expand the audience for content. The popular YouTube creator Mr. Beast made headlines several years ago for dubbing his videos in other languages like Spanish and Mandarin, but at the time he was using a more manual and costly process for doing so. YouTube late last year introduced a tool that lets creators instead automatically create AI voiceovers in other languages. While the true value of new AI technology remains in question, there is hope in Hollywood that generative AI will be able to speed up the production process and reduce costs for visual effects. Companies like Netflix have disrupted the industry significantly by offering boatloads of cash for content only to pull back, causing a lot of instability and belt-tightening. Accessibility of AI could lead to its negligent use, however, as companies cut corners and do not use it with intentionality. Tools like OpenAI's Sora can generate short video clips entirely from text prompts, but the product was criticized after release for its poor quality. Others from companies like Runway and Google have received better reception, though studios still face significant pushback from actors over their use.

Swedish Movie with AI English Dubbing to Screen in the US

Artificial intelligence (AI) is playing an ever-increased role in television and movie streaming. Thanks to AMC and AI company Flawless, AI is coming to theaters, too. Swedish science fiction movie Watch the Skies arrives in American theaters in May, with AI-powered editing that promises to match the actors' mouths with the English dub. AI dubbing, or "vubbing" (visual dubbing) as Flawless calls it, is not unique, although it is still relatively novel. While currently only available to enterprise customers, Adobe's Firefly AI tools can translate audio from one language to another with lip sync. "Make anyone appear as though they are speaking in another language by matching the mouth movements of newly translated audio," Adobe promises. As Variety reports, Watch the Skies will be presented by AMC with similar technology. Flawless AI is performing the work. Flawless is a major player in the professional video editing space and has worked with a wide range of partners, including Lionsgate, Netflix, Lenovo, the National Hockey League (NHL), Grubhub, Highdive, 1986, CNN, and more. The company specializes in dialogue replacement, what Flawless calls TrueSync. Flawless promises that Watch the Skies, known as UFO Sweden in Sweden, will deliver perfectly synced mouth movements with the English language dub, which is derived from voiceover work done by the original Swedish cast, making Flawless' AI tech SAG-AFTRA friendly. Variety spoke to Flawless co-founder Scott Mann last year about the ethics of AI-powered video editing and what syncing technology could mean for filmmakers. "Showing our materials to filmmakers, especially over the past year, they realize the potential from going to a local stage to a global stage. It's a huge opportunity to get your work out and it's been invigorating. They are so excited about showing their work in a wider audience, and especially in America," Mann said last year. "The Swedish language is a barrier when you want to reach out around the world," says Albin Pettersson of Crazy Pictures, the producer of Watch the Skies. "We were contacted by the guys at Flawless to take the movie and make a new version of it in English, but not a dubbed English version. They call it 'vubbing,' visual dub." "This technology enables things that we couldn't do before. Flawless and their technology gives us the opportunity to release the film for a much larger audience," says the film's writer and director, Victor Danell. "We can go from our small country to the whole world." AMC Theatres will show Watch the Skies on 100 screens in the 20 biggest markets in the United States. Flawless partnered with distributor XYZ Films to show Watch the Skies starting May 9.

Swedish Film 'Watch the Skies' Set for US Release With AI 'Visual Dubbing' - Decrypt

When Swedish UFO film "Watch the Skies" hits U.S. cinemas this May, audiences won't be able to tell that it wasn't made in English. The film is the first full theatrical release to showcase "visual dubbing" technology from AI firm Flawless, which enables actors' performances to be digitally lip-synced with foreign-language dubs. "Watch the Skies" follows rebellious teenager Denise, who teams up with a club of UFO watchers to solve the mystery of her missing father. The film was shot in Swedish, and released there under the title "UFO Sweden." However, for its U.S. release, the original cast have re-recorded their performances in English, with Flawless using its TrueSync machine learning technology to digitally alter their lip movements so that they sync up with the English dialogue. "Flawless and their technology gives us the opportunity to release the film for a much larger audience," said writer-director Victor Danell in a making-of featurette. The filmmakers stress that the use of the technique has "full endorsement from SAG," the actors' union, which went on strike in 2023 amid concerns over the "threat" posed by AI to the profession. "A lot of filmmakers and a lot of actors will be afraid of this technology at first, but we have the creative control, and to act out the film in English was a real exciting experience," said Danell, adding that, "It's still our movie, it's still the actors' performance, and that's the key part." In a 2023 presentation, Flawless co-founder Scott Mann explained that the company's TrueSync technology uses deep learning to create a volumetric 3D representation of the actors' faces throughout a film, which can then be altered to match the dubbed dialogue. TrueSync was previously used on the 2022 film "Fall" to remove swearing for a PG-13 edit, while another of the company's products, DeepEditor, enables an actor's performance to be extracted from one scene and applied to another scene, without the need for reshoots. The company has partnered with distributor XYZ Films to release a slate of features localized using TrueSync, including "Run Lola Run" director Tom Tykwer's upcoming "The Light," horror feature "Vincent Must Die," and South Korean film "Smugglers."