OpenAI Sora has a beautiful user interface. The quality of its video generation, however, often fails to impress.
On 9 December, 2024, OpenAI finally released Sora, its AI model for video generation. First teased the previous February, Sora was by far the year's most anticipated video generator. After all, OpenAI's ChatGPT is the most popular AI chatbot on the planet, and Dall-E is among the best image generators. If anyone could set a new high bar, surely it would be OpenAI?
But the release of Sora fell short of expectations.
"We were so hyped up for Sora. Then when it came out, we were all like 'hmm...I'm not so sure.' The other tools caught up," said Chrissie Cremers, co-founder of Aigency Amsterdam. She's hopeful the tool will live up to OpenAI's reputation for updating models frequently. For now, though, Sora is frustratingly inconsistent.
The reasons for its inconsistency relate to broader problems that challenge most generative AI systems. Video generators, like image generators, typically use a diffusion model trained to create an image from random noise. But video generators face the added hurdle of generating a series of images and presenting them in a sequence. That requires deeper knowledge about how the world works.
Consider a game of baseball. Does the pitcher throw the ball towards home plate, or does the batter throw it at the pitcher's mound? Though it's a simple question for anyone familiar with the sport, answering it also requires an intuitive understanding of linear time and real-world physics that video generators find difficult. And so when an AI makes a video of a baseball play, maybe the catcher throws the pitch, or maybe the ball is thrown before the pitcher moves. Or maybe the ball screams at unreal speed in a random direction.
While Sora didn't live up to expectations, OpenAI's competitors didn't sit still in 2024, and many released both new foundation models and new features. These include models that provide higher-resolution outputs, lip-syncing of virtual characters, and fun visual effects. Here are five generators that you can start tinkering with right now.
Runway released its first foundation video generation model in 2023 and immediately established itself as a go-to option for creatives. Its strength is the visual quality of is generation in shorter shots with slow motion. Like Sora, it can struggle with fast motion but does better when asked to generate slow, cinematic shots of static or slow-moving objects.
The company also has an impressive video-to-video tool called Act One. Designed for "expressive character performances," it converts video of an actor's real-world performance into an AI animated character. Ross Symons, co-founder of creative AI company ZenRobot, said Runway is "very good" at lip syncing, which is critical to a convincing result.
Kling.ai is an AI video generation model from Kuaishou Technology, a Chinese company best known for its short-form social media app, Kwai, which has over 200 million users. The company is now making inroads in AI with its own foundation video generation model, Kling.
Symons praised Kling's ability to generate convincing movement in more demanding prompts. "I find that it has the ability to create motion that is a lot more natural, in terms of humans moving, or an animal moving," he said. Kuaishou released its latest model, Kling 1.6, on 19 December, 2024. Though available directly from the company's website, Kling.ai, the model can also be accessed through some third-party partners, such as Krea.
Luma AI is a small San Francisco start-up that trains foundation models. Its video generation model, Dream Machine, has carved out a reputation as a go-to tool for discerning creatives. Its following is based less on the quality of its video model (though it's still competitive) and more on its long list of useful features, which includes AI storyboards, image and video modification, and reference styles of more consistent results.
The most notable feature released in 2024, however, was keyframes. The feature lets users prompt the model with a start and end frame, providing more creative control. Symons, who has a background in animation, said the workflow felt natural and allowed for "subtle transitions between frames." Luma's advantage here might not hold, though, as competitors are beginning to add the feature.
Hailuo is a foundation video generation model from Minimax, a Shanghai-based company backed by giants like Alibaba and MiHoYo. Hailuo has earned praise for high-quality results with convincing action. "It has some good motion, like fighting, which is interesting for filmmakers," said Cremers.
This also highlights how models from companies based in the United States and China often differ in their approach to video generation.
OpenAI's Sora may sometimes generate videos of fights, action scenes, or well-known pop culture figures, but these prompts often lead to a content policy violation. In other cases, the generated video seems intentionally naïve about the prompt (ask Sora to generate a video of Brad Pitt, for example, and the results show no obvious resemblance). Hailuo seems less discerning, though it will still refuse particularly explicit and violent prompts.
All the video generation models discussed so far have something in common: they skew towards a slick, professional detailed style. That's often desirable, but what if you want something a bit more casual? That's where Pika steps in.
"I love Pika's social first strategy," said Cremers. "It's the model for social media, which is a very smart move." Pika generated buzz in October of 2024 with Pikaffects, a tool that creates short, sharable clips centered around special effects like squishing, melting, or inflating an object in a photo. Pika's most popular viral video -- a mundane toilet that suddenly does very strange things -- received over 19 million views on TikTok, and several more racked up millions.
Cremers and Symons both mentioned another essential tool for AI video generation: Midjourney.
No, you haven't missed anything. Midjourney's video generation tool, though said to be underway, remains unreleased. However, Midjourney is the preferred image generation tool for many AI creatives, and the images generated by Midjourney can be used for an image-to-video or keyframe prompt.
Topaz Labs' Video AI is another tool worth mention. It doesn't generate videos but instead uses AI to upscale them. That's valuable for professional work, as AI video generators tend to output video at lower resolutions (between 360p and 1080p).
The importance of Midjourney and Topaz Labs' Video AI highlights features video generators are likely to pursue in 2025. While better motion, consistency, and realistic physics are obvious areas where AI video can improve, there's also room for them to compete with better storyboarding, higher output resolutions, and better control over camera movement.
