In February, we published our first story about SORA; OpenAI had just released the first SORA clips, and at the time we described it as the DALL·E of video. SORA is a diffusion model that generates videos much longer and more coherent than any competitor. By having the model predict many frames at once, they solved the challenging problem of keeping a subject consistent even when it temporarily disappears from view. SORA can generate a complete video up to one minute long in one go. At the time, OpenAI also released technical notes indicating that it could (in the future) extend generated videos to make them longer or seamlessly blend two videos.
Over the past few weeks, selected production teams have been given limited access to SORA. Most notable among them is the Shy Kids team, who made the SORA short Air Head. Sidney Leeder produced the film. Walter Woodman is the writer and director, while Patrick Cederberg handled post-production. This Toronto team has been nicknamed “punk rock Pixar,” and their work has been Emmy-nominated and shortlisted for the Oscars. This week, we sat down with Patrick for a long conversation about the current state of SORA.
Shy Kids is a Canadian production company known for its eclectic, innovative approach to media production. Originally a collective of creatives from different fields such as film, music, and television, Shy Kids gained recognition for its unique storytelling style and engaging content. The company often explores adolescence, social anxiety, and the complexities of modern life while maintaining a distinctive whimsical and sincere tone. Their work demonstrates a keen eye for visual storytelling and is often closely integrated with original music, making their pieces resonant and memorable. By embracing new AI technology and creativity, Shy Kids has successfully carved out a niche and pushed what is possible.
SORA: mid-April 2024.
SORA is in development and is being actively improved through feedback from teams such as Shy Kids, but here is how it currently works. It is important to recognize that SORA works almost at a pre-alpha stage. It has not been released, nor is it in beta.
“Trying it was a lot of fun,” Patrick commented. “It’s a very powerful tool, and we’re already thinking about how to integrate it into our existing pipeline. But I think for any generative AI tool, control is still the most desirable and elusive thing right now.”
[baike]User interface[/baike]The user interface allows artists to enter text prompts; OpenAI’s ChatGPT then converts them into longer strings that trigger clip generation. At present, there are no other inputs; it is not yet multimodal. This matters because although SORA is praised for object consistency within a shot, nothing helps match anything in the first shot with the second. Even if you run the same prompt a second time, the result will differ. “The closest we can get is to be super-descriptive in the prompt,” Patrick explained. “Explaining the character’s clothing and the type of balloon is how we maintained consistency, because from one shot to the next/generation to generation, there isn’t yet a set of features that fully controls consistency.”
The individual clips are excellent, and the technology they represent is amazing, but how those clips can be used depends on your understanding of implicit or explicit shot generation. Suppose you ask SORA for a long tracking shot in a kitchen with a banana on the table. In that case, it will rely on an implicit understanding of “banana-ness” to generate a video showing a banana. Through training data, it “learns” implicit aspects of banana-ness: for example, “yellow,” “curved,” “darker at the ends,” and so on. It does not have an actual recorded image of a banana. It does not have a “banana stock library” database; it has a much smaller compressed hidden or “latent space” for knowing what a banana is. Each time it runs, it displays another interpretation of that latent space. Your prompt taps into the implicit understanding of banana-ness.
[baike]The problem of over-association[/baike]For Air Head, scenes were made by generating multiple videos to form an approximate script, but there was no explicit way to keep the actual yellow balloon head consistent across shots. Sometimes, when the team prompted for a yellow balloon, it was not even yellow. Other times, a face was embedded in it, or a face seemed to be drawn on the front of the balloon. Because many balloons have strings, the balloon-head character (nicknamed Sonny, the balloon man) often ended up with a string tied to the front of his shirt. Because it implicitly associates strings with balloons, the string needed to be removed in post.
An unwanted face on the balloon from the original SORA output.
[baike]Some problems encountered[/baike]Air Head used only SORA-generated footage, but most of it was graded, processed, and stabilized, and all of it was upscaled or compressed. The clips the team used were generated at lower resolution and then compressed using SORA or AI tools outside OpenAI. “You can get to 720p,” Patrick explained. “I believe there is a 1080 capability, but it takes a while (to render). We made all the Air Head clips at 480 resolution for speed, then used Topaz for upright processing.”
Prompt “time”: slot machine.
The original prompt automatically expands, but it is also displayed along the timeline. “You can go into those larger keyframes and start adjusting information according to the variation you want to generate.” Patrick explained, “During actual generation, there is a little temporal control over where these different actions happen, but it is not precise... It’s a bit like shooting in the dark—like a slot machine—as to whether it actually does these things at this time.” Of course, Shy Kids was using the earliest prototype, and SORA is still being developed.
In addition to choosing resolution, SORA also allows users to choose aspect ratio, such as portrait or landscape (or square). This was useful in the shot tilting up from Sonny’s jeans to his balloon head. Unfortunately, SORA could not natively render such a movement; it always wants the main focus of the shot—the balloon head—to be in frame. So the team rendered the shot in portrait mode and then manually created the upward pan in post by cropping.
[baike]Shot control issues[/baike]For many genAI tools, a valuable source of information is metadata that accompanies training data, such as camera metadata. For example, if you train on still photos, camera metadata provides shot size, aperture, and many other key pieces of information for the model to train on. For film shots, concepts such as “tracking,” “pan,” “tilt,” or “push in” are not terms or concepts captured by metadata. Object permanence is crucial to shotmaking, and so is being able to describe a shot, which Patrick notes was not something SORA initially had. “Nine different people would have nine different ideas about how to describe a shot in a film scene. And, before artists got their hands on the tool, (OpenAI) researchers weren’t really thinking like filmmakers.” Shy Kids knew they only had early access, but “the initial version around camera angles was a bit random.” Whether SORA actually records the prompt request or understands it is unknown, because the researchers were focused simply on image generation. Shy Kids was almost shocked at how surprised OpenAI was by the request. “But I suppose when you’re just researchers and not thinking about how storytellers would use it... SORA is improving, but I still think control isn’t good enough. You can type ‘camera pan,’ and I think 6 out of 10 times you’ll succeed.” This is not a unique problem; almost all major video genAI companies face the same issue. Runway AI is perhaps the most advanced in terms of providing a UI for describing camera movement, but Runway’s quality and rendered clip length are not as good as SORA’s.
[baike]Rendering time issues[/baike]Clips can be rendered for different durations, such as 3 seconds, 5 seconds, 10 seconds, 20 seconds, and up to 1 minute. Render time varies depending on the time of day and cloud usage demand. “Generally, each render takes about 10 to 20 minutes,” Patrick recalled. “In my experience, the render duration I choose has little impact on render time. If the render is 3 to 20 seconds, the render time won’t differ much between 10 and 20 minutes. We would usually do this because if you have the full 20 seconds, you want more opportunities to cut/edit, which increases the chances of getting something good.”
[baike]Many detail issues[/baike]Although all images were generated in SORA, the balloon still required a lot of post work. In addition to isolating the balloon so it could be recolored, Sonny’s face sometimes appeared on the balloon, as if his face had been drawn on with a marker, and this was removed in AfterEffects. Other similar artifacts were also frequently removed.
[baike]300:1 shooting ratio[/baike]Shy Kids’ approach was to do post-production and editing like a documentary, with a large amount of footage from which you can weave a story, rather than shooting strictly to a script. The short has a script, but the team needed to be flexible. “It was just shooting a ton of footage, then trying to edit it into voiceover in an interesting way,” Patrick recalled.
For the minute-and-a-half of footage that ended up in the film, Patrick estimates they “generated hundreds of generations, each 10 to 20 seconds.” He added: “I’m bad at math, but I guess the ratio of raw footage to the final product is about 300:1.”
[baike]Slow-motion time unification[/baike]In Air Head, the team did not composite multiple shots together. For example, the shot of the balloon floating over the racetrack was generated almost as seen, in one shot. However, they are working on a new film that mixes multiple shots and composites them into one clip.
Interestingly, many Air Head clips looked as if they were shot in slow motion, even though the prompt did not ask for that. The reason this happened is unknown, so many clips had to be retimed to look like real-time footage. Obviously, this is easier to do than the reverse operation of slowing down fast action, but inferring this from training data seems a bit strange. “I don’t know why, but it does seem like a lot of clips are at 50% to 75% speed,” he added. “So quite a bit of time went into adjusting the timing so it didn’t feel like a big slow-motion project.”
[baike]Handling lighting[/baike]Shy Kids used the term “35mm film” as a keyword in their prompts and found that prompting 35mm gave them the level of consistency they sought. “If we needed high contrast, we could say high contrast, and saying key light usually got us close,” Patrick said. “We still had to fully color grade it, and we made our own digital film look, and we applied grain and flicker to blend them together.” There were no other options, such as masks or depth passes.
[baike]Copyright and practical needs[/baike]OpenAI tries to be respectful and does not allow the generation of copyright-infringing material or images that appear to come from someone else. For example, if you prompt “35mm film in a future spaceship, a man walking forward holding a lightsaber,” SORA will not allow the clip to be generated because it is too close to Star Wars. But Shy Kids accidentally encountered this problem in early tests. Patrick recalled that when they first sat down just to test SORA, “We had a shot from behind a character; it was somewhat like Aronofsky’s tracking shot. I think my brain was just being dumb because I was tired, but I put in ‘Aronofsky-style shot’ and it got rejected,” he recalled. The Hitchcock zoom is another thing that now comes through osmosis (a technical term), but SORA will reject that prompt for copyright reasons.
[baike]Voiceover issues[/baike]Shy Kids is known not only for strong visual skills but also for strong audio skills. The music in the short is their own. “It’s a song from our previous catalog, and we decided almost immediately to use it because the song is called ‘Wind,’” Patrick said. “We all loved it.”
Patrick himself is the voice of Sonny. “Sometimes we would feel that the film’s rhythm needed another beat. So I would write another line, record it, and come up with more SORA generations, which is another powerful use of the tool in post-production: when you’re stuck and need to fill a gap, it’s a great way to start brainstorming, just spit out clips and see what you can use to fill a rhythm problem.”
[baike]Lessons learned[/baike]SORA is excellent; the Shy Kids team made Air Head with only 3 people in about 1.5 to 2 weeks. The team has already started working on a brilliant, self-aware, and possibly ironic sequel. “The sequel will tell, in a news style, about the balloon man Sonny and his reaction to fame and his subsequent falling out with the world,” Patrick said. “We’re exploring new techniques!” The team wants to be more technical in their experiments, integrating SORA elements into AE composites with real live-action footage and using SORA as a supplemental VFX tool.
SORA is very new, and the basic framework OpenAI outlined and demonstrated for SORA has not yet been used in the early tests. Whether the current form of SORA will be released soon is uncertain, but it has made incredible progress in a specific type of implicit image generation. For high-end projects, it may still take some time to reach the level of specificity directors need. For many other projects, it will be more than “close enough” while also delivering stunning images. Air Head still required a great deal of editing and human guidance to make this compelling and interesting narrative film. “I just think people have to treat SORA as a real part of their pipeline; but if they don’t want to be involved in anything like that, that’s fine too.”
|NIUBOYI
( 京ICP备17025393号 )|网站地图
