Openai Sora OpenAI Sora text-to-video release, feature introduction, and usage instructions

Official website: https://openai.com/sora


Video generation models as world simulators

We explore large-scale training of generative models on video data. Specifically, we jointly train text-conditional diffusion models on videos and images of variable durations, resolutions, and aspect ratios. We leverage a Transformer architecture that operates on spacetime patches of latent codes for videos and images. Our largest model, Sora, is capable of generating high-fidelity videos up to one minute long. Our results suggest that scaling video generation models is a promising path toward building general-purpose simulators of the physical world.

Today, red team members can use Sora to assess harms or risks in critical areas. OpenAI is also providing access to a number of visual artists, designers, and filmmakers to gather feedback on how to improve the model to be most helpful to creative professionals.

OpenAI is sharing its research progress early so that it can begin collaborating with people outside OpenAI and gathering feedback, giving the public insight into upcoming AI capabilities.

Sora is capable of generating complex scenes with multiple characters, specific types of motion, and accurate details of subjects and backgrounds. The model not only understands what the user asks for in the prompt, but also understands how these things exist in the physical world.

The model has a deep understanding of language, enabling it to accurately interpret prompts and generate compelling characters that express vibrant emotions. Sora can also create multiple shots within a single generated video, accurately preserving characters and visual style.


Weaknesses

The current model has weaknesses. It may struggle to accurately simulate the physics of complex scenes and may fail to understand specific instances of cause and effect. For example, a person may take a bite of a cookie, but afterward the cookie may not have a bite mark.
The model may also confuse spatial details of a prompt, such as confusing left and right, and may struggle to precisely describe events that unfold over time, such as following a specific camera trajectory.


Safety measures
Before using Sora in OpenAI products, we will take several important safety measures. We are working with red team members (domain experts in areas such as misinformation, hateful content, and bias), who will test the model adversarially.

We are also building tools to help detect misleading content, such as detection classifiers that can determine when Sora has generated a video. If we deploy the model in OpenAI products, we plan to include C2PA metadata in the future.

In addition to developing new technology to prepare for deployment, we are leveraging existing safety methods built for products using DALL·E 3, which also apply to Sora.

For example, in OpenAI products, our text classifiers will check and reject text input prompts that violate our usage policies, such as text input prompts requesting extreme violence, sexual content, hateful imagery, celebrity likenesses, or the IP of others. We have also developed robust image classifiers that check the frames of every generated video to help ensure it complies with our usage policies before being shown to users.


Usage policies

We will work with policymakers, educators, and artists around the world to understand their concerns and identify positive use cases for this new technology. Despite extensive research and testing, we cannot predict all the beneficial ways people will use our technology, nor all the ways people will misuse it. That is why we believe that learning from real-world use over time is a key component of creating and releasing increasingly safe AI systems.


Research technology

Sora is a diffusion model that starts generating a video from something that looks like static noise and then gradually transforms it by removing noise through multiple steps.

Sora is capable of generating an entire video at once or extending a generated video to make it longer. By providing the model with predictions for multiple frames at once, we solve a challenging problem: ensuring that the subject remains consistent even when it temporarily leaves the field of view. Similar to GPT models, Sora uses a Transformer architecture, unlocking superior scaling performance.

We represent videos and images as collections of smaller data units called patches, each similar to a token in GPT. By unifying the way we represent data, we can train diffusion Transformers on a broader range of visual data than before, spanning different durations, resolutions, and aspect ratios.

Sora builds on past research into DALL·E and GPT models. It uses the recaptioning technique from DALL·E 3, which involves generating highly descriptive captions for visual training data. As a result, the model is able to follow users' text instructions in generated videos more faithfully.

In addition to being able to generate videos solely from text instructions, the model can also take existing still images and generate videos from them, accurately animating the content of the image and paying attention to small details. The model can also take existing videos and extend them or fill in missing frames. See our technical report for more information.

Sora is a foundation for models that can understand and simulate the real world, and we believe this capability will be an important milestone toward achieving AGI.