OpenAI revealed Sora to the world on February 15, 2024 by sharing a handful of remarkable AI generated videos and a research paper on X. Sora wasn't the first artificial intelligence video model, but it was the first to show such high levels of consistency, duration and photo realism. While the output seems impressive, so far only videos generated by OpenAI staff have been shared on either X or TikTok, although some were made with prompts suggested by fans.
The technology behind Sora is an adapted version of the models built for DALL-E 3, OpenAI's generative image platform but with additional features for fine-tuned control. Sora is a diffusion transformer model, that is it marries the type of image generation model behind Stable Diffusion with the token-based generators powering ChatGPT. A video is generated in a latent space and "denoised," or formed in 3D patches and then put through a video decompressor to turn into a standard, human viewable output.
OpenAI says it trained its model on publicaly available videos, public domain content and copyrighted videos where it had purchased the licence in advance. It hasn't said exactly how many videos went into the training data and is unlikely to ever reveal that information. It is thought to be in the millions. The company used a video-to-text engine to create captions and labels from ingested video files to further fine-tune Sora on real-world content. Rumors and speculation suggest that OpenAI also made use of synthetic video content, such as that generated using Unreal Engine 5 as this would also give it information on the physics of the worlds inside the video clips it ingested.
OpenAI hasn't set a release data for Sora yet, but the CTA Mira Murati says it will come out sometime in 2024, and possibly before the summer. When released it will be available and priced similarly to OpenAI's image generation model DALL-E, likely integrated into the premium version of ChatGPT.
Reference: OpenAI Sora: Everything you need to know By Ryan Morrison last updated March 14, 2024