Diffusion models are designed to create specific images based on user inputs, typically through text prompts. To enhance the image generation process, these models are often paired with large language models (LLMs), which provide guidance by evaluating how well the generated images match the textual descriptions. This pairing helps refine the output, steering the diffusion model towards images that align closely with the given prompts.
The connection between text and images utilized by these LLMs is not arbitrary. Most contemporary text-to-image and text-to-video models are trained on extensive datasets containing billions of text-image or text-video pairs sourced from the internet. This process has raised concerns among content creators regarding the implications of using such data without consent, as the results reflect an online representation of the world that may include biases and inappropriate content.
While diffusion models are commonly associated with image generation, their applications extend to various data types including audio and video. For video generation, the models must process sequences of images known as frames, rather than focusing on individual images.
To efficiently handle the substantial computational requirements of video generation, many diffusion models implement a method called latent diffusion. This technique operates within a “latent space,” where raw data from video frames and text prompts are transformed into a compressed mathematical representation. This representation retains only the essential features of the data while discarding less critical information, which streamlines the processing demands.
This concept parallels how videos are streamed over the internet; they are transmitted in a compressed format to reduce latency, and then decompressed by the receiving device for viewing.
Source: https://www.technologyreview.com/2025/09/12/1123562/how-do-ai-models-generate-videos/