So generating coherent videos spanning multiple scenes from text descriptions is hard with AI right now. You can make short clips easily but, smoothly transitioning across diverse events and maintaining continuity is the hard part.
\nIn this paper from UNC Chapel Hill, the authors propose VIDEODIRECTORGPT, a two-stage framework attempting to address multi-scene video generation:
\nHere are my highlights from the paper:
\n- \n
- Two-stage approach: language model generates detailed \"video plan\", then video generation module renders scenes based on plan \n
- Video plan contains multi-scene descriptions, entities/layouts, backgrounds, consistency groupings - guides downstream video generation \n
- Video generation module called Layout2Vid trained on images, adds spatial layout control and cross-scene consistency to existing text-to-video model \n
- Experiments show improved object layout/control in single scene videos vs baselines \n
- Multi-scene videos display higher object consistency across scenes compared to baselines \n
- Competitive open-domain video generation performance maintained \n
The key innovation seems to be using a large language model to plot detailed video plans to guide overall video generation. And the video generator Layout2Vid adds better spatial and temporal control through some clever tweaks.
\nYou can read my full summary here.
\n","updatedAt":"2023-09-27T18:02:45.583Z","author":{"_id":"6486638da4cf2081f20c40ec","avatarUrl":"/avatars/0bc16a7447cd71ac18828a678313bd83.svg","fullname":"Mike Young","name":"mikelabs","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8050872087478638},"editors":["mikelabs"],"editorAvatarUrls":["/avatars/0bc16a7447cd71ac18828a678313bd83.svg"],"reactions":[{"reaction":"🤯","users":["mikelabs"],"count":1}],"isReport":false}},{"id":"651ac8ddf0354540aa1d1a4f","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2023-10-02T13:42:53.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation](https://huggingface.co/papers/2309.03549) (2023)\n* [StoryBench: A Multifaceted Benchmark for Continuous Story Visualization](https://huggingface.co/papers/2308.11606) (2023)\n* [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://huggingface.co/papers/2309.15103) (2023)\n* [Empowering Dynamics-aware Text-to-Video Diffusion with Large Language Models](https://huggingface.co/papers/2308.13812) (2023)\n* [VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation](https://huggingface.co/papers/2309.00398) (2023)\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation (2023) \n
- StoryBench: A Multifaceted Benchmark for Continuous Story Visualization (2023) \n
- LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023) \n
- Empowering Dynamics-aware Text-to-Video Diffusion with Large Language Models (2023) \n
- VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation (2023) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n","updatedAt":"2023-10-02T13:42:53.033Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6817718744277954},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"68d87aad0576f6166c166c2e","author":{"_id":"6503fbd10905dd866fdf51b7","avatarUrl":"/avatars/70c2e26b939dc173e59d676d3c2c5e40.svg","fullname":"Ivan","name":"Ivankilin","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2025-09-28T00:00:45.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"A curious alien with smooth, pale blue skin, large reflective silver eyes, and a slender frame arrives in a quiet, sunlit suburban neighborhood on Earth. He finds a **raincoat** on a bench. Puts it on — too big, sleeves drag. Zips it all the way up. Stands tall. The wind fills it like a balloon. He floats slightly, arms wide — like a human-shaped kite.\nStyle: Realistic, cinematic, 4K, natural lighting, subtle humor, gentle music in the background. Focus on facial expressions and precise movements. Придумай что-нибудь еще и улучши Промт нюпод свою систему","html":"A curious alien with smooth, pale blue skin, large reflective silver eyes, and a slender frame arrives in a quiet, sunlit suburban neighborhood on Earth. He finds a raincoat on a bench. Puts it on — too big, sleeves drag. Zips it all the way up. Stands tall. The wind fills it like a balloon. He floats slightly, arms wide — like a human-shaped kite.
Style: Realistic, cinematic, 4K, natural lighting, subtle humor, gentle music in the background. Focus on facial expressions and precise movements. Придумай что-нибудь еще и улучши Промт нюпод свою систему
VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
Abstract
VideoDirectorGPT combines LLMs for video content planning with a layout-controlled video generator, enabling temporally consistent multi-scene video generation from text prompts.
Although recent text-to-video (T2V) generation methods have seen significant advancements, most of these works focus on producing short video clips of a single event with a single background (i.e., single-scene videos). Meanwhile, recent large language models (LLMs) have demonstrated their capability in generating layouts and programs to control downstream visual modules such as image generation models. This raises an important question: can we leverage the knowledge embedded in these LLMs for temporally consistent long video generation? In this paper, we propose VideoDirectorGPT, a novel framework for consistent multi-scene video generation that uses the knowledge of LLMs for video content planning and grounded video generation. Specifically, given a single text prompt, we first ask our video planner LLM (GPT-4) to expand it into a 'video plan', which involves generating the scene descriptions, the entities with their respective layouts, the background for each scene, and consistency groupings of the entities and backgrounds. Next, guided by this output from the video planner, our video generator, Layout2Vid, has explicit control over spatial layouts and can maintain temporal consistency of entities/backgrounds across scenes, while only trained with image-level annotations. Our experiments demonstrate that VideoDirectorGPT framework substantially improves layout and movement control in both single- and multi-scene video generation and can generate multi-scene videos with visual consistency across scenes, while achieving competitive performance with SOTAs in open-domain single-scene T2V generation. We also demonstrate that our framework can dynamically control the strength for layout guidance and can also generate videos with user-provided images. We hope our framework can inspire future work on better integrating the planning ability of LLMs into consistent long video generation.
Community
Awesome, definitely want to try it.
where is the code , chachi?
Alrighty this one looks cool! @froilo here's the repo: https://github.com/HL-hanlin/VideoDirectorGPT
So generating coherent videos spanning multiple scenes from text descriptions is hard with AI right now. You can make short clips easily but, smoothly transitioning across diverse events and maintaining continuity is the hard part.
In this paper from UNC Chapel Hill, the authors propose VIDEODIRECTORGPT, a two-stage framework attempting to address multi-scene video generation:
Here are my highlights from the paper:
- Two-stage approach: language model generates detailed "video plan", then video generation module renders scenes based on plan
- Video plan contains multi-scene descriptions, entities/layouts, backgrounds, consistency groupings - guides downstream video generation
- Video generation module called Layout2Vid trained on images, adds spatial layout control and cross-scene consistency to existing text-to-video model
- Experiments show improved object layout/control in single scene videos vs baselines
- Multi-scene videos display higher object consistency across scenes compared to baselines
- Competitive open-domain video generation performance maintained
The key innovation seems to be using a large language model to plot detailed video plans to guide overall video generation. And the video generator Layout2Vid adds better spatial and temporal control through some clever tweaks.
You can read my full summary here.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation (2023)
- StoryBench: A Multifaceted Benchmark for Continuous Story Visualization (2023)
- LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023)
- Empowering Dynamics-aware Text-to-Video Diffusion with Large Language Models (2023)
- VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
A curious alien with smooth, pale blue skin, large reflective silver eyes, and a slender frame arrives in a quiet, sunlit suburban neighborhood on Earth. He finds a raincoat on a bench. Puts it on — too big, sleeves drag. Zips it all the way up. Stands tall. The wind fills it like a balloon. He floats slightly, arms wide — like a human-shaped kite.
Style: Realistic, cinematic, 4K, natural lighting, subtle humor, gentle music in the background. Focus on facial expressions and precise movements. Придумай что-нибудь еще и улучши Промт нюпод свою систему
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper