We introduce Video Grounded Large Multi-modal Model (VideoGLaMM), a video large multimodal model, capable of pixel-level visual grounding, featuring an end-to-end alignment mechanism.
\nTo achieve fine-grained spatio-temporal alignment, we introduce a benchmark grounded conversation generation (GCG) dataset consisting of 38k grounded video-QA triplet pairs and 83k objects and roughly 671k fine-grained spatio-temporal masks.
\nWe also assess the performance of VideoGLaMM across diverse tasks spanning grounded conversation generation, visual grounding, and referring video segmentation, where it achieves state-of-the-art performance
\n","updatedAt":"2024-11-08T10:46:32.366Z","author":{"_id":"641fecb25bccaa42483db102","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/641fecb25bccaa42483db102/HPTuSuWyIBHgg9afcKZ7p.jpeg","fullname":"Shehan Munasinghe","name":"shehan97","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.7746431827545166},"editors":["shehan97"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/641fecb25bccaa42483db102/HPTuSuWyIBHgg9afcKZ7p.jpeg"],"reactions":[],"isReport":false}},{"id":"672debd6fb3aca41ad8fbf2e","author":{"_id":"641fecb25bccaa42483db102","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/641fecb25bccaa42483db102/HPTuSuWyIBHgg9afcKZ7p.jpeg","fullname":"Shehan Munasinghe","name":"shehan97","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false},"createdAt":"2024-11-08T10:45:42.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Project Page: https://mbzuai-oryx.github.io/VideoGLaMM/","html":"Project Page: https://mbzuai-oryx.github.io/VideoGLaMM/
\n","updatedAt":"2024-11-08T10:45:42.694Z","author":{"_id":"641fecb25bccaa42483db102","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/641fecb25bccaa42483db102/HPTuSuWyIBHgg9afcKZ7p.jpeg","fullname":"Shehan Munasinghe","name":"shehan97","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.35090935230255127},"editors":["shehan97"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/641fecb25bccaa42483db102/HPTuSuWyIBHgg9afcKZ7p.jpeg"],"reactions":[],"isReport":false}},{"id":"67300d8b9e631f353ac71877","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-11-10T01:34:03.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [PiTe: Pixel-Temporal Alignment for Large Video-Language Model](https://huggingface.co/papers/2409.07239) (2024)\n* [Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model](https://huggingface.co/papers/2409.13407) (2024)\n* [Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models](https://huggingface.co/papers/2410.03290) (2024)\n* [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://huggingface.co/papers/2409.19603) (2024)\n* [EAGLE: Egocentric AGgregated Language-video Engine](https://huggingface.co/papers/2409.17523) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- PiTe: Pixel-Temporal Alignment for Large Video-Language Model (2024) \n
- Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model (2024) \n
- Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models (2024) \n
- One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (2024) \n
- EAGLE: Egocentric AGgregated Language-video Engine (2024) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos
Abstract
VideoGLaMM, a Large Multimodal Model, achieves precise pixel-level grounding in videos using a dual vision encoder, spatio-temporal decoder, and tunable Visual-Language adapters, outperforming existing approaches on various multimodal tasks.
Fine-grained alignment between videos and text is challenging due to complex spatial and temporal dynamics in videos. Existing video-based Large Multimodal Models (LMMs) handle basic conversations but struggle with precise pixel-level grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed for fine-grained pixel-level grounding in videos based on user-provided textual inputs. Our design seamlessly connects three key components: a Large Language Model, a dual vision encoder that emphasizes both spatial and temporal details, and a spatio-temporal decoder for accurate mask generation. This connection is facilitated via tunable V-L and L-V adapters that enable close Vision-Language (VL) alignment. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. To enable fine-grained grounding, we curate a multimodal dataset featuring detailed visually-grounded conversations using a semiautomatic annotation pipeline, resulting in a diverse set of 38k video-QA triplets along with 83k objects and 671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded Conversation Generation, Visual Grounding, and Referring Video Segmentation. Experimental results show that our model consistently outperforms existing approaches across all three tasks.
Community
We introduce Video Grounded Large Multi-modal Model (VideoGLaMM), a video large multimodal model, capable of pixel-level visual grounding, featuring an end-to-end alignment mechanism.
To achieve fine-grained spatio-temporal alignment, we introduce a benchmark grounded conversation generation (GCG) dataset consisting of 38k grounded video-QA triplet pairs and 83k objects and roughly 671k fine-grained spatio-temporal masks.
We also assess the performance of VideoGLaMM across diverse tasks spanning grounded conversation generation, visual grounding, and referring video segmentation, where it achieves state-of-the-art performance
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- PiTe: Pixel-Temporal Alignment for Large Video-Language Model (2024)
- Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model (2024)
- Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models (2024)
- One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (2024)
- EAGLE: Egocentric AGgregated Language-video Engine (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper