Given complex referring expressions in natural language, VIDEOMOLMO demonstrates improved spatio-temporal reasoning in visual grounding. By decomposing the visual grounding task into
sequential steps—pointing (denoted by star) followed by generating masks (in red) -VIDEOMOLMO
produces more accurate and coherent segmentation masks compared to prior approaches.\n","updatedAt":"2025-06-18T03:01:41.142Z","author":{"_id":"656864e12d73834278a8dea7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg","fullname":"Ahmed Heakl","name":"ahmedheakl","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":61,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8024119138717651},"editors":["ahmedheakl"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg"],"reactions":[],"isReport":false}},{"id":"68522c3157987e83f5276876","author":{"_id":"656864e12d73834278a8dea7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg","fullname":"Ahmed Heakl","name":"ahmedheakl","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":61,"isUserFollowing":false},"createdAt":"2025-06-18T03:02:09.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"\n\nVideoMolmo Architecture. The visual encoder extracts multi-crop features from the current frame and the past l frames. These temporal features provide contextual cues and are processed by the Temporal Module M via multi-head cross-attention, where the query comes from the current frame, and key and value from the mean of previous frames. The output is fused with the original features to enrich temporal cues while preserving the spatial details of the current frame. The combined visual-textual representations are then passed to the LLM to predict grounded points. These points are converted into masks using our Bidirectional Temporal Mask Fusion module, ensuring temporally consistent pixel-level grounding.","html":"
VideoMolmo Architecture. The visual encoder extracts multi-crop features from the current frame and the past l frames. These temporal features provide contextual cues and are processed by the Temporal Module M via multi-head cross-attention, where the query comes from the current frame, and key and value from the mean of previous frames. The output is fused with the original features to enrich temporal cues while preserving the spatial details of the current frame. The combined visual-textual representations are then passed to the LLM to predict grounded points. These points are converted into masks using our Bidirectional Temporal Mask Fusion module, ensuring temporally consistent pixel-level grounding.
VIDEOMOLMO annotation pipeline: We construct point-level supervision from framelevel masks using a semi-automatic process. For each frame, k points are sampled on the mask and
passed to SAM2 to generate candidate masks. The point with the highest-IoU candidate mask (w.r.t. ground truth) is selected as the optimal annotation.
More examples.
\n\n\n\n\n\n","updatedAt":"2025-06-18T03:03:00.580Z","author":{"_id":"656864e12d73834278a8dea7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg","fullname":"Ahmed Heakl","name":"ahmedheakl","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":61,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5401440262794495},"editors":["ahmedheakl"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/656864e12d73834278a8dea7/sfAWS2eyPtFHb_2GZIypp.jpeg"],"reactions":[],"isReport":false}},{"id":"6853694233d67d5d81db3cde","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-06-19T01:34:58.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving](https://huggingface.co/papers/2506.06218) (2025)\n* [SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models](https://huggingface.co/papers/2505.18812) (2025)\n* [Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency](https://huggingface.co/papers/2506.07497) (2025)\n* [HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios](https://huggingface.co/papers/2506.09650) (2025)\n* [FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion](https://huggingface.co/papers/2506.01111) (2025)\n* [DaMO: A Data-Efficient Multimodal Orchestrator for Temporal Reasoning with Video LLMs](https://huggingface.co/papers/2506.11558) (2025)\n* [SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes](https://huggingface.co/papers/2506.01558) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (2025) \n
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (2025) \n
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (2025) \n
- HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios (2025) \n
- FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion (2025) \n
- DaMO: A Data-Efficient Multimodal Orchestrator for Temporal Reasoning with Video LLMs (2025) \n
- SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
VideoMolmo: Spatio-Temporal Grounding Meets Pointing
Abstract
VideoMolmo, a multimodal model incorporating a temporal attention mechanism and SAM2 for mask fusion, enhances spatio-temporal pointing accuracy and reasoning capabilities in diverse real-world scenarios.
Spatio-temporal localization is vital for precise interactions across diverse domains, from biological research to autonomous navigation and interactive interfaces. Current video-based approaches, while proficient in tracking, lack the sophisticated reasoning capabilities of large language models, limiting their contextual understanding and generalization. We introduce VideoMolmo, a large multimodal model tailored for fine-grained spatio-temporal pointing conditioned on textual descriptions. Building upon the Molmo architecture, VideoMolmo incorporates a temporal module utilizing an attention mechanism to condition each frame on preceding frames, ensuring temporal consistency. Additionally, our novel temporal mask fusion pipeline employs SAM2 for bidirectional point propagation, significantly enhancing coherence across video sequences. This two-step decomposition, i.e., first using the LLM to generate precise pointing coordinates, then relying on a sequential mask-fusion module to produce coherent segmentation, not only simplifies the task for the language model but also enhances interpretability. Due to the lack of suitable datasets, we curate a comprehensive dataset comprising 72k video-caption pairs annotated with 100k object points. To evaluate the generalization of VideoMolmo, we introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving, Video-GUI Interaction, and Robotics. We also evaluate our model on Referring Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to existing models, VideoMolmo substantially improves spatio-temporal pointing accuracy and reasoning capability. Our code and models are publicly available at https://github.com/mbzuai-oryx/VideoMolmo.
Community
VideoMolmo is a a large multimodal model tailored for fine-grained spatio-temporal pointing conditioned on textual descriptions. Building upon the Molmo architecture, VideoMolmo incorporates a temporal module utilizing an attention mechanism to condition each frame on preceding frames, ensuring temporal consistency. Additionally, our novel temporal mask fusion pipeline employs SAM2 for bidirectional point propagation, significantly enhancing coherence across video sequences. This two-step decomposition i.e., first using the LLM to generate precise pointing coordinates, then relying on a sequential mask-fusion module to produce coherent segmentation, not only simplifies the task for the language model but also enhances interpretability. Due to the lack of suitable datasets, we curate a comprehensive dataset comprising 72k video-caption pairs annotated with 100k object points. To evaluate the generalization of VideoMolmo, we introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving, Video-GUI Interaction, and Robotics. We also evaluate our model on Referring Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to existing models, VideoMolmo substantially improves spatio-temporal pointing accuracy and reasoning capability.
Given complex referring expressions in natural language, VIDEOMOLMO demonstrates improved spatio-temporal reasoning in visual grounding. By decomposing the visual grounding task into
sequential steps—pointing (denoted by star) followed by generating masks (in red) -VIDEOMOLMO
produces more accurate and coherent segmentation masks compared to prior approaches.
VideoMolmo Architecture. The visual encoder extracts multi-crop features from the current frame and the past l frames. These temporal features provide contextual cues and are processed by the Temporal Module M via multi-head cross-attention, where the query comes from the current frame, and key and value from the mean of previous frames. The output is fused with the original features to enrich temporal cues while preserving the spatial details of the current frame. The combined visual-textual representations are then passed to the LLM to predict grounded points. These points are converted into masks using our Bidirectional Temporal Mask Fusion module, ensuring temporally consistent pixel-level grounding.
VIDEOMOLMO annotation pipeline: We construct point-level supervision from framelevel masks using a semi-automatic process. For each frame, k points are sampled on the mask and
passed to SAM2 to generate candidate masks. The point with the highest-IoU candidate mask (w.r.t. ground truth) is selected as the optimal annotation.
More examples.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (2025)
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (2025)
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (2025)
- HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios (2025)
- FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion (2025)
- DaMO: A Data-Efficient Multimodal Orchestrator for Temporal Reasoning with Video LLMs (2025)
- SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 1
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper