https://huggingface.co/weizhiwang/Open-Qwen2VL\n","updatedAt":"2025-04-02T03:32:51.597Z","author":{"_id":"63d34004b734eaa4d4faeccf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/zf6d1p0GN8gsagi8N6y4V.jpeg","fullname":"Weizhi Wang","name":"weizhiwang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":21,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.892184317111969},"editors":["weizhiwang"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/zf6d1p0GN8gsagi8N6y4V.jpeg"],"reactions":[],"isReport":false}},{"id":"67ed033f0a67a8af26b42f75","author":{"_id":"644662145004f2cb3af08b27","avatarUrl":"/avatars/5f2af24c7410a5db46374d0b84fb479d.svg","fullname":"Avishai Elmakies","name":"avishai-elmakies","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false},"createdAt":"2025-04-02T09:28:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This looks great! We need to push more open-source and Compute-Efficient methods to train good models. We released a similar paper recently on training speech language models in a compute-constrained setting https://huggingface.co/papers/2502.15814 ","html":"
This looks great! We need to push more open-source and Compute-Efficient methods to train good models. We released a similar paper recently on training speech language models in a compute-constrained setting https://huggingface.co/papers/2502.15814
\n","updatedAt":"2025-04-02T09:28:31.720Z","author":{"_id":"644662145004f2cb3af08b27","avatarUrl":"/avatars/5f2af24c7410a5db46374d0b84fb479d.svg","fullname":"Avishai Elmakies","name":"avishai-elmakies","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9239227175712585},"editors":["avishai-elmakies"],"editorAvatarUrls":["/avatars/5f2af24c7410a5db46374d0b84fb479d.svg"],"reactions":[],"isReport":false}},{"id":"67ed0ae84a5c7f6759b7e0c9","author":{"_id":"62d648291fa3e4e7ae3fa6e8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png","fullname":"ben burtenshaw","name":"burtenshaw","type":"user","isPro":true,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":4319,"isUserFollowing":false},"createdAt":"2025-04-02T10:01:12.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"Open-Qwen2VL is interesting but is Table 1 correct in relation to Idefics? The models and datasets are available on the hub, and much of the code for datasets and models are on github.\n\n\n\nHere's a non-exhaustive list of open resources:\n- data creation https://github.com/huggingface/OBELICS\n- datasets https://huggingface.co/collections/HuggingFaceM4/obelics-6509a2ef647a3ea442ce2fbd\n- data paper https://huggingface.co/papers/2306.16527\n- models https://huggingface.co/collections/HuggingFaceM4/idefics-6509a1aaabdde5290e80b855\n","html":"Open-Qwen2VL is interesting but is Table 1 correct in relation to Idefics? The models and datasets are available on the hub, and much of the code for datasets and models are on github.
\nHere's a non-exhaustive list of open resources:
\n- \n
- data creation https://github.com/huggingface/OBELICS \n
- datasets https://huggingface.co/collections/HuggingFaceM4/obelics-6509a2ef647a3ea442ce2fbd \n
- data paper https://huggingface.co/papers/2306.16527 \n
- models https://huggingface.co/collections/HuggingFaceM4/idefics-6509a1aaabdde5290e80b855 \n
Hi, we apologize for the wrong information in Table 1. We update the Table 1 for both arxiv paper and project website.
\n","updatedAt":"2025-04-02T11:07:47.619Z","author":{"_id":"63d34004b734eaa4d4faeccf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/zf6d1p0GN8gsagi8N6y4V.jpeg","fullname":"Weizhi Wang","name":"weizhiwang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":21,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.79395991563797},"editors":["weizhiwang"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/zf6d1p0GN8gsagi8N6y4V.jpeg"],"reactions":[{"reaction":"🤗","users":["andito","burtenshaw"],"count":2}],"isReport":false,"parentCommentId":"67ed0ae84a5c7f6759b7e0c9"}}]},{"id":"67ed0d4d119f1c10a0d6fba4","author":{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","fullname":"Andres Marafioti","name":"andito","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":501,"isUserFollowing":false},"createdAt":"2025-04-02T10:11:25.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"All of the pre-training codebase is open as well: https://github.com/huggingface/smollm/tree/main/vision/m4\n","html":"All of the pre-training codebase is open as well: https://github.com/huggingface/smollm/tree/main/vision/m4
\n","updatedAt":"2025-04-02T10:11:25.256Z","author":{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","fullname":"Andres Marafioti","name":"andito","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":501,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8825057744979858},"editors":["andito"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg"],"reactions":[{"reaction":"👍","users":["vishaal27"],"count":1}],"isReport":false}},{"id":"67ed0d64754527bcbffeee8b","author":{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","fullname":"Andres Marafioti","name":"andito","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":501,"isUserFollowing":false},"createdAt":"2025-04-02T10:11:48.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"So, sequence packing scripts and data filtering techniques should also be open for idefics.","html":"So, sequence packing scripts and data filtering techniques should also be open for idefics.
\n","updatedAt":"2025-04-02T10:11:48.051Z","author":{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","fullname":"Andres Marafioti","name":"andito","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":501,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.909347653388977},"editors":["andito"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg"],"reactions":[{"reaction":"👍","users":["vishaal27"],"count":1}],"isReport":false}},{"id":"67ede5a04aa6f6bda1822201","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-04-03T01:34:24.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Should VLMs be Pre-trained with Image Data?](https://huggingface.co/papers/2503.07603) (2025)\n* [FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression](https://huggingface.co/papers/2502.18512) (2025)\n* [BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queries](https://huggingface.co/papers/2503.12446) (2025)\n* [LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning](https://huggingface.co/papers/2503.15621) (2025)\n* [M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance](https://huggingface.co/papers/2502.18778) (2025)\n* [OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models](https://huggingface.co/papers/2503.08686) (2025)\n* [HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding](https://huggingface.co/papers/2503.14694) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Should VLMs be Pre-trained with Image Data? (2025) \n
- FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression (2025) \n
- BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queries (2025) \n
- LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning (2025) \n
- M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance (2025) \n
- OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models (2025) \n
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2504.00595
Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources
Published on Apr 1, 2025
Abstract
Open-Qwen2VL is a 2B-parameter multimodal LLM pre-trained efficiently on image-text pairs with improved data filtering and sequence packing, outperforming state-of-the-art models on various benchmarks.
AI-generated summary
The reproduction of state-of-the-art multimodal LLM pre-training faces barriers at every stage of the pipeline, including high-quality data filtering, multimodal data mixture strategies, sequence packing techniques, and training frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic image resolution and multimodal sequence packing to significantly enhance pre-training efficiency. The training dataset was carefully curated using both MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based filtering methods, substantially improving data quality and training efficiency. The Open-Qwen2VL pre-training is conducted on academic level 8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista, indicating the remarkable training efficiency of Open-Qwen2VL. We open-source all aspects of our work, including compute-efficient and data-efficient training details, data filtering methods, sequence packing scripts, pre-training data in WebDataset format, FSDP-based training codebase, and both base and instruction-tuned model checkpoints. We redefine "fully open" for multimodal LLMs as the complete release of: 1) the training codebase, 2) detailed data filtering techniques, and 3) all pre-training and supervised fine-tuning data used to develop the model.
Community
This looks great! We need to push more open-source and Compute-Efficient methods to train good models. We released a similar paper recently on training speech language models in a compute-constrained setting https://huggingface.co/papers/2502.15814
Open-Qwen2VL is interesting but is Table 1 correct in relation to Idefics? The models and datasets are available on the hub, and much of the code for datasets and models are on github.
Here's a non-exhaustive list of open resources:
Hi, we apologize for the wrong information in Table 1. We update the Table 1 for both arxiv paper and project website.
All of the pre-training codebase is open as well: https://github.com/huggingface/smollm/tree/main/vision/m4
So, sequence packing scripts and data filtering techniques should also be open for idefics.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Should VLMs be Pre-trained with Image Data? (2025)
- FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression (2025)
- BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queries (2025)
- LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning (2025)
- M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance (2025)
- OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models (2025)
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 3
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2504.00595 in a Space README.md to link it from this page.