https://huggingface.co/spaces/MMInstruction/VL-RewardBench\n","updatedAt":"2024-11-27T10:52:41.783Z","author":{"_id":"6038d6d0612f5eef3cc05ea9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg","fullname":"Lei Li","name":"tobiaslee","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":24,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6125556230545044},"editors":["tobiaslee"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg"],"reactions":[],"isReport":false}},{"id":"6747c89970bb973bca41dd08","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-11-28T01:34:17.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment](https://huggingface.co/papers/2410.09421) (2024)\n* [MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models](https://huggingface.co/papers/2410.09733) (2024)\n* [LLaVA-Critic: Learning to Evaluate Multimodal Models](https://huggingface.co/papers/2410.02712) (2024)\n* [MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark](https://huggingface.co/papers/2410.11538) (2024)\n* [Vision-Language Models Can Self-Improve Reasoning via Reflection](https://huggingface.co/papers/2411.00855) (2024)\n* [MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models](https://huggingface.co/papers/2410.10139) (2024)\n* [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://huggingface.co/papers/2410.19168) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment (2024) \n
- MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models (2024) \n
- LLaVA-Critic: Learning to Evaluate Multimodal Models (2024) \n
- MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark (2024) \n
- Vision-Language Models Can Self-Improve Reasoning via Reflection (2024) \n
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (2024) \n
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2411.17451
VLRewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models
Published on Nov 26, 2024
Abstract
VL-RewardBench is a comprehensive benchmark designed to challenge vision-language generative reward models across various tasks, demonstrating their limitations and providing insights for improvement.
AI-generated summary
Vision-language generative reward models (VL-GenRMs) play a crucial role in aligning and evaluating multimodal AI systems, yet their own evaluation remains under-explored. Current assessment methods primarily rely on AI-annotated preference labels from traditional VL tasks, which can introduce biases and often fail to effectively challenge state-of-the-art models. To address these limitations, we introduce VL-RewardBench, a comprehensive benchmark spanning general multimodal queries, visual hallucination detection, and complex reasoning tasks. Through our AI-assisted annotation pipeline combining sample selection with human verification, we curate 1,250 high-quality examples specifically designed to probe model limitations. Comprehensive evaluation across 16 leading large vision-language models, demonstrates VL-RewardBench's effectiveness as a challenging testbed, where even GPT-4o achieves only 65.4% accuracy, and state-of-the-art open-source models such as Qwen2-VL-72B, struggle to surpass random-guessing. Importantly, performance on VL-RewardBench strongly correlates (Pearson's r > 0.9) with MMMU-Pro accuracy using Best-of-N sampling with VL-GenRMs. Analysis experiments uncover three critical insights for improving VL-GenRMs: (i) models predominantly fail at basic visual perception tasks rather than reasoning tasks; (ii) inference-time scaling benefits vary dramatically by model capacity; and (iii) training VL-GenRMs to learn to judge substantially boosts judgment capability (+14.7% accuracy for a 7B VL-GenRM). We believe VL-RewardBench along with the experimental insights will become a valuable resource for advancing VL-GenRMs.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment (2024)
- MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models (2024)
- LLaVA-Critic: Learning to Evaluate Multimodal Models (2024)
- MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark (2024)
- Vision-Language Models Can Self-Improve Reasoning via Reflection (2024)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (2024)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend