https://github.com/Thisisus7/ING-VP 👍👍👍

\n","updatedAt":"2024-10-10T09:28:54.110Z","author":{"_id":"646b43deb1202bc77c1024a4","avatarUrl":"/avatars/cf791574ab986bac274e7fbcf04e2a59.svg","fullname":"hangyu guo","name":"Rosiness","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":5,"isUserFollowing":false}},"numEdits":0,"editors":["Rosiness"],"editorAvatarUrls":["/avatars/cf791574ab986bac274e7fbcf04e2a59.svg"],"reactions":[],"isReport":false}},{"id":"6708805024f50fea01c55b56","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false},"createdAt":"2024-10-11T01:33:04.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://huggingface.co/papers/2409.02813) (2024)\n* [MathScape: Evaluating MLLMs in multimodal Math Scenarios through a Hierarchical Benchmark](https://huggingface.co/papers/2408.07543) (2024)\n* [Atari-GPT: Investigating the Capabilities of Multimodal Large Language Models as Low-Level Policies for Atari Games](https://huggingface.co/papers/2408.15950) (2024)\n* [OmniBench: Towards The Future of Universal Omni-Language Models](https://huggingface.co/papers/2409.15272) (2024)\n* [MMR: Evaluating Reading Ability of Large Multimodal Models](https://huggingface.co/papers/2408.14594) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-10-11T01:33:04.087Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false}},"numEdits":0,"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2410.06555","authors":[{"_id":"67079d5bdba78ae6a5605e43","user":{"_id":"647c23b6c3c809c69d345c45","avatarUrl":"/avatars/b039807814f5a778ca327ead232fc67b.svg","isPro":false,"fullname":"Haoran Zhang","user":"RickStark","type":"user"},"name":"Haoran Zhang","status":"extracted_pending","statusLastChangedAt":"2024-10-14T10:18:58.685Z","hidden":false},{"_id":"67079d5bdba78ae6a5605e44","user":{"_id":"646b43deb1202bc77c1024a4","avatarUrl":"/avatars/cf791574ab986bac274e7fbcf04e2a59.svg","isPro":false,"fullname":"hangyu guo","user":"Rosiness","type":"user"},"name":"Hangyu Guo","status":"admin_assigned","statusLastChangedAt":"2024-10-10T09:46:00.887Z","hidden":false},{"_id":"67079d5bdba78ae6a5605e45","name":"Shuyue Guo","hidden":false},{"_id":"67079d5bdba78ae6a5605e46","name":"Meng Cao","hidden":false},{"_id":"67079d5bdba78ae6a5605e47","user":{"_id":"65bb11cb00a03997849e9e85","avatarUrl":"/avatars/17022b0254192a837f4fe00d84389cda.svg","isPro":false,"fullname":"Wenhao Huang","user":"StephenHuang","type":"user"},"name":"Wenhao Huang","status":"claimed_verified","statusLastChangedAt":"2024-10-13T20:32:31.028Z","hidden":false},{"_id":"67079d5bdba78ae6a5605e48","user":{"_id":"65377c30e48353201e6fdda0","avatarUrl":"/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg","isPro":false,"fullname":"Jiaheng Liu","user":"CheeryLJH","type":"user"},"name":"Jiaheng Liu","status":"admin_assigned","statusLastChangedAt":"2024-10-10T09:45:29.766Z","hidden":false},{"_id":"67079d5bdba78ae6a5605e49","user":{"_id":"638efcf4c67af472d316d424","avatarUrl":"/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg","isPro":false,"fullname":"Ge Zhang","user":"zhangysk","type":"user"},"name":"Ge Zhang","status":"admin_assigned","statusLastChangedAt":"2024-10-10T09:45:24.241Z","hidden":false}],"publishedAt":"2024-10-09T05:17:38.000Z","submittedOnDailyAt":"2024-10-10T07:58:54.104Z","title":"ING-VP: MLLMs cannot Play Easy Vision-based Games Yet","submittedOnDailyBy":{"_id":"646b43deb1202bc77c1024a4","avatarUrl":"/avatars/cf791574ab986bac274e7fbcf04e2a59.svg","isPro":false,"fullname":"hangyu guo","user":"Rosiness","type":"user"},"summary":"As multimodal large language models (MLLMs) continue to demonstrate\nincreasingly competitive performance across a broad spectrum of tasks, more\nintricate and comprehensive benchmarks have been developed to assess these\ncutting-edge models. These benchmarks introduce new challenges to core\ncapabilities such as perception, reasoning, and planning. However, existing\nmultimodal benchmarks fall short in providing a focused evaluation of\nmulti-step planning based on spatial relationships in images. To bridge this\ngap, we present ING-VP, the first INteractive Game-based Vision Planning\nbenchmark, specifically designed to evaluate the spatial imagination and\nmulti-step reasoning abilities of MLLMs. ING-VP features 6 distinct games,\nencompassing 300 levels, each with 6 unique configurations. A single model\nengages in over 60,000 rounds of interaction. The benchmark framework allows\nfor multiple comparison settings, including image-text vs. text-only inputs,\nsingle-step vs. multi-step reasoning, and with-history vs. without-history\nconditions, offering valuable insights into the model's capabilities. We\nevaluated numerous state-of-the-art MLLMs, with the highest-performing model,\nClaude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the\nanticipated standard. This work aims to provide a specialized evaluation\nframework to drive advancements in MLLMs' capacity for complex spatial\nreasoning and planning. The code is publicly available at\nhttps://github.com/Thisisus7/ING-VP.git.","upvotes":8,"discussionId":"67079d5ddba78ae6a5605e9f","githubRepo":"https://github.com/thisisus7/ing-vp","githubRepoAddedBy":"auto","ai_summary":"ING-VP is a new benchmark designed to evaluate multimodal large language models' spatial imagination and multi-step reasoning through interactive game-based vision planning tasks.","ai_keywords":["multimodal large language models","perception","reasoning","planning","spatial relationships","interactive game-based vision planning","spatial imagination","multi-step reasoning","image-text inputs","single-step reasoning","multi-step reasoning","with-history","without-history","accuracy"],"githubStars":12},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"646b43deb1202bc77c1024a4","avatarUrl":"/avatars/cf791574ab986bac274e7fbcf04e2a59.svg","isPro":false,"fullname":"hangyu guo","user":"Rosiness","type":"user"},{"_id":"647c23b6c3c809c69d345c45","avatarUrl":"/avatars/b039807814f5a778ca327ead232fc67b.svg","isPro":false,"fullname":"Haoran Zhang","user":"RickStark","type":"user"},{"_id":"64e572a9e56d371e42e1ecb4","avatarUrl":"/avatars/56c57dd2badaaf5a3a881f1a4d9e2d08.svg","isPro":false,"fullname":"mengcao","user":"constanceCM","type":"user"},{"_id":"638efcf4c67af472d316d424","avatarUrl":"/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg","isPro":false,"fullname":"Ge Zhang","user":"zhangysk","type":"user"},{"_id":"65377c30e48353201e6fdda0","avatarUrl":"/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg","isPro":false,"fullname":"Jiaheng Liu","user":"CheeryLJH","type":"user"},{"_id":"6707c949ceaa2578b5645450","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zPlCru7Y9bQBCTYcAJ6Ah.png","isPro":false,"fullname":"Jeff badman","user":"Lastat22","type":"user"},{"_id":"668cd4bbe990292e5f6974d3","avatarUrl":"/avatars/d1747b2372e94500ecb5fb56809b482d.svg","isPro":false,"fullname":"Jinyeong Kim","user":"rubatoyeong","type":"user"},{"_id":"65bb11cb00a03997849e9e85","avatarUrl":"/avatars/17022b0254192a837f4fe00d84389cda.svg","isPro":false,"fullname":"Wenhao Huang","user":"StephenHuang","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2410.06555

ING-VP: MLLMs cannot Play Easy Vision-based Games Yet

Published on Oct 9, 2024

· Submitted by

hangyu guo on Oct 10, 2024

Upvote

Authors:

Haoran Zhang ,

Hangyu Guo ,

Wenhao Huang ,

Jiaheng Liu ,

Ge Zhang

Abstract

ING-VP is a new benchmark designed to evaluate multimodal large language models' spatial imagination and multi-step reasoning through interactive game-based vision planning tasks.

AI-generated summary

As multimodal large language models (MLLMs) continue to demonstrate increasingly competitive performance across a broad spectrum of tasks, more intricate and comprehensive benchmarks have been developed to assess these cutting-edge models. These benchmarks introduce new challenges to core capabilities such as perception, reasoning, and planning. However, existing multimodal benchmarks fall short in providing a focused evaluation of multi-step planning based on spatial relationships in images. To bridge this gap, we present ING-VP, the first INteractive Game-based Vision Planning benchmark, specifically designed to evaluate the spatial imagination and multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games, encompassing 300 levels, each with 6 unique configurations. A single model engages in over 60,000 rounds of interaction. The benchmark framework allows for multiple comparison settings, including image-text vs. text-only inputs, single-step vs. multi-step reasoning, and with-history vs. without-history conditions, offering valuable insights into the model's capabilities. We evaluated numerous state-of-the-art MLLMs, with the highest-performing model, Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the anticipated standard. This work aims to provide a specialized evaluation framework to drive advancements in MLLMs' capacity for complex spatial reasoning and planning. The code is publicly available at https://github.com/Thisisus7/ING-VP.git.