Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-01-10T01:33:57.028Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7230423092842102},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2501.04575","authors":[{"_id":"677f5089a0843d06966ac68e","user":{"_id":"66dd4feb14f4776a44b071f2","avatarUrl":"/avatars/8b06fa3019999abb762de2b8d9961e2b.svg","isPro":false,"fullname":"Yuhang Liu","user":"CausalLLMs","type":"user"},"name":"Yuhang Liu","status":"admin_assigned","statusLastChangedAt":"2025-01-09T20:24:38.607Z","hidden":false},{"_id":"677f5089a0843d06966ac68f","user":{"_id":"64245f2c089d5fae56b4549a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg","isPro":false,"fullname":"Pengxiang Li","user":"pengxiang","type":"user"},"name":"Pengxiang Li","status":"claimed_verified","statusLastChangedAt":"2025-01-09T10:06:37.729Z","hidden":false},{"_id":"677f5089a0843d06966ac690","user":{"_id":"63909970937867f0cb400b09","avatarUrl":"/avatars/a360d5d5a3605e7312b078312eee79b6.svg","isPro":false,"fullname":"zishu wei","user":"shu06","type":"user"},"name":"Zishu Wei","status":"admin_assigned","statusLastChangedAt":"2025-01-09T20:24:47.683Z","hidden":false},{"_id":"677f5089a0843d06966ac691","user":{"_id":"6719f1ad725123d503b5ef3c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6719f1ad725123d503b5ef3c/P7yYxNVROsKPOVsqMu0jp.jpeg","isPro":false,"fullname":"Congkai Xie","user":"congkai","type":"user"},"name":"Congkai Xie","status":"admin_assigned","statusLastChangedAt":"2025-01-09T20:24:53.767Z","hidden":false},{"_id":"677f5089a0843d06966ac692","user":{"_id":"65897684f8b453e1f57cdb26","avatarUrl":"/avatars/80096d6c808805e1a84a68fb6194a7d4.svg","isPro":false,"fullname":"huxueyu","user":"huxueyu","type":"user"},"name":"Xueyu Hu","status":"admin_assigned","statusLastChangedAt":"2025-01-09T20:25:09.450Z","hidden":false},{"_id":"677f5089a0843d06966ac693","user":{"_id":"65601713baf1f5f902292bb6","avatarUrl":"/avatars/57c65cff29ffb4f25b943b5baccdc795.svg","isPro":false,"fullname":"Xinchen Xu","user":"SHU-Xuxc","type":"user"},"name":"Xinchen Xu","status":"admin_assigned","statusLastChangedAt":"2025-01-09T20:25:15.603Z","hidden":false},{"_id":"677f5089a0843d06966ac694","name":"Shengyu Zhang","hidden":false},{"_id":"677f5089a0843d06966ac695","user":{"_id":"650dde4ce14eeb01d42b37a1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png","isPro":false,"fullname":"Xiaotian Han","user":"xiaotianhan","type":"user"},"name":"Xiaotian Han","status":"claimed_verified","statusLastChangedAt":"2025-01-09T19:57:55.175Z","hidden":false},{"_id":"677f5089a0843d06966ac696","name":"Hongxia Yang","hidden":false},{"_id":"677f5089a0843d06966ac697","name":"Fei Wu","hidden":false}],"publishedAt":"2025-01-08T15:45:21.000Z","submittedOnDailyAt":"2025-01-09T02:05:19.486Z","title":"InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning\n and Reflection","submittedOnDailyBy":{"_id":"64245f2c089d5fae56b4549a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg","isPro":false,"fullname":"Pengxiang Li","user":"pengxiang","type":"user"},"summary":"Graphical User Interface (GUI) Agents, powered by multimodal large language\nmodels (MLLMs), have shown great potential for task automation on computing\ndevices such as computers and mobile phones. However, existing agents face\nchallenges in multi-step reasoning and reliance on textual annotations,\nlimiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based\nGUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1\nenhances fundamental skills such as GUI understanding and grounding, while\nStage 2 integrates hierarchical reasoning and expectation-reflection reasoning\nskills using synthesized data to enable native reasoning abilities of the\nagents. InfiGUIAgent achieves competitive performance on several GUI\nbenchmarks, highlighting the impact of native reasoning skills in enhancing GUI\ninteraction for automation tasks. Resources are available at\nhttps://github.com/Reallm-Labs/InfiGUIAgent.","upvotes":25,"discussionId":"677f5089a0843d06966ac6e3","githubRepo":"https://github.com/reallm-labs/infiguiagent","githubRepoAddedBy":"auto","ai_summary":"InfiGUIAgent, an MLLM-based GUI Agent, enhances multi-step reasoning and GUI interaction through a two-stage supervised fine-tuning pipeline.","ai_keywords":["multimodal large language models","GUI Agents","supervised fine-tuning","GUI understanding","hierarchical reasoning","expectation-reflection reasoning","synthesized data","GUI benchmarks"],"githubStars":73},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64245f2c089d5fae56b4549a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg","isPro":false,"fullname":"Pengxiang Li","user":"pengxiang","type":"user"},{"_id":"62722849517c0ca41f7cd13d","avatarUrl":"/avatars/bb1f8f2f2665944930cb5a7ce19c47d4.svg","isPro":false,"fullname":"Yuhang Liu","user":"SiriusL","type":"user"},{"_id":"65897684f8b453e1f57cdb26","avatarUrl":"/avatars/80096d6c808805e1a84a68fb6194a7d4.svg","isPro":false,"fullname":"huxueyu","user":"huxueyu","type":"user"},{"_id":"677f568fedc5741bc210e71e","avatarUrl":"/avatars/f02675184d5fe8d79e4d384d4a82887d.svg","isPro":false,"fullname":"Eric Zhang","user":"EricZhangZJU","type":"user"},{"_id":"66a697248b85fa8a34005aed","avatarUrl":"/avatars/3d7f26676ad7e91e540929f1f04b33fd.svg","isPro":false,"fullname":"Tao Xiong","user":"YuanDaozeiii","type":"user"},{"_id":"671a6de96a427e75e07b0425","avatarUrl":"/avatars/cd16c46389639b9f6b74afa52709de6e.svg","isPro":false,"fullname":"易标","user":"EaseJimmy","type":"user"},{"_id":"6505dc6e6ba49887d3f964b7","avatarUrl":"/avatars/cee672d675848846d32568b24230ffc8.svg","isPro":false,"fullname":"huazi","user":"hua-zi","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6033c55f60e3dd96631c908d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6033c55f60e3dd96631c908d/jy7cHHCBhnlzHKGbXIbj0.jpeg","isPro":false,"fullname":"Shyam Sunder Kumar","user":"theainerd","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"677feddf088ec73dbdbc45b4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LiOBPl8KrbilWR-Azl8wJ.png","isPro":false,"fullname":"Eiman Badwy","user":"Eimanbadwy2110","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2501.04575

InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection

Published on Jan 8, 2025

· Submitted by

Pengxiang Li on Jan 9, 2025

Upvote

Authors:

Yuhang Liu ,

Pengxiang Li ,

Zishu Wei ,

Congkai Xie ,

Xueyu Hu ,

Xinchen Xu ,

Xiaotian Han ,

Abstract

InfiGUIAgent, an MLLM-based GUI Agent, enhances multi-step reasoning and GUI interaction through a two-stage supervised fine-tuning pipeline.

AI-generated summary

Graphical User Interface (GUI) Agents, powered by multimodal large language models (MLLMs), have shown great potential for task automation on computing devices such as computers and mobile phones. However, existing agents face challenges in multi-step reasoning and reliance on textual annotations, limiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1 enhances fundamental skills such as GUI understanding and grounding, while Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning skills using synthesized data to enable native reasoning abilities of the agents. InfiGUIAgent achieves competitive performance on several GUI benchmarks, highlighting the impact of native reasoning skills in enhancing GUI interaction for automation tasks. Resources are available at https://github.com/Reallm-Labs/InfiGUIAgent.

View arXiv page View PDF GitHub 73 auto Add to collection

Community

pengxiang

Paper author Paper submitter Jan 9, 2025

InfiGUIAgent is a multimodal LLM-based GUI agent that improves performance through a two-stage supervised fine-tuning approach:

Stage 1 enhances basic skills (GUI understanding and grounding)
Stage 2 integrates hierarchical and expectation-reflection reasoning.

This training method enables "native reasoning abilities," showing competitive performance on GUI benchmarks and improving automated GUI interactions.