Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

\n","updatedAt":"2023-10-14T10:28:07.562Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7203500270843506},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"666592579f5a0b3229c82315","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":176,"isUserFollowing":false},"createdAt":"2024-06-09T11:30:31.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"# Octopus: Revolutionizing Vision-Language Programming with Environmental Feedback\n\nhttps://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/ibJjMMZykWh7ZMe83ShTX.mp4 \n\n## Links 🔗:\n👉 Subscribe: https://www.youtube.com/@Arxflix\n👉 Twitter: https://x.com/arxflix\n👉 LMNT (Partner): https://lmnt.com/\n\n\nBy Arxflix\n![9t4iCUHx_400x400-1.jpg](https://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/v4S5zBurs0ouGNwYj1GEd.jpeg)","html":"

\n\t\n\t\t\n\t\n\t\n\t\tOctopus: Revolutionizing Vision-Language Programming with Environmental Feedback\n\t\n

\n\n

\n\t\n\t\t\n\t\n\t\n\t\tLinks 🔗:\n\t\n

👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/

By Arxflix
$\"9t4iCUHx_400x400-1.jpg\"$

\n","updatedAt":"2024-06-09T11:30:31.720Z","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":176,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5213712453842163},"editors":["blanchon"],"editorAvatarUrls":["/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg"],"reactions":[{"reaction":"🔥","users":["Choiszt"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2310.08588","authors":[{"_id":"6529471c019a20ed658f07d9","user":{"_id":"62b5777f593a2c49da69dc02","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg","isPro":false,"fullname":"Jingkang Yang","user":"Jingkang","type":"user"},"name":"Jingkang Yang","status":"claimed_verified","statusLastChangedAt":"2023-10-13T14:42:54.592Z","hidden":false},{"_id":"6529471c019a20ed658f07da","user":{"_id":"652965773a416e1f2173443b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg","isPro":true,"fullname":"Yuhao Dong","user":"THUdyh","type":"user"},"name":"Yuhao Dong","status":"admin_assigned","statusLastChangedAt":"2023-10-13T15:50:15.190Z","hidden":false},{"_id":"6529471c019a20ed658f07db","name":"Shuai Liu","hidden":false},{"_id":"6529471c019a20ed658f07dc","user":{"_id":"62d3f7d84b0933c48f3cdd9c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/Tab1vxtxLatWzXS8NVIyo.png","isPro":true,"fullname":"Bo Li","user":"luodian","type":"user"},"name":"Bo Li","status":"claimed_verified","statusLastChangedAt":"2024-08-09T07:49:39.633Z","hidden":false},{"_id":"6529471c019a20ed658f07dd","user":{"_id":"652961d24267f8c80220bbfb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0yCP-XYJu6PwTGkoyhKQB.jpeg","isPro":false,"fullname":"Ziyue Wang","user":"annewzy","type":"user"},"name":"Ziyue Wang","status":"claimed_verified","statusLastChangedAt":"2023-10-16T07:24:45.666Z","hidden":false},{"_id":"6529471c019a20ed658f07de","user":{"_id":"6470f5aad9360cd9d8e9310d","avatarUrl":"/avatars/86efd78d7ff12bded166f0a5e3fe2763.svg","isPro":false,"fullname":"Jiang Chencheng","user":"jchencheng","type":"user"},"name":"Chencheng Jiang","status":"admin_assigned","statusLastChangedAt":"2023-10-13T15:42:07.445Z","hidden":false},{"_id":"6529471c019a20ed658f07df","user":{"_id":"652802a42949e6071c703550","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652802a42949e6071c703550/dQF73cLlAoSLtgi8JzbnI.jpeg","isPro":false,"fullname":"Haoran Tan","user":"Tan0118","type":"user"},"name":"Haoran Tan","status":"claimed_verified","statusLastChangedAt":"2023-10-21T09:22:51.468Z","hidden":false},{"_id":"6529471c019a20ed658f07e0","name":"Jiamu Kang","hidden":false},{"_id":"6529471c019a20ed658f07e1","user":{"_id":"62a993d80472c0b7f94027df","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png","isPro":false,"fullname":"Zhang Yuanhan","user":"ZhangYuanhan","type":"user"},"name":"Yuanhan Zhang","status":"admin_assigned","statusLastChangedAt":"2023-10-13T15:39:27.007Z","hidden":false},{"_id":"6529471c019a20ed658f07e2","user":{"_id":"62ac6656de8bfbb93094b8fd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/62ac6656de8bfbb93094b8fd/ruJGOUcsG9UgDhTQx9Sz6.png","isPro":false,"fullname":"Kaiyang Zhou","user":"kaiyangzhou","type":"user"},"name":"Kaiyang Zhou","status":"admin_assigned","statusLastChangedAt":"2023-10-13T14:44:00.926Z","hidden":false},{"_id":"6529471c019a20ed658f07e3","user":{"_id":"62ab1ac1d48b4d8b048a3473","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png","isPro":false,"fullname":"Ziwei Liu","user":"liuziwei7","type":"user"},"name":"Ziwei Liu","status":"admin_assigned","statusLastChangedAt":"2023-10-13T14:43:53.257Z","hidden":false}],"publishedAt":"2023-10-12T17:59:58.000Z","submittedOnDailyAt":"2023-10-13T12:03:18.820Z","title":"Octopus: Embodied Vision-Language Programmer from Environmental Feedback","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"Large vision-language models (VLMs) have achieved substantial progress in\nmultimodal perception and reasoning. Furthermore, when seamlessly integrated\ninto an embodied agent, it signifies a crucial stride towards the creation of\nautonomous and context-aware systems capable of formulating plans and executing\ncommands with precision. In this paper, we introduce Octopus, a novel VLM\ndesigned to proficiently decipher an agent's vision and textual task objectives\nand to formulate intricate action sequences and generate executable code. Our\ndesign allows the agent to adeptly handle a wide spectrum of tasks, ranging\nfrom mundane daily chores in simulators to sophisticated interactions in\ncomplex video games. Octopus is trained by leveraging GPT-4 to control an\nexplorative agent to generate training data, i.e., action blueprints and the\ncorresponding executable code, within our experimental environment called\nOctoVerse. We also collect the feedback that allows the enhanced training\nscheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a\nseries of experiments, we illuminate Octopus's functionality and present\ncompelling results, and the proposed RLEF turns out to refine the agent's\ndecision-making. By open-sourcing our model architecture, simulator, and\ndataset, we aspire to ignite further innovation and foster collaborative\napplications within the broader embodied AI community.","upvotes":38,"discussionId":"6529471e019a20ed658f083e","githubRepo":"https://github.com/dongyh20/octopus","githubRepoAddedBy":"auto","ai_summary":"Octopus, a novel vision-language model, integrates multimodal perception and reasoning to perform diverse tasks using GPT-4 and Reinforcement Learning with Environmental Feedback in an experimental environment called OctoVerse.","ai_keywords":["VLMs","multimodal perception","reasoning","embodied agent","action sequences","executable code","GPT-4","Reinforcement Learning with Environmental Feedback","RLEF","OctoVerse"],"githubStars":294},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"62b5777f593a2c49da69dc02","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg","isPro":false,"fullname":"Jingkang Yang","user":"Jingkang","type":"user"},{"_id":"62825d6c590d129165cd63c1","avatarUrl":"/avatars/fec05c6f7799b52249309a41e9053434.svg","isPro":false,"fullname":"Ralph","user":"RalphX1","type":"user"},{"_id":"62ab1ac1d48b4d8b048a3473","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png","isPro":false,"fullname":"Ziwei Liu","user":"liuziwei7","type":"user"},{"_id":"652961d24267f8c80220bbfb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0yCP-XYJu6PwTGkoyhKQB.jpeg","isPro":false,"fullname":"Ziyue Wang","user":"annewzy","type":"user"},{"_id":"65296343f0e232695d8fa7d1","avatarUrl":"/avatars/79b8fa650de1e5ceea8fe82a62c8a641.svg","isPro":false,"fullname":"谭浩然","user":"Tan011899","type":"user"},{"_id":"6349988f21f7b96c188d93ba","avatarUrl":"/avatars/8e9f648ef0d981f3b397d22f178c722c.svg","isPro":false,"fullname":"Dorbmon","user":"Dorbmon","type":"user"},{"_id":"652965ca680fa36b3bc9cb53","avatarUrl":"/avatars/e658476eebe702d66bbebecb73c087ad.svg","isPro":false,"fullname":"Billie","user":"BillieWong","type":"user"},{"_id":"652965773a416e1f2173443b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg","isPro":true,"fullname":"Yuhao Dong","user":"THUdyh","type":"user"},{"_id":"5dd96eb166059660ed1ee413","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/NQtzmrDdbG0H8qkZvRyGk.jpeg","isPro":true,"fullname":"Julien Chaumond","user":"julien-c","type":"user"},{"_id":"652967a1caddc9e608c89094","avatarUrl":"/avatars/07f3d9570871e9ec1d189edc8d16ab2a.svg","isPro":false,"fullname":"kangjiamu","user":"kangjiamu","type":"user"},{"_id":"6529692c68f1d7d1d2363c29","avatarUrl":"/avatars/83582d358e67150ef990a3cf570e7702.svg","isPro":false,"fullname":"Stewart","user":"CyanSheep","type":"user"},{"_id":"65296da23a416e1f217481b1","avatarUrl":"/avatars/15a84ce0989a870c7bdc303a59a98845.svg","isPro":false,"fullname":"Chesco","user":"Chesconut","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":1}">

Papers

arxiv:2310.08588

Octopus: Embodied Vision-Language Programmer from Environmental Feedback

Published on Oct 12, 2023

· Submitted by

AK on Oct 13, 2023

#1 Paper of the day

Upvote

Authors:

Jingkang Yang ,

Yuhao Dong ,

Bo Li ,

Ziyue Wang ,

Chencheng Jiang ,

Haoran Tan ,

Yuanhan Zhang ,

Kaiyang Zhou ,

Ziwei Liu

Abstract

Octopus, a novel vision-language model, integrates multimodal perception and reasoning to perform diverse tasks using GPT-4 and Reinforcement Learning with Environmental Feedback in an experimental environment called OctoVerse.

AI-generated summary

Large vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning. Furthermore, when seamlessly integrated into an embodied agent, it signifies a crucial stride towards the creation of autonomous and context-aware systems capable of formulating plans and executing commands with precision. In this paper, we introduce Octopus, a novel VLM designed to proficiently decipher an agent's vision and textual task objectives and to formulate intricate action sequences and generate executable code. Our design allows the agent to adeptly handle a wide spectrum of tasks, ranging from mundane daily chores in simulators to sophisticated interactions in complex video games. Octopus is trained by leveraging GPT-4 to control an explorative agent to generate training data, i.e., action blueprints and the corresponding executable code, within our experimental environment called OctoVerse. We also collect the feedback that allows the enhanced training scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a series of experiments, we illuminate Octopus's functionality and present compelling results, and the proposed RLEF turns out to refine the agent's decision-making. By open-sourcing our model architecture, simulator, and dataset, we aspire to ignite further innovation and foster collaborative applications within the broader embodied AI community.