The following papers were recommended by the Semantic Scholar API
\n- \n
- GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning (2025) \n
- Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering (2025) \n
- Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1 (2025) \n
- Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning (2025) \n
- Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling (2025) \n
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025) \n
- Improving Retrospective Language Agents via Joint Policy Gradient Optimization (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Is the code available?
\n","updatedAt":"2025-05-06T07:43:49.860Z","author":{"_id":"62f6930ad278a8f3e7850a1e","avatarUrl":"/avatars/fbd8b15413306036e021cc1348b88622.svg","fullname":"Michael Guan","name":"LoveLord","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6960443258285522},"editors":["LoveLord"],"editorAvatarUrls":["/avatars/fbd8b15413306036e021cc1348b88622.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2504.16078","authors":[{"_id":"68087a231e425a6eee93570d","user":{"_id":"64c3849269b1a6796052eac7","avatarUrl":"/avatars/9f0c832d5b51b659c7bb83074f02a648.svg","isPro":false,"fullname":"Thomas Schmied","user":"thomasschmied","type":"user"},"name":"Thomas Schmied","status":"admin_assigned","statusLastChangedAt":"2025-04-23T13:54:06.731Z","hidden":false},{"_id":"68087a231e425a6eee93570e","user":{"_id":"6679bdb03043e7ed261a7699","avatarUrl":"/avatars/482ce2c4b2daf194161a0210e66d96de.svg","isPro":false,"fullname":"Jörg Bornschein","user":"jbornschein","type":"user"},"name":"Jörg Bornschein","status":"admin_assigned","statusLastChangedAt":"2025-04-23T13:54:12.909Z","hidden":false},{"_id":"68087a231e425a6eee93570f","name":"Jordi Grau-Moya","hidden":false},{"_id":"68087a231e425a6eee935710","user":{"_id":"680e82a9252bc7d0a3b1228d","avatarUrl":"/avatars/3028d97c7702f259469a80cbb4ed13de.svg","isPro":false,"fullname":"Markus Wulfmeier","user":"mwulfmeier","type":"user"},"name":"Markus Wulfmeier","status":"claimed_verified","statusLastChangedAt":"2025-04-28T12:50:27.722Z","hidden":false},{"_id":"68087a231e425a6eee935711","user":{"_id":"64b9310403124195cd9778ec","avatarUrl":"/avatars/57c594d3d0f97d3010b15b6a0806451c.svg","isPro":false,"fullname":"Razvan Pascanu","user":"razp","type":"user"},"name":"Razvan Pascanu","status":"admin_assigned","statusLastChangedAt":"2025-04-23T13:54:25.503Z","hidden":false}],"publishedAt":"2025-04-22T17:57:14.000Z","submittedOnDailyAt":"2025-04-23T03:57:30.931Z","title":"LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making\n Abilities","submittedOnDailyBy":{"_id":"64c3849269b1a6796052eac7","avatarUrl":"/avatars/9f0c832d5b51b659c7bb83074f02a648.svg","isPro":false,"fullname":"Thomas Schmied","user":"thomasschmied","type":"user"},"summary":"The success of Large Language Models (LLMs) has sparked interest in various\nagentic applications. A key hypothesis is that LLMs, leveraging common sense\nand Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently\nsolve complex domains. However, LLM agents have been found to suffer from\nsub-optimal exploration and the knowing-doing gap, the inability to effectively\nact on knowledge present in the model. In this work, we systematically study\nwhy LLMs perform sub-optimally in decision-making scenarios. In particular, we\nclosely examine three prevalent failure modes: greediness, frequency bias, and\nthe knowing-doing gap. We propose mitigation of these shortcomings by\nfine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales.\nOur experiments across multi-armed bandits, contextual bandits, and\nTic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making\nabilities of LLMs by increasing exploration and narrowing the knowing-doing\ngap. Finally, we study both classic exploration mechanisms, such as\nepsilon-greedy, and LLM-specific approaches, such as self-correction and\nself-consistency, to enable more effective fine-tuning of LLMs for\ndecision-making.","upvotes":21,"discussionId":"68087a241e425a6eee93576b","ai_summary":"Reinforcement learning fine-tuning with Chain-of-Thought rationales enhances Large Language Models' decision-making abilities by addressing greediness, frequency bias, and the knowing-doing gap.","ai_keywords":["Large Language Models","Chain-of-Thought (CoT)","decision-making","sub-optimal exploration","knowing-doing gap","Reinforcement Learning (RL)","multi-armed bandits","contextual bandits","Tic-tac-toe","$\\epsilon$-greedy","self-correction","self-consistency"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64c3849269b1a6796052eac7","avatarUrl":"/avatars/9f0c832d5b51b659c7bb83074f02a648.svg","isPro":false,"fullname":"Thomas Schmied","user":"thomasschmied","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"68088826d504fe57dd706276","avatarUrl":"/avatars/cec2dac585008119a21e3e577ba8326b.svg","isPro":false,"fullname":"Hannah Spahits","user":"Hnnnnnh","type":"user"},{"_id":"66446d2fe7ca43b97c6f41fe","avatarUrl":"/avatars/b9936fd8bce78160bfa362e26594001c.svg","isPro":false,"fullname":"Boyuan Chen","user":"BoyuanChen","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"6581f9514adaee05cf640f81","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6581f9514adaee05cf640f81/sXvEEraq2QlSIyWHlSmpa.jpeg","isPro":false,"fullname":"Xi","user":"xi0v","type":"user"},{"_id":"665b133508d536a8ac804f7d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Uwi0OnANdTbRbHHQvGqvR.png","isPro":false,"fullname":"Paulson","user":"Pnaomi","type":"user"},{"_id":"6270324ebecab9e2dcf245de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6270324ebecab9e2dcf245de/cMbtWSasyNlYc9hvsEEzt.jpeg","isPro":false,"fullname":"Kye Gomez","user":"kye","type":"user"},{"_id":"6567aec66c1ae87d4ec42272","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6567aec66c1ae87d4ec42272/2OfjRgfNeURrYH6noQkeZ.jpeg","isPro":true,"fullname":"Mert Erbak","user":"merterbak","type":"user"},{"_id":"62f32eab52ad88c930bb3f3b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677134945205-62f32eab52ad88c930bb3f3b.png","isPro":false,"fullname":"Asankhaya Sharma","user":"codelion","type":"user"},{"_id":"64d4615cf8082bf19b916492","avatarUrl":"/avatars/8e1b59565ec5e4b31090cf1b911781b9.svg","isPro":false,"fullname":"wongyukim","user":"wongyukim","type":"user"},{"_id":"633a00248f27255b6b54ea5f","avatarUrl":"/avatars/8ad54c2d8a42093923cbdd6f15e0d7a7.svg","isPro":false,"fullname":"dfuhoiysOHSVFh82934gfjklb","user":"huba-buba","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
Abstract
Reinforcement learning fine-tuning with Chain-of-Thought rationales enhances Large Language Models' decision-making abilities by addressing greediness, frequency bias, and the knowing-doing gap.
The success of Large Language Models (LLMs) has sparked interest in various agentic applications. A key hypothesis is that LLMs, leveraging common sense and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently solve complex domains. However, LLM agents have been found to suffer from sub-optimal exploration and the knowing-doing gap, the inability to effectively act on knowledge present in the model. In this work, we systematically study why LLMs perform sub-optimally in decision-making scenarios. In particular, we closely examine three prevalent failure modes: greediness, frequency bias, and the knowing-doing gap. We propose mitigation of these shortcomings by fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales. Our experiments across multi-armed bandits, contextual bandits, and Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making abilities of LLMs by increasing exploration and narrowing the knowing-doing gap. Finally, we study both classic exploration mechanisms, such as epsilon-greedy, and LLM-specific approaches, such as self-correction and self-consistency, to enable more effective fine-tuning of LLMs for decision-making.
Community
This work studies why LLMs perform sub-optimally in decision-making scenarios and proposes mitigation of their shortcomings by fine-tuning via Reinforcement Learning on self-generated CoT rationales.
Hi,
Can you clarify how can I apply this idea in this problem statement-
Fine tuning gemma 3 1b model on medical data for synthetic discharge note generation. I have real discharge notes.
What can be my RL environment ? Can i ask a bigger model like gemini flash 2.0 to give reward signal ? How can I create the RL environment?
If human in the loop feedback comes for reward/penalty then it will take a lot of time for going through all the data.
Please guide on how can I achieve my goal in the best way possible in limited hardware like google colab pro A100 40gb gpu or use another llm or gemma 3 flavour ?
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning (2025)
- Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering (2025)
- Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1 (2025)
- Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuning (2025)
- Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Sampling (2025)
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025)
- Improving Retrospective Language Agents via Joint Policy Gradient Optimization (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Is the code available?
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper