The following papers were recommended by the Semantic Scholar API
\n- \n
- RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning (2025) \n
- Incentivizing Strong Reasoning from Weak Supervision (2025) \n
- Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning (2025) \n
- Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers (2025) \n
- Behavior Injection: Preparing Language Models for Reinforcement Learning (2025) \n
- CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models (2025) \n
- S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
Abstract
LLMs exhibit robustness to reward noise during post-training and achieve high performance using reasoning pattern rewards (RPR) in conjunction with noisy reward models.
Recent studies on post-training large language models (LLMs) for reasoning through reinforcement learning (RL) typically focus on tasks that can be accurately verified and rewarded, such as solving math problems. In contrast, our research investigates the impact of reward noise, a more practical consideration for real-world scenarios involving the post-training of LLMs using reward models. We found that LLMs demonstrate strong robustness to substantial reward noise. For example, manually flipping 40% of the reward function's outputs in math tasks still allows a Qwen-2.5-7B model to achieve rapid convergence, improving its performance on math tasks from 5% to 72%, compared to the 75% accuracy achieved by a model trained with noiseless rewards. Surprisingly, by only rewarding the appearance of key reasoning phrases (namely reasoning pattern reward, RPR), such as ``first, I need to''-without verifying the correctness of answers, the model achieved peak downstream performance (over 70% accuracy for Qwen-2.5-7B) comparable to models trained with strict correctness verification and accurate rewards. Recognizing the importance of the reasoning process over the final results, we combined RPR with noisy reward models. RPR helped calibrate the noisy reward models, mitigating potential false negatives and enhancing the LLM's performance on open-ended tasks. These findings suggest the importance of improving models' foundational abilities during the pre-training phase while providing insights for advancing post-training techniques. Our code and scripts are available at https://github.com/trestad/Noisy-Rewards-in-Learning-to-Reason.
Community
We found that LLMs demonstrate strong robustness to substantial reward noise. For example, manually flipping 40% of the reward function's outputs in math tasks still allows a Qwen-2.5-7B model to achieve rapid convergence, improving its performance on math tasks from 5% to 72%, compared to the 75% accuracy achieved by a model trained with noiseless rewards. We hypothesize that outputs leading to incorrect answers may still contain valuable information—specifically, useful reasoning patterns.
To test this hypothesis, we only reward the appearance of key reasoning phrases (namely reasoning pattern reward, RPR), such as ``first, I need to''—without verifying the correctness of answers. The model achieved peak downstream performance (over 70% accuracy for Qwen-2.5-7B) comparable to models trained with strict correctness verification and accurate rewards. This indicates that the gains from reinforcement learning might primarily come from teaching the model to adopt appropriate reasoning styles. The fundamental problem-solving abilities required are largely acquired during pretraining.
Recognizing the importance of the reasoning process, we combined RPR with noisy reward models. RPR helped calibrate the noisy reward models by mitigating potential false negative rewards. With calibrated RMs, LLMs' performance in open-ended NLP tasks are enhanced and smaller models are able to successfully acquire reasoning capabilities through RL.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning (2025)
- Incentivizing Strong Reasoning from Weak Supervision (2025)
- Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning (2025)
- Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answers (2025)
- Behavior Injection: Preparing Language Models for Reinforcement Learning (2025)
- CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models (2025)
- S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper