https://arxiv.org/pdf/2509.03403
Code: Coming soon at https://github.com/Chenluye99/PROF

\n","updatedAt":"2025-09-05T00:44:30.137Z","author":{"_id":"65eec5c1d7d63c2ed0615421","avatarUrl":"/avatars/8c32f5e7d4b1940088bdec73c0b86fab.svg","fullname":"Chenlu Ye","name":"Chenlu123","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5820605158805847},"editors":["Chenlu123"],"editorAvatarUrls":["/avatars/8c32f5e7d4b1940088bdec73c0b86fab.svg"],"reactions":[],"isReport":false}},{"id":"68bc1b37d60b08b180d927b0","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-09-06T11:29:59.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner](https://huggingface.co/papers/2507.23317) (2025)\n* [StepWiser: Stepwise Generative Judges for Wiser Reasoning](https://huggingface.co/papers/2508.19229) (2025)\n* [CAPO: Towards Enhancing LLM Reasoning through Verifiable Generative Credit Assignment](https://huggingface.co/papers/2508.02298) (2025)\n* [Posterior-GRPO: Rewarding Reasoning Processes in Code Generation](https://huggingface.co/papers/2508.05170) (2025)\n* [Libra: Assessing and Improving Reward Model by Learning to Think](https://huggingface.co/papers/2507.21645) (2025)\n* [Promoting Efficient Reasoning with Verifiable Stepwise Reward](https://huggingface.co/papers/2508.10293) (2025)\n* [Cooper: Co-Optimizing Policy and Reward Models in Reinforcement Learning for Large Language Models](https://huggingface.co/papers/2508.05613) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-09-06T11:29:59.017Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7324404120445251},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2509.03403","authors":[{"_id":"68ba320f736018af705e8e1b","name":"Chenlu Ye","hidden":false},{"_id":"68ba320f736018af705e8e1c","name":"Zhou Yu","hidden":false},{"_id":"68ba320f736018af705e8e1d","name":"Ziji Zhang","hidden":false},{"_id":"68ba320f736018af705e8e1e","name":"Hao Chen","hidden":false},{"_id":"68ba320f736018af705e8e1f","name":"Narayanan Sadagopan","hidden":false},{"_id":"68ba320f736018af705e8e20","name":"Jing Huang","hidden":false},{"_id":"68ba320f736018af705e8e21","name":"Tong Zhang","hidden":false},{"_id":"68ba320f736018af705e8e22","name":"Anurag Beniwal","hidden":false}],"publishedAt":"2025-09-03T15:28:51.000Z","submittedOnDailyAt":"2025-09-04T23:14:30.126Z","title":"Beyond Correctness: Harmonizing Process and Outcome Rewards through RL\n Training","submittedOnDailyBy":{"_id":"65eec5c1d7d63c2ed0615421","avatarUrl":"/avatars/8c32f5e7d4b1940088bdec73c0b86fab.svg","isPro":false,"fullname":"Chenlu Ye","user":"Chenlu123","type":"user"},"summary":"Reinforcement learning with verifiable rewards (RLVR) has emerged to be a\npredominant paradigm for mathematical reasoning tasks, offering stable\nimprovements in reasoning ability. However, Outcome Reward Models (ORMs) in\nRLVR are too coarse-grained to distinguish flawed reasoning within correct\nanswers or valid reasoning within incorrect answers. This lack of granularity\nintroduces noisy and misleading gradients significantly and hinders further\nprogress in reasoning process quality. While Process Reward Models (PRMs) offer\nfine-grained guidance for intermediate steps, they frequently suffer from\ninaccuracies and are susceptible to reward hacking.\n To resolve this dilemma, we introduce PRocess cOnsistency Filter (PROF), an\neffective data process curation method that harmonizes noisy, fine-grained\nprocess rewards with accurate, coarse-grained outcome rewards. Rather than\nnaively blending PRM and ORM in the objective function\n(arXiv:archive/2506.18896), PROF leverages their complementary strengths\nthrough consistency-driven sample selection. Our approach retains correct\nresponses with higher averaged process values and incorrect responses with\nlower averaged process values, while maintaining positive/negative training\nsample balance. Extensive experiments demonstrate that our method not only\nconsistently improves the final accuracy over 4% compared to the blending\napproaches, but also strengthens the quality of intermediate reasoning steps.\nCodes and training recipes are available at https://github.com/Chenluye99/PROF.","upvotes":23,"discussionId":"68ba3210736018af705e8e23","githubRepo":"https://github.com/Chenluye99/PROF","githubRepoAddedBy":"auto","ai_summary":"PROF, a data curation method, improves reinforcement learning for mathematical reasoning by harmonizing process and outcome rewards, enhancing both final accuracy and intermediate reasoning quality.","ai_keywords":["Reinforcement learning","verifiable rewards","RLVR","Outcome Reward Models","ORMs","Process Reward Models","PRMs","PRocess cOnsistency Filter","PROF","consistency-driven sample selection","intermediate reasoning steps"],"githubStars":11},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6335150931a2be3938c99db6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6335150931a2be3938c99db6/g8pUPvi9ZI2ztW9fwee5_.png","isPro":false,"fullname":"Dokyoon","user":"leeloolee","type":"user"},{"_id":"6528bca21ce90a096145ecf8","avatarUrl":"/avatars/19a7c8be35eedc5d0127107075501c57.svg","isPro":false,"fullname":"AoLI","user":"qieyou","type":"user"},{"_id":"6618d1c3c1167c8d8702b19d","avatarUrl":"/avatars/24611ca6c4158f4978ac8476a87d8d9c.svg","isPro":false,"fullname":"Ruida WANG","user":"RickyDeSkywalker","type":"user"},{"_id":"66f8689725464a7989b75845","avatarUrl":"/avatars/43a61a528c5779103eaf5687ba44ee14.svg","isPro":false,"fullname":"Jiarui Yao","user":"FlippyDora","type":"user"},{"_id":"65eec5c1d7d63c2ed0615421","avatarUrl":"/avatars/8c32f5e7d4b1940088bdec73c0b86fab.svg","isPro":false,"fullname":"Chenlu Ye","user":"Chenlu123","type":"user"},{"_id":"6726688788f2f9df271830ab","avatarUrl":"/avatars/1b7db87f241b0caa3e5d08298d5ff0c0.svg","isPro":false,"fullname":"Yuxing Liu","user":"yuxing6","type":"user"},{"_id":"6606434fa1f10a4f761b8d2f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6606434fa1f10a4f761b8d2f/jM2IQN-xFJi-Z2VcUPn93.jpeg","isPro":false,"fullname":"Pengcheng Wang","user":"PengchengW","type":"user"},{"_id":"64cb1ad1667f4f80852f6050","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64cb1ad1667f4f80852f6050/iOn5q_RyyBS99tObrO5Tc.png","isPro":false,"fullname":"Rui Pan","user":"research4pan","type":"user"},{"_id":"672599b9fab7cbfa939ab183","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/qD0INOZFSssS3FNg193Xk.png","isPro":false,"fullname":"Yifan Hao","user":"yifanhao99","type":"user"},{"_id":"64d45451c34a346181b130dd","avatarUrl":"/avatars/9bb8205b889337df5d321539c9b5d69d.svg","isPro":true,"fullname":"Rui Yang","user":"Ray2333","type":"user"},{"_id":"66954438643c258b90adc7ee","avatarUrl":"/avatars/68e704b97bda7bfbdfe0611e8fa19ab4.svg","isPro":true,"fullname":"Jerry Huang","user":"jerry128","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2509.03403

Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

Published on Sep 3, 2025

· Submitted by

Chenlu Ye on Sep 4, 2025

Upvote

Authors:

Abstract

PROF, a data curation method, improves reinforcement learning for mathematical reasoning by harmonizing process and outcome rewards, enhancing both final accuracy and intermediate reasoning quality.

AI-generated summary

Reinforcement learning with verifiable rewards (RLVR) has emerged to be a predominant paradigm for mathematical reasoning tasks, offering stable improvements in reasoning ability. However, Outcome Reward Models (ORMs) in RLVR are too coarse-grained to distinguish flawed reasoning within correct answers or valid reasoning within incorrect answers. This lack of granularity introduces noisy and misleading gradients significantly and hinders further progress in reasoning process quality. While Process Reward Models (PRMs) offer fine-grained guidance for intermediate steps, they frequently suffer from inaccuracies and are susceptible to reward hacking. To resolve this dilemma, we introduce PRocess cOnsistency Filter (PROF), an effective data process curation method that harmonizes noisy, fine-grained process rewards with accurate, coarse-grained outcome rewards. Rather than naively blending PRM and ORM in the objective function (arXiv:archive/2506.18896), PROF leverages their complementary strengths through consistency-driven sample selection. Our approach retains correct responses with higher averaged process values and incorrect responses with lower averaged process values, while maintaining positive/negative training sample balance. Extensive experiments demonstrate that our method not only consistently improves the final accuracy over 4% compared to the blending approaches, but also strengthens the quality of intermediate reasoning steps. Codes and training recipes are available at https://github.com/Chenluye99/PROF.