https://github.com/HKU-HealthAI/A-GRAE for the code base, and
https://yu7-code.github.io/A-GRAE-web/ for the project page

\n","updatedAt":"2026-02-13T14:41:11.936Z","author":{"_id":"663058bc2653ec94f4a6235f","avatarUrl":"/avatars/f55b8c3c8100d6b6d65ba61abc4fb014.svg","fullname":"Liangqiong Qu","name":"Liangqiong-QU","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":3,"identifiedLanguage":{"language":"en","probability":0.9057829976081848},"editors":["Liangqiong-QU"],"editorAvatarUrls":["/avatars/f55b8c3c8100d6b6d65ba61abc4fb014.svg"],"reactions":[],"isReport":false}},{"id":"698fd2042e87fc8e258ddbed","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2026-02-14T01:38:12.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Thickening-to-Thinning: Reward Shaping via Human-Inspired Learning Dynamics for LLM Reasoning](https://huggingface.co/papers/2602.04265) (2026)\n* [Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning](https://huggingface.co/papers/2602.02099) (2026)\n* [F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare](https://huggingface.co/papers/2602.06717) (2026)\n* [AMIR-GRPO: Inducing Implicit Preference Signals into GRPO](https://huggingface.co/papers/2601.03661) (2026)\n* [Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning Models](https://huggingface.co/papers/2601.03969) (2026)\n* [ETR: Outcome-Guided Elastic Trust Regions for Policy Optimization](https://huggingface.co/papers/2601.03723) (2026)\n* [Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards](https://huggingface.co/papers/2512.21625) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2026-02-14T01:38:12.514Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7482794523239136},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2602.05548","authors":[{"_id":"698e8c95cace060ff123ac11","name":"Zhiqi Yu","hidden":false},{"_id":"698e8c95cace060ff123ac12","name":"Zhangquan Chen","hidden":false},{"_id":"698e8c95cace060ff123ac13","name":"Mengting Liu","hidden":false},{"_id":"698e8c95cace060ff123ac14","name":"Heye Zhang","hidden":false},{"_id":"698e8c95cace060ff123ac15","user":{"_id":"663058bc2653ec94f4a6235f","avatarUrl":"/avatars/f55b8c3c8100d6b6d65ba61abc4fb014.svg","isPro":false,"fullname":"Liangqiong Qu","user":"Liangqiong-QU","type":"user"},"name":"Liangqiong Qu","status":"claimed_verified","statusLastChangedAt":"2026-02-13T09:36:55.677Z","hidden":false}],"publishedAt":"2026-02-05T11:07:14.000Z","submittedOnDailyAt":"2026-02-13T00:01:39.201Z","title":"Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation","submittedOnDailyBy":{"_id":"663058bc2653ec94f4a6235f","avatarUrl":"/avatars/f55b8c3c8100d6b6d65ba61abc4fb014.svg","isPro":false,"fullname":"Liangqiong Qu","user":"Liangqiong-QU","type":"user"},"summary":"Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we argue that these bottlenecks stem from an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE). This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically suppressing the advantages of correct trajectories encourages essential exploration. (ii) learning efficiency is maximized by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs.","upvotes":12,"discussionId":"698e8c95cace060ff123ac16","projectPage":"https://yu7-code.github.io/A-GRAE-web/","githubRepo":"https://github.com/HKU-HealthAI/A-GRAE","githubRepoAddedBy":"user","ai_summary":"Asymmetric Group Relative Advantage Estimation addresses exploration and difficulty adaptation challenges in reinforcement learning with large language models by dynamically modulating exploration incentives and sample difficulty focus.","ai_keywords":["Reinforcement Learning with Verifiable Rewards","GRPO","Group Relative Advantage Estimation","GRAE","asymmetric suppression","curriculum learning","sample-difficulty focus","exploration incentives","large language models","multi-modal large language models"],"githubStars":10,"organization":{"_id":"67ea9ecfc234715db8dbf339","name":"hkuhk","fullname":"The University of Hong Kong","avatar":"https://cdn-uploads.huggingface.co/production/uploads/67ea9e8d2d95c10a0da11b0c/FNnR4M7YqKRuG43N5771B.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"663058bc2653ec94f4a6235f","avatarUrl":"/avatars/f55b8c3c8100d6b6d65ba61abc4fb014.svg","isPro":false,"fullname":"Liangqiong Qu","user":"Liangqiong-QU","type":"user"},{"_id":"6613612f16a9b102745ec0e1","avatarUrl":"/avatars/3c6b9090ce1d15f67678a70370a073f6.svg","isPro":false,"fullname":"Yu","user":"bigfisher7","type":"user"},{"_id":"67887a429ede23ff0a964c98","avatarUrl":"/avatars/12dea48e4b37e8a7f5c9bbbc733f2348.svg","isPro":false,"fullname":"xing xie","user":"xing0916","type":"user"},{"_id":"668f440894dfc0ed1a7006ed","avatarUrl":"/avatars/fa0d328300b03bcbbf9b3a7532f28458.svg","isPro":false,"fullname":"Pengxin Guo","user":"gpx333","type":"user"},{"_id":"66632d290875aaaa914e7335","avatarUrl":"/avatars/07185efaa8a170f555ebf3eeef88bb6c.svg","isPro":false,"fullname":"Zonggen Li","user":"ZonggenLi","type":"user"},{"_id":"641b065a1911d3be6742ef04","avatarUrl":"/avatars/bceb4ad2a278b6e2a40af0b89c4fb48e.svg","isPro":true,"fullname":"Yinong Wang","user":"jcccy","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"6440f16a5d600fb095198f15","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6440f16a5d600fb095198f15/-J89Lelh3PPRVcc1ya_VS.png","isPro":false,"fullname":"Jiawei Liu","user":"nachifur","type":"user"},{"_id":"668d29504587d0a82beb4caa","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/668d29504587d0a82beb4caa/UmwTOEK5kQ4wuNn3D_qW5.jpeg","isPro":false,"fullname":"Yanzhao Shi","user":"Yanzhaoshi","type":"user"},{"_id":"66d8512c54209e9101811e8e","avatarUrl":"/avatars/62dfd8e6261108f2508efe678d5a2a57.svg","isPro":false,"fullname":"M Saad Salman","user":"MSS444","type":"user"},{"_id":"684d57f26e04c265777ead3f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/cuOj-bQqukSZreXgUJlfm.png","isPro":false,"fullname":"Joakim Lee","user":"Reinforcement4All","type":"user"},{"_id":"6458a99c3b81018d6b93aecb","avatarUrl":"/avatars/d90f16e8d3e49e095fea4cbd899837df.svg","isPro":false,"fullname":"jankin","user":"jankin123","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0,"organization":{"_id":"67ea9ecfc234715db8dbf339","name":"hkuhk","fullname":"The University of Hong Kong","avatar":"https://cdn-uploads.huggingface.co/production/uploads/67ea9e8d2d95c10a0da11b0c/FNnR4M7YqKRuG43N5771B.png"}}">

Papers

arxiv:2602.05548

Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

Published on Feb 5

· Submitted by

Liangqiong Qu on Feb 13

The University of Hong Kong

Upvote

Authors:

Liangqiong Qu

Abstract

Asymmetric Group Relative Advantage Estimation addresses exploration and difficulty adaptation challenges in reinforcement learning with large language models by dynamically modulating exploration incentives and sample difficulty focus.

AI-generated summary

Reinforcement Learning with Verifiable Rewards (RLVR), particularly GRPO, has become the standard for eliciting LLM reasoning. However, its efficiency in exploration and difficulty adaptation remains an open challenge. In this work, we argue that these bottlenecks stem from an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE). This symmetry induces two critical limitations: (i) at the group level, strict symmetry in weights between correct and incorrect trajectories leaves unsampled action logits unchanged, thereby hindering exploration of novel correct solution. (ii) at the sample level, the algorithm implicitly prioritizes medium-difficulty samples, remaining agnostic to the non-stationary demands of difficulty focus. Through controlled experiments, we reveal that this symmetric property is sub-optimal, yielding two pivotal insights: (i) asymmetrically suppressing the advantages of correct trajectories encourages essential exploration. (ii) learning efficiency is maximized by a curriculum-like transition-prioritizing simpler samples initially before gradually shifting to complex ones. Motivated by these findings, we propose Asymmetric GRAE (A-GRAE), which dynamically modulates exploration incentives and sample-difficulty focus. Experiments across seven benchmarks demonstrate that A-GRAE consistently improves GRPO and its variants across both LLMs and MLLMs.