Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /home/zhenxiangba/zhenxiangba.com/public_html/phproxy-improved-master/index.php on line 456 Paper page - BOND: Aligning LLMs with Best-of-N Distillation
Congrats on this new work, would be cool to have it implemented in TRL (similar to DPO and other human preference tuning algorithms): https://github.com/huggingface/trl.
\n
Let me know if I need to connect you with the team!
\n
Cheers, Niels Open-source @ HF
\n","updatedAt":"2024-07-25T19:32:15.941Z","author":{"_id":"5f1158120c833276f61f1a84","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg","fullname":"Niels Rogge","name":"nielsr","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":1096,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8937240242958069},"editors":["nielsr"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"669f6d6d43a9a8c8554d8732"}}]},{"id":"66a059dd52e938fff6d3235c","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-07-24T01:33:17.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Adversarial Moment-Matching Distillation of Large Language Models](https://huggingface.co/papers/2406.02959) (2024)\n* [WARP: On the Benefits of Weight Averaged Rewarded Policies](https://huggingface.co/papers/2406.16768) (2024)\n* [SAIL: Self-Improving Efficient Online Alignment of Large Language Models](https://huggingface.co/papers/2406.15567) (2024)\n* [BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling](https://huggingface.co/papers/2406.00832) (2024)\n* [Robust Preference Optimization through Reward Model Distillation](https://huggingface.co/papers/2405.19316) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\n
The following papers were recommended by the Semantic Scholar API
BOND, a novel RLHF algorithm, distills the Best-of-N sampling strategy without significant computational overhead, enhancing the quality of generative language models.
AI-generated summary
Reinforcement learning from human feedback (RLHF) is a key driver of quality
and safety in state-of-the-art large language models. Yet, a surprisingly
simple and strong inference-time strategy is Best-of-N sampling that selects
the best generation among N candidates. In this paper, we propose Best-of-N
Distillation (BOND), a novel RLHF algorithm that seeks to emulate Best-of-N but
without its significant computational overhead at inference time. Specifically,
BOND is a distribution matching algorithm that forces the distribution of
generations from the policy to get closer to the Best-of-N distribution. We use
the Jeffreys divergence (a linear combination of forward and backward KL) to
balance between mode-covering and mode-seeking behavior, and derive an
iterative formulation that utilizes a moving anchor for efficiency. We
demonstrate the effectiveness of our approach and several design choices
through experiments on abstractive summarization and Gemma models. Aligning
Gemma policies with BOND outperforms other RLHF algorithms by improving results
on several benchmarks.
We present J-BOND 🕴️, a novel alignment method that steers the LLM towards the Best-of-N distribution via online distillation. This allows inheriting the strong properties of Best-of-N sampling, while requiring only a single sample at inference time.
To achieve this, J-BOND minimizes the Jeffreys divergence between the training policy and the Best-of-N distribution, trading off mode covering (forward KL) and mode seeking (backward KL) achieving the best of both divergences. Moreover, it implements an iterative distillation approach aiming at distilling the Best-of-N version of an Exponential Moving Average (EMA) anchor policy. This allows keeping reduced sample complexity and stable optimization, while the policy continuously improves its performance. We demonstrate our design choices and overall approach on an abstractive summarization task and for the fine tuning of Gemma. Aligning Gemma policies with J-BOND led to superior performance than standard RLHF baselines, with improvements on several benchmarks.
Congrats on this new work, would be cool to have it implemented in TRL (similar to DPO and other human preference tuning algorithms): https://github.com/huggingface/trl.
Let me know if I need to connect you with the team!