@librarian-bot\n\t recommend

\n","updatedAt":"2025-06-16T05:42:09.644Z","author":{"_id":"5fad8602b8423e1d80b8a965","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5fad8602b8423e1d80b8a965/tRqTwcZmrGka8c1vFq2wX.jpeg","fullname":"Victor Gallego","name":"vicgalle","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":141,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7918877601623535},"editors":["vicgalle"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/5fad8602b8423e1d80b8a965/tRqTwcZmrGka8c1vFq2wX.jpeg"],"reactions":[],"isReport":false}},{"id":"6850c68c899fa19ee6f74458","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false},"createdAt":"2025-06-17T01:36:12.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [SGM: A Framework for Building Specification-Guided Moderation Filters](https://huggingface.co/papers/2505.19766) (2025)\n* [R3: Robust Rubric-Agnostic Reward Models](https://huggingface.co/papers/2505.13388) (2025)\n* [RewardAnything: Generalizable Principle-Following Reward Models](https://huggingface.co/papers/2506.03637) (2025)\n* [WikiPersonas: What Can We Learn From Personalized Alignment to Famous People?](https://huggingface.co/papers/2505.13257) (2025)\n* [DreamBoothDPO: Improving Personalized Generation using Direct Preference Optimization](https://huggingface.co/papers/2505.20975) (2025)\n* [Enhanced Whole Page Optimization via Mixed-Grained Reward Mechanism-Adapted Language Models](https://huggingface.co/papers/2506.09084) (2025)\n* [Aligning Large Language Models with Implicit Preferences from User-Generated Content](https://huggingface.co/papers/2506.04463) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2025-06-17T01:36:12.060Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7195132374763489},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2506.11702","authors":[{"_id":"684fae8360b4a34dbe007ac9","user":{"_id":"5fad8602b8423e1d80b8a965","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5fad8602b8423e1d80b8a965/tRqTwcZmrGka8c1vFq2wX.jpeg","isPro":false,"fullname":"Victor Gallego","user":"vicgalle","type":"user"},"name":"Víctor Gallego","status":"extracted_confirmed","statusLastChangedAt":"2025-06-16T05:42:55.745Z","hidden":false}],"publishedAt":"2025-06-13T12:17:38.000Z","submittedOnDailyAt":"2025-06-16T04:12:09.638Z","title":"Configurable Preference Tuning with Rubric-Guided Synthetic Data","submittedOnDailyBy":{"_id":"5fad8602b8423e1d80b8a965","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5fad8602b8423e1d80b8a965/tRqTwcZmrGka8c1vFq2wX.jpeg","isPro":false,"fullname":"Victor Gallego","user":"vicgalle","type":"user"},"summary":"Models of human feedback for AI alignment, such as those underpinning Direct\nPreference Optimization (DPO), often bake in a singular, static set of\npreferences, limiting adaptability. This paper challenges the assumption of\nmonolithic preferences by introducing Configurable Preference Tuning (CPT), a\nnovel framework for endowing language models with the ability to dynamically\nadjust their behavior based on explicit, human-interpretable directives. CPT\nleverages synthetically generated preference data, conditioned on system\nprompts derived from structured, fine-grained rubrics that define desired\nattributes like writing style. By fine-tuning with these rubric-guided\npreferences, the LLM learns to modulate its outputs at inference time in\nresponse to the system prompt, without retraining. This approach not only\noffers fine-grained control but also provides a mechanism for modeling more\nnuanced and context-dependent human feedback. Several experimental artifacts,\nsuch as training code, generated datasets and fine-tuned models are released at\nhttps://github.com/vicgalle/configurable-preference-tuning","upvotes":1,"discussionId":"684fae8460b4a34dbe007aca","githubRepo":"https://github.com/vicgalle/configurable-preference-tuning","githubRepoAddedBy":"user","ai_summary":"Configurable Preference Tuning enables language models to dynamically adjust their behavior based on human-interprettable directives, using rubric-guided preference data for fine-tuning and inference-time modulation.","ai_keywords":["Configurable Preference Tuning","Direct Preference Optimization","language models","fine-grained control","rubric-guided preferences","inference-time modulation"],"githubStars":2},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"5fad8602b8423e1d80b8a965","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5fad8602b8423e1d80b8a965/tRqTwcZmrGka8c1vFq2wX.jpeg","isPro":false,"fullname":"Victor Gallego","user":"vicgalle","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2506.11702

Configurable Preference Tuning with Rubric-Guided Synthetic Data

Published on Jun 13, 2025

· Submitted by

Victor Gallego on Jun 16, 2025

Upvote

Authors:

Víctor Gallego

Abstract

Configurable Preference Tuning enables language models to dynamically adjust their behavior based on human-interprettable directives, using rubric-guided preference data for fine-tuning and inference-time modulation.

AI-generated summary

Models of human feedback for AI alignment, such as those underpinning Direct Preference Optimization (DPO), often bake in a singular, static set of preferences, limiting adaptability. This paper challenges the assumption of monolithic preferences by introducing Configurable Preference Tuning (CPT), a novel framework for endowing language models with the ability to dynamically adjust their behavior based on explicit, human-interpretable directives. CPT leverages synthetically generated preference data, conditioned on system prompts derived from structured, fine-grained rubrics that define desired attributes like writing style. By fine-tuning with these rubric-guided preferences, the LLM learns to modulate its outputs at inference time in response to the system prompt, without retraining. This approach not only offers fine-grained control but also provides a mechanism for modeling more nuanced and context-dependent human feedback. Several experimental artifacts, such as training code, generated datasets and fine-tuned models are released at https://github.com/vicgalle/configurable-preference-tuning

View arXiv page View PDF GitHub 2 Add to collection