https://github.com/seokhyunan/response-tuning.

\n","updatedAt":"2024-10-10T11:06:37.371Z","author":{"_id":"617017b7544a1c62b298b87d","avatarUrl":"/avatars/a569d976eb26807bd126de63a6b808fd.svg","fullname":"Seokhyun An","name":"seokhyun","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"editors":["seokhyun"],"editorAvatarUrls":["/avatars/a569d976eb26807bd126de63a6b808fd.svg"],"reactions":[],"isReport":false}},{"id":"670880d196832e023799f5fc","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-10-11T01:35:13.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data](https://huggingface.co/papers/2409.00096) (2024)\n* [SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe](https://huggingface.co/papers/2410.05248) (2024)\n* [Self-Judge: Selective Instruction Following with Alignment Self-Evaluation](https://huggingface.co/papers/2409.00935) (2024)\n* [RNR: Teaching Large Language Models to Follow Roles and Rules](https://huggingface.co/papers/2409.13733) (2024)\n* [WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback](https://huggingface.co/papers/2408.15549) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-10-11T01:35:13.033Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2410.02465","authors":[{"_id":"6707b25a40b1f23d30085d85","user":{"_id":"617017b7544a1c62b298b87d","avatarUrl":"/avatars/a569d976eb26807bd126de63a6b808fd.svg","isPro":false,"fullname":"Seokhyun An","user":"seokhyun","type":"user"},"name":"Seokhyun An","status":"claimed_verified","statusLastChangedAt":"2024-10-10T10:59:05.265Z","hidden":false},{"_id":"6707b25a40b1f23d30085d86","user":{"_id":"64d7162b3ca2924d6e8593e4","avatarUrl":"/avatars/51b28e4652bbb9d35adbd62aece59cd0.svg","isPro":false,"fullname":"Hyounghun Kim","user":"khh3323","type":"user"},"name":"Hyounghun Kim","status":"claimed_verified","statusLastChangedAt":"2024-10-10T14:03:08.497Z","hidden":false}],"publishedAt":"2024-10-03T13:15:19.000Z","submittedOnDailyAt":"2024-10-10T09:36:37.367Z","title":"Response Tuning: Aligning Large Language Models without Instruction","submittedOnDailyBy":{"_id":"617017b7544a1c62b298b87d","avatarUrl":"/avatars/a569d976eb26807bd126de63a6b808fd.svg","isPro":false,"fullname":"Seokhyun An","user":"seokhyun","type":"user"},"summary":"Instruction tuning-supervised fine-tuning using instruction-response pairs-is\na foundational step in transitioning pre-trained Large Language Models (LLMs)\ninto helpful and safe chat assistants. Our hypothesis is that establishing an\nadequate output space can enable such a transition given the capabilities\ninherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT),\nwhich eliminates the instruction-conditioning step in instruction tuning and\nsolely focuses on response space supervision. Our experiments demonstrate that\nRT models, trained only using responses, can effectively respond to a wide\nrange of instructions and exhibit helpfulness comparable to that of their\ninstruction-tuned counterparts. Furthermore, we observe that controlling the\ntraining response distribution can significantly improve their user preference\nor elicit target behaviors such as refusing assistance for unsafe queries. Our\nfindings illuminate the role of establishing an adequate output space in\nalignment, highlighting the potential of the extensive inherent capabilities of\npre-trained LLMs.","upvotes":13,"discussionId":"6707b25b40b1f23d30085dd1","githubRepo":"https://github.com/seokhyunan/response-tuning","githubRepoAddedBy":"auto","ai_summary":"Response Tuning, an approach focusing on response space supervision without instruction conditioning, effectively transitions pre-trained LLMs into helpful and safe chat assistants.","ai_keywords":["instruction tuning","instruction-response pairs","Large Language Models (LLMs)","Response Tuning (RT)","response space supervision","user preference","target behaviors","alignment","capabilities"],"githubStars":8},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"617017b7544a1c62b298b87d","avatarUrl":"/avatars/a569d976eb26807bd126de63a6b808fd.svg","isPro":false,"fullname":"Seokhyun An","user":"seokhyun","type":"user"},{"_id":"66cec2430c76eb16dccea9a9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2y56awsS8juXgeyQwJ7hp.png","isPro":false,"fullname":"Daehyun KIM","user":"Dhyuniiiiiiiiiii","type":"user"},{"_id":"63332274431dafafae24be22","avatarUrl":"/avatars/58259c3f5533df7e3c5c40ba8526f19e.svg","isPro":false,"fullname":"Bae","user":"minwook","type":"user"},{"_id":"64e9ab01e74f54587cb71dd3","avatarUrl":"/avatars/374599b991b5bf64c3380be3ece8a287.svg","isPro":false,"fullname":"Wooyeol Lee","user":"thisiswooyeol","type":"user"},{"_id":"64d7162b3ca2924d6e8593e4","avatarUrl":"/avatars/51b28e4652bbb9d35adbd62aece59cd0.svg","isPro":false,"fullname":"Hyounghun Kim","user":"khh3323","type":"user"},{"_id":"64ca5553c7f30fbf7b651997","avatarUrl":"/avatars/c2765aef0c0f53534e04ab60821dec40.svg","isPro":false,"fullname":"Chanbin Lee","user":"terateil","type":"user"},{"_id":"6707c949ceaa2578b5645450","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zPlCru7Y9bQBCTYcAJ6Ah.png","isPro":false,"fullname":"Jeff badman","user":"Lastat22","type":"user"},{"_id":"66264aa3ec2491aadcf7084b","avatarUrl":"/avatars/9853496d6370164c43d51f98441bc425.svg","isPro":false,"fullname":"MZK","user":"bibpap","type":"user"},{"_id":"65decc75beffeb39ba679eba","avatarUrl":"/avatars/735b678bd5863a0c1b1bdd3bbf8858fa.svg","isPro":true,"fullname":"r","user":"oceansweep","type":"user"},{"_id":"62a42f22c683d02f5b63320c","avatarUrl":"/avatars/bc611abe9c4ef8d378123cb8ac9fdbf2.svg","isPro":true,"fullname":"Qiyuan Zhang","user":"DonJoey","type":"user"},{"_id":"6270324ebecab9e2dcf245de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6270324ebecab9e2dcf245de/cMbtWSasyNlYc9hvsEEzt.jpeg","isPro":false,"fullname":"Kye Gomez","user":"kye","type":"user"},{"_id":"64d4615cf8082bf19b916492","avatarUrl":"/avatars/8e1b59565ec5e4b31090cf1b911781b9.svg","isPro":false,"fullname":"wongyukim","user":"wongyukim","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2410.02465

Response Tuning: Aligning Large Language Models without Instruction

Published on Oct 3, 2024

· Submitted by

Seokhyun An on Oct 10, 2024

Upvote

Authors:

Seokhyun An ,

Hyounghun Kim

Abstract

Response Tuning, an approach focusing on response space supervision without instruction conditioning, effectively transitions pre-trained LLMs into helpful and safe chat assistants.

AI-generated summary

Instruction tuning-supervised fine-tuning using instruction-response pairs-is a foundational step in transitioning pre-trained Large Language Models (LLMs) into helpful and safe chat assistants. Our hypothesis is that establishing an adequate output space can enable such a transition given the capabilities inherent in pre-trained LLMs. To verify this, we propose Response Tuning (RT), which eliminates the instruction-conditioning step in instruction tuning and solely focuses on response space supervision. Our experiments demonstrate that RT models, trained only using responses, can effectively respond to a wide range of instructions and exhibit helpfulness comparable to that of their instruction-tuned counterparts. Furthermore, we observe that controlling the training response distribution can significantly improve their user preference or elicit target behaviors such as refusing assistance for unsafe queries. Our findings illuminate the role of establishing an adequate output space in alignment, highlighting the potential of the extensive inherent capabilities of pre-trained LLMs.