Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024)
Scaling Sparse Fine-Tuning to Large Language Models (2024)
Towards a Unified Language Model for Knowledge-Intensive Tasks Utilizing External Corpus (2024)
TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones (2023)
Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-02-17T01:21:47.796Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7246900200843811},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"65d311188120c6edb82e3ce9","author":{"_id":"638eb5f949de7ae552dd6211","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638eb5f949de7ae552dd6211/mJkQJGpn9tXV37N2VLFCh.jpeg","fullname":"Derek Thomas","name":"derek-thomas","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":131,"isUserFollowing":false},"createdAt":"2024-02-19T08:28:08.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"Awesome work!! Really cool idea, I love the paper, really well done.","html":"

Awesome work!! Really cool idea, I love the paper, really well done.

\n","updatedAt":"2024-02-19T08:28:48.118Z","author":{"_id":"638eb5f949de7ae552dd6211","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638eb5f949de7ae552dd6211/mJkQJGpn9tXV37N2VLFCh.jpeg","fullname":"Derek Thomas","name":"derek-thomas","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":131,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.4282374382019043},"editors":["derek-thomas"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/638eb5f949de7ae552dd6211/mJkQJGpn9tXV37N2VLFCh.jpeg"],"reactions":[{"reaction":"❤️","users":["Muennighoff","Jinkin"],"count":2}],"isReport":false}},{"id":"65d311a171ca8e594f896dfc","author":{"_id":"638eb5f949de7ae552dd6211","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638eb5f949de7ae552dd6211/mJkQJGpn9tXV37N2VLFCh.jpeg","fullname":"Derek Thomas","name":"derek-thomas","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":131,"isUserFollowing":false},"createdAt":"2024-02-19T08:30:25.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"\n![image.png](https://cdn-uploads.huggingface.co/production/uploads/638eb5f949de7ae552dd6211/ohW6CwtweaOXsJ9ZHoOO7.png)\n","html":"

$\"image.png\"$

\n","updatedAt":"2024-02-19T08:30:25.041Z","author":{"_id":"638eb5f949de7ae552dd6211","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/638eb5f949de7ae552dd6211/mJkQJGpn9tXV37N2VLFCh.jpeg","fullname":"Derek Thomas","name":"derek-thomas","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":131,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.37009355425834656},"editors":["derek-thomas"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/638eb5f949de7ae552dd6211/mJkQJGpn9tXV37N2VLFCh.jpeg"],"reactions":[{"reaction":"❤️","users":["Muennighoff","asg145","tachyphylaxis","Jinkin"],"count":4}],"isReport":false}},{"id":"661e701515eb952d8d255059","author":{"_id":"648d51064e46c331f0806b87","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/v6rEJybo42A251aXU2CaO.png","fullname":"HuangJunqin","name":"Jinkin","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false},"createdAt":"2024-04-16T12:33:25.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Solid Work !","html":"

Solid Work !

\n","updatedAt":"2024-04-16T12:33:25.114Z","author":{"_id":"648d51064e46c331f0806b87","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/v6rEJybo42A251aXU2CaO.png","fullname":"HuangJunqin","name":"Jinkin","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.3173835575580597},"editors":["Jinkin"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/v6rEJybo42A251aXU2CaO.png"],"reactions":[{"reaction":"❤️","users":["Muennighoff"],"count":1}],"isReport":false}},{"id":"6664c586f4b20b92ca3515dd","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":176,"isUserFollowing":false},"createdAt":"2024-06-08T20:56:38.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"# Revolutionizing NLP: Meet GRIT - The Unified Model for Text Generation and Embedding\n\nhttps://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/p5cZOsvylgnPS2OitLM47.mp4 \n\n## Links 🔗:\n👉 Subscribe: https://www.youtube.com/@Arxflix\n👉 Twitter: https://x.com/arxflix\n👉 LMNT (Partner): https://lmnt.com/\n\n\nBy Arxflix\n![9t4iCUHx_400x400-1.jpg](https://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/v4S5zBurs0ouGNwYj1GEd.jpeg)","html":"

\n\t\n\t\t\n\t\n\t\n\t\tRevolutionizing NLP: Meet GRIT - The Unified Model for Text Generation and Embedding\n\t\n

\n\n

\n\t\n\t\t\n\t\n\t\n\t\tLinks 🔗:\n\t\n

👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/

By Arxflix
$\"9t4iCUHx_400x400-1.jpg\"$

\n","updatedAt":"2024-06-08T20:56:38.737Z","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":176,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.4929705560207367},"editors":["blanchon"],"editorAvatarUrls":["/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2402.09906","authors":[{"_id":"65cec094e1c8a3d33cf694d7","user":{"_id":"5f1eb362eec0ad2a071ad6e2","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png","isPro":false,"fullname":"Niklas Muennighoff","user":"Muennighoff","type":"user"},"name":"Niklas Muennighoff","status":"admin_assigned","statusLastChangedAt":"2024-02-16T10:30:18.105Z","hidden":false},{"_id":"65cec094e1c8a3d33cf694d8","user":{"_id":"632d53951538d4798a73c849","avatarUrl":"/avatars/b7d0a895e669bcd1303c4716b5401c36.svg","isPro":false,"fullname":"Hongjin SU","user":"multi-train","type":"user"},"name":"Hongjin Su","status":"claimed_verified","statusLastChangedAt":"2024-07-19T11:25:09.019Z","hidden":false},{"_id":"65cec094e1c8a3d33cf694d9","name":"Liang Wang","hidden":false},{"_id":"65cec094e1c8a3d33cf694da","user":{"_id":"6341342a9948f573f36d77da","avatarUrl":"/avatars/813717cb180b2ce566421b43dadabb43.svg","isPro":false,"fullname":"nyanyanya","user":"nyanyanya","type":"user"},"name":"Nan Yang","status":"claimed_verified","statusLastChangedAt":"2024-05-13T07:49:59.502Z","hidden":false},{"_id":"65cec094e1c8a3d33cf694db","user":{"_id":"6368c512fbfe97c16a40baba","avatarUrl":"/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg","isPro":false,"fullname":"Furu Wei","user":"thegenerality","type":"user"},"name":"Furu Wei","status":"admin_assigned","statusLastChangedAt":"2024-02-16T10:34:15.306Z","hidden":false},{"_id":"65cec094e1c8a3d33cf694dc","name":"Tao Yu","hidden":false},{"_id":"65cec094e1c8a3d33cf694dd","user":{"_id":"6230c6ecfd8b720a5648f6c4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1652253065548-6230c6ecfd8b720a5648f6c4.jpeg","isPro":false,"fullname":"Amanpreet Singh","user":"aps","type":"user"},"name":"Amanpreet Singh","status":"claimed_verified","statusLastChangedAt":"2024-02-19T09:34:42.650Z","hidden":false},{"_id":"65cec094e1c8a3d33cf694de","user":{"_id":"61dc997715b47073db1620dc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg","isPro":false,"fullname":"Douwe Kiela","user":"douwekiela","type":"user"},"name":"Douwe Kiela","status":"admin_assigned","statusLastChangedAt":"2024-02-16T10:34:07.576Z","hidden":false}],"publishedAt":"2024-02-15T12:12:19.000Z","submittedOnDailyAt":"2024-02-16T02:09:46.630Z","title":"Generative Representational Instruction Tuning","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"All text-based language problems can be reduced to either generation or\nembedding. Current models only perform well at one or the other. We introduce\ngenerative representational instruction tuning (GRIT) whereby a large language\nmodel is trained to handle both generative and embedding tasks by\ndistinguishing between them through instructions. Compared to other open\nmodels, our resulting GritLM 7B sets a new state of the art on the Massive Text\nEmbedding Benchmark (MTEB) and outperforms all models up to its size on a range\nof generative tasks. By scaling up further, GritLM 8x7B outperforms all open\ngenerative language models that we tried while still being among the best\nembedding models. Notably, we find that GRIT matches training on only\ngenerative or embedding data, thus we can unify both at no performance loss.\nAmong other benefits, the unification via GRIT speeds up Retrieval-Augmented\nGeneration (RAG) by > 60% for long documents, by no longer requiring separate\nretrieval and generation models. Models, code, etc. are freely available at\nhttps://github.com/ContextualAI/gritlm.","upvotes":54,"discussionId":"65cec095e1c8a3d33cf694f9","githubRepo":"https://github.com/contextualai/gritlm","githubRepoAddedBy":"auto","ai_summary":"GrIT allows large language models to excel at both generative and embedding tasks through task instruction, leading to new state-of-the-art performance without sacrificing efficiency.","ai_keywords":["generative representational instruction tuning","GRIT","GritLM","Massive Text Embedding Benchmark","MTEB","generative language models","embedding models","Retrieval-Augmented Generation","RAG"],"githubStars":686},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"612ee6a7b960e78c6d2319d4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg","isPro":false,"fullname":"Qian Liu","user":"SivilTaram","type":"user"},{"_id":"637f0eb22438d7485b8ef5d7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/637f0eb22438d7485b8ef5d7/70h7dekqj7LuBobOXckmJ.jpeg","isPro":false,"fullname":"Ming Li","user":"limingcv","type":"user"},{"_id":"6101c620900eaa0057c2ce1d","avatarUrl":"/avatars/bd282166c120711c65b5409dc860ac58.svg","isPro":false,"fullname":"Abdel-Dayane Marcos","user":"admarcosai","type":"user"},{"_id":"6167229d6b5a2f75073ec599","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6167229d6b5a2f75073ec599/8o_8YndZpdjoFmpYpnU48.jpeg","isPro":false,"fullname":"Matt Mistele","user":"mmistele","type":"user"},{"_id":"6486bb3d4c025cf3c41e7767","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6486bb3d4c025cf3c41e7767/1I6yYhv06rHX1_1iIOdjP.png","isPro":false,"fullname":"Quentin Tardif","user":"ntnq","type":"user"},{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"},{"_id":"6436e9c41046b34b54395d92","avatarUrl":"/avatars/856c6e777c21aa02824639430ed29a5b.svg","isPro":false,"fullname":"Ahmet Ustun","user":"ahmetustun","type":"user"},{"_id":"6555125a4f361968f0e3aad7","avatarUrl":"/avatars/e7692d82804338f21ecdc6e731f5c5ea.svg","isPro":false,"fullname":"marinaretikof","user":"marinaretik","type":"user"},{"_id":"6311bca0ae8896941da24e66","avatarUrl":"/avatars/48de64894fc3c9397e26e4d6da3ff537.svg","isPro":false,"fullname":"Fynn Kröger","user":"fynnkroeger","type":"user"},{"_id":"64b26c035e1230a79f897880","avatarUrl":"/avatars/5427b05b3ef627d4d8281f9a33bb98ab.svg","isPro":false,"fullname":"zhangwenbin","user":"ExceedZhang","type":"user"},{"_id":"63b6f2e752c02ae8acbaa4d8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1672934038280-noauth.jpeg","isPro":false,"fullname":"Habibullah Akbar","user":"ChavyvAkvar","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":2}">

Papers

arxiv:2402.09906

Generative Representational Instruction Tuning

Published on Feb 15, 2024

· Submitted by

AK on Feb 16, 2024

#2 Paper of the day

Upvote

Authors:

Niklas Muennighoff ,

Hongjin Su ,

Nan Yang ,

Furu Wei ,

Amanpreet Singh ,

Douwe Kiela

Abstract

GrIT allows large language models to excel at both generative and embedding tasks through task instruction, leading to new state-of-the-art performance without sacrificing efficiency.

AI-generated summary

All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8x7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm.