Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models (2024)
Long-Context Language Modeling with Parallel Context Encoding (2024)
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (2024)
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference (2024)
Cross-Architecture Transfer Learning for Linear-Cost Inference Transformers (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-04-13T01:14:41.980Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":317,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7475188970565796},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"6619ee883557013b634b38f5","author":{"_id":"65afe3fd7c11edbf6e1a1277","avatarUrl":"/avatars/a35c3e29d712d1fb062b5eb8887d46a6.svg","fullname":"Robin Williams","name":"bfuzzy1","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":10,"isUserFollowing":false},"createdAt":"2024-04-13T02:31:36.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Nice job!","html":"

Nice job!

\n","updatedAt":"2024-04-13T02:31:36.682Z","author":{"_id":"65afe3fd7c11edbf6e1a1277","avatarUrl":"/avatars/a35c3e29d712d1fb062b5eb8887d46a6.svg","fullname":"Robin Williams","name":"bfuzzy1","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":10,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5790669918060303},"editors":["bfuzzy1"],"editorAvatarUrls":["/avatars/a35c3e29d712d1fb062b5eb8887d46a6.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2404.07839","authors":[{"_id":"661890a365e0419878fcb603","user":{"_id":"65d6647110d158dece540b5c","avatarUrl":"/avatars/fc0bbdeb0c23ceb9afbf2a658fc888f2.svg","isPro":false,"fullname":"Aleksandar Botev","user":"botevmg","type":"user"},"name":"Aleksandar Botev","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:01:11.116Z","hidden":false},{"_id":"661890a365e0419878fcb604","user":{"_id":"661415d9f9e58fe02bd8661d","avatarUrl":"/avatars/7a28bb507b8bac72a8b161e774b19fbd.svg","isPro":false,"fullname":"Soham De","user":"sohamde","type":"user"},"name":"Soham De","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:01:25.519Z","hidden":false},{"_id":"661890a365e0419878fcb605","name":"Samuel L Smith","hidden":false},{"_id":"661890a365e0419878fcb606","user":{"_id":"660c2ccb3aa98dc4b27de7b3","avatarUrl":"/avatars/2b19094e1b876a1be1f763dc941ff0bf.svg","isPro":false,"fullname":"Anushan Fernando","user":"AnushanF","type":"user"},"name":"Anushan Fernando","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:01:44.512Z","hidden":false},{"_id":"661890a365e0419878fcb607","user":{"_id":"660dd120fc9742e05187cffd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/660dd120fc9742e05187cffd/bFZHJMJcN6rJGo_Yala89.png","isPro":false,"fullname":"George-Cristian Muraru","user":"gmuraru","type":"user"},"name":"George-Cristian Muraru","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:01:50.783Z","hidden":false},{"_id":"661890a365e0419878fcb608","user":{"_id":"66167eec24c94fd8a2427513","avatarUrl":"/avatars/24ff7acfbf064a3a37dcad25ef9d45ee.svg","isPro":false,"fullname":"Ruba Haroun","user":"RubaHaroun","type":"user"},"name":"Ruba Haroun","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:01:57.539Z","hidden":false},{"_id":"661890a365e0419878fcb609","user":{"_id":"65fc77aad6206abe6cb611f1","avatarUrl":"/avatars/306334124d90d1b4d4f82fb47f56156f.svg","isPro":false,"fullname":"Leonard Berrada","user":"lberrada","type":"user"},"name":"Leonard Berrada","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:02:04.421Z","hidden":false},{"_id":"661890a365e0419878fcb60a","user":{"_id":"64b9310403124195cd9778ec","avatarUrl":"/avatars/57c594d3d0f97d3010b15b6a0806451c.svg","isPro":false,"fullname":"Razvan Pascanu","user":"razp","type":"user"},"name":"Razvan Pascanu","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:02:10.349Z","hidden":false},{"_id":"661890a365e0419878fcb60b","name":"Pier Giuseppe Sessa","hidden":false},{"_id":"661890a365e0419878fcb60c","user":{"_id":"65fc7c0da57b88a765aea493","avatarUrl":"/avatars/440f88769da1113d6158fe7e0514ead3.svg","isPro":false,"fullname":"Robert Dadashi","user":"ddsh","type":"user"},"name":"Robert Dadashi","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:02:23.830Z","hidden":false},{"_id":"661890a365e0419878fcb60d","user":{"_id":"667a7b2e1764931987803b46","avatarUrl":"/avatars/da2ee162f33c681e3d1c4d1e6d44dcbd.svg","isPro":false,"fullname":"Léonard Hussenot","user":"leonardhussenot","type":"user"},"name":"Léonard Hussenot","status":"claimed_verified","statusLastChangedAt":"2025-03-28T14:22:40.708Z","hidden":false},{"_id":"661890a365e0419878fcb60e","user":{"_id":"65afb7dbdd6bdfd73cd8e609","avatarUrl":"/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg","isPro":false,"fullname":"Johan Ferret","user":"ferretj","type":"user"},"name":"Johan Ferret","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:02:35.672Z","hidden":false},{"_id":"661890a365e0419878fcb60f","name":"Sertan Girgin","hidden":false},{"_id":"661890a365e0419878fcb610","name":"Olivier Bachem","hidden":false},{"_id":"661890a365e0419878fcb611","user":{"_id":"65d667858ac5cf899e3b66c4","avatarUrl":"/avatars/340468ee8cd9125eceae23a9b8e536a5.svg","isPro":false,"fullname":"Alek Andreev","user":"alekandreev","type":"user"},"name":"Alek Andreev","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:02:54.909Z","hidden":false},{"_id":"661890a365e0419878fcb612","name":"Kathleen Kenealy","hidden":false},{"_id":"661890a365e0419878fcb613","name":"Thomas Mesnard","hidden":false},{"_id":"661890a365e0419878fcb614","name":"Cassidy Hardin","hidden":false},{"_id":"661890a365e0419878fcb615","user":{"_id":"65d7644d19bf1e60614dee6e","avatarUrl":"/avatars/422a8d28fa9d9ff2c3ec1406352cb346.svg","isPro":false,"fullname":"Surya Bhupatiraju","user":"suryabhupa","type":"user"},"name":"Surya Bhupatiraju","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:03:24.765Z","hidden":false},{"_id":"661890a365e0419878fcb616","name":"Shreya Pathak","hidden":false},{"_id":"661890a365e0419878fcb617","user":{"_id":"65d74b9b7713dc8fd8956d1c","avatarUrl":"/avatars/5d5873861134bb079753c48304a7b7c2.svg","isPro":false,"fullname":"Laurent Sifre","user":"laurentsifre","type":"user"},"name":"Laurent Sifre","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:03:35.899Z","hidden":false},{"_id":"661890a365e0419878fcb618","name":"Morgane Rivière","hidden":false},{"_id":"661890a365e0419878fcb619","user":{"_id":"6226953f7551f131b4eba5c5","avatarUrl":"/avatars/eec8fa43fc37f546d9fef118a138d8a4.svg","isPro":false,"fullname":"Mihir Sanjay Kale","user":"mihirkale","type":"user"},"name":"Mihir Sanjay Kale","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:03:50.288Z","hidden":false},{"_id":"661890a365e0419878fcb61a","name":"Juliette Love","hidden":false},{"_id":"661890a365e0419878fcb61b","user":{"_id":"6558091ecafc48de36f848a8","avatarUrl":"/avatars/3b7e8327d369436d02fe8c58dde036c2.svg","isPro":false,"fullname":"Pouya Tafti","user":"pdt","type":"user"},"name":"Pouya Tafti","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:04:09.855Z","hidden":false},{"_id":"661890a365e0419878fcb61c","name":"Armand Joulin","hidden":false},{"_id":"661890a365e0419878fcb61d","user":{"_id":"62f82ebd870a3f98bbf9e62a","avatarUrl":"/avatars/3b03299e128abf2c8ba82db704855de7.svg","isPro":false,"fullname":"Noah Fiedel","user":"nfiedel","type":"user"},"name":"Noah Fiedel","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:04:21.963Z","hidden":false},{"_id":"661890a365e0419878fcb61e","user":{"_id":"62a47ec0fdd7a13aac264904","avatarUrl":"/avatars/f2ff23645203ed2399cfe8b447af437a.svg","isPro":false,"fullname":"Evan Senter","user":"evansenter","type":"user"},"name":"Evan Senter","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:04:30.429Z","hidden":false},{"_id":"661890a365e0419878fcb61f","name":"Yutian Chen","hidden":false},{"_id":"661890a365e0419878fcb620","user":{"_id":"64f5121aa30d6b0a26c1b7e1","avatarUrl":"/avatars/862fe8ed200e4759835661bab161ad09.svg","isPro":false,"fullname":"Srivatsan Srinivasan","user":"srivatr","type":"user"},"name":"Srivatsan Srinivasan","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:04:56.234Z","hidden":false},{"_id":"661890a365e0419878fcb621","name":"Guillaume Desjardins","hidden":false},{"_id":"661890a365e0419878fcb622","name":"David Budden","hidden":false},{"_id":"661890a365e0419878fcb623","name":"Arnaud Doucet","hidden":false},{"_id":"661890a365e0419878fcb624","name":"Sharad Vikram","hidden":false},{"_id":"661890a365e0419878fcb625","name":"Adam Paszke","hidden":false},{"_id":"661890a365e0419878fcb626","name":"Trevor Gale","hidden":false},{"_id":"661890a365e0419878fcb627","name":"Sebastian Borgeaud","hidden":false},{"_id":"661890a365e0419878fcb628","name":"Charlie Chen","hidden":false},{"_id":"661890a365e0419878fcb629","name":"Andy Brock","hidden":false},{"_id":"661890a365e0419878fcb62a","name":"Antonia Paterson","hidden":false},{"_id":"661890a365e0419878fcb62b","name":"Jenny Brennan","hidden":false},{"_id":"661890a365e0419878fcb62c","user":{"_id":"626b23fc822e3d85324babed","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg","isPro":false,"fullname":"Meg Risdal","user":"megrisdal","type":"user"},"name":"Meg Risdal","status":"admin_assigned","statusLastChangedAt":"2024-04-12T08:28:39.010Z","hidden":false},{"_id":"661890a365e0419878fcb62d","name":"Raj Gundluru","hidden":false},{"_id":"661890a365e0419878fcb62e","name":"Nesh Devanathan","hidden":false},{"_id":"661890a365e0419878fcb62f","user":{"_id":"644fadb9fc5e5ff919aed863","avatarUrl":"/avatars/9b646ff3eeda1f01cc2b1cb86d1634f9.svg","isPro":false,"fullname":"Paul Mooney","user":"paulmooney","type":"user"},"name":"Paul Mooney","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:05:57.471Z","hidden":false},{"_id":"661890a365e0419878fcb630","user":{"_id":"641415cd5189abde8cbbd900","avatarUrl":"/avatars/401807f3174492ab64b0197427f21e0b.svg","isPro":false,"fullname":"Nilay Chauhan","user":"nilaychauhan","type":"user"},"name":"Nilay Chauhan","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:07:38.639Z","hidden":false},{"_id":"661890a365e0419878fcb631","user":{"_id":"627a9aa0f967ef475abe8037","avatarUrl":"/avatars/47cf94d6c015577d0a860b617c9ea2b8.svg","isPro":false,"fullname":"Phil Culliton","user":"PhilCulliton","type":"user"},"name":"Phil Culliton","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:07:30.197Z","hidden":false},{"_id":"661890a365e0419878fcb632","user":{"_id":"637a4a31e20e8765c5220b77","avatarUrl":"/avatars/eb8e5b3bf2c1c99dad4f35ac24788669.svg","isPro":false,"fullname":"Luiz Gustavo Martins","user":"gusthema","type":"user"},"name":"Luiz GUStavo Martins","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:07:22.567Z","hidden":false},{"_id":"661890a365e0419878fcb633","name":"Elisa Bandy","hidden":false},{"_id":"661890a365e0419878fcb634","user":{"_id":"65f35e8bc2d214f884656a67","avatarUrl":"/avatars/2a46a832328a1269d0077097f6a121ca.svg","isPro":false,"fullname":"David Huntsperger","user":"dhuntsperger","type":"user"},"name":"David Huntsperger","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:07:10.781Z","hidden":false},{"_id":"661890a365e0419878fcb635","name":"Glenn Cameron","hidden":false},{"_id":"661890a365e0419878fcb636","user":{"_id":"62441cb7456803e95009a08f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674683851722-62441cb7456803e95009a08f.jpeg","isPro":false,"fullname":"Arthur Zucker","user":"ArthurZ","type":"user"},"name":"Arthur Zucker","status":"claimed_verified","statusLastChangedAt":"2024-04-12T14:49:59.500Z","hidden":false},{"_id":"661890a365e0419878fcb637","name":"Tris Warkentin","hidden":false},{"_id":"661890a365e0419878fcb638","user":{"_id":"60e473dcdefbf317ecd88035","avatarUrl":"/avatars/e5c069e6cfc73ea5d3d6c2a437490cdd.svg","isPro":false,"fullname":"LUDOVIC PERAN","user":"LudovicPeran","type":"user"},"name":"Ludovic Peran","status":"admin_assigned","statusLastChangedAt":"2024-04-18T12:06:46.212Z","hidden":false},{"_id":"661890a365e0419878fcb639","name":"Minh Giang","hidden":false},{"_id":"661890a365e0419878fcb63a","name":"Zoubin Ghahramani","hidden":false},{"_id":"661890a365e0419878fcb63b","name":"Clément Farabet","hidden":false},{"_id":"661890a365e0419878fcb63c","name":"Koray Kavukcuoglu","hidden":false},{"_id":"661890a365e0419878fcb63d","name":"Demis Hassabis","hidden":false},{"_id":"661890a365e0419878fcb63e","name":"Raia Hadsell","hidden":false},{"_id":"661890a365e0419878fcb63f","name":"Yee Whye Teh","hidden":false},{"_id":"661890a365e0419878fcb640","name":"Nando de Frietas","hidden":false}],"publishedAt":"2024-04-11T15:27:22.000Z","submittedOnDailyAt":"2024-04-12T00:08:44.229Z","title":"RecurrentGemma: Moving Past Transformers for Efficient Open Language\n Models","submittedOnDailyBy":{"_id":"60f1abe7544c2adfd699860c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg","isPro":false,"fullname":"AK","user":"akhaliq","type":"user"},"summary":"We introduce RecurrentGemma, an open language model which uses Google's novel\nGriffin architecture. Griffin combines linear recurrences with local attention\nto achieve excellent performance on language. It has a fixed-sized state, which\nreduces memory use and enables efficient inference on long sequences. We\nprovide a pre-trained model with 2B non-embedding parameters, and an\ninstruction tuned variant. Both models achieve comparable performance to\nGemma-2B despite being trained on fewer tokens.","upvotes":48,"discussionId":"661890a465e0419878fcb65d","githubRepo":"https://github.com/google-deepmind/recurrentgemma","githubRepoAddedBy":"auto","ai_summary":"RecurrentGemma, using Griffin architecture, achieves excellent language performance with efficient memory usage and fewer training tokens compared to Gemma-2B.","ai_keywords":["RecurrentGemma","Griffin architecture","linear recurrences","local attention","fixed-sized state","efficient inference","long sequences"],"githubStars":663},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6101c620900eaa0057c2ce1d","avatarUrl":"/avatars/bd282166c120711c65b5409dc860ac58.svg","isPro":false,"fullname":"Abdel-Dayane Marcos","user":"admarcosai","type":"user"},{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","isPro":true,"fullname":"Julien BLANCHON","user":"blanchon","type":"user"},{"_id":"64316678dec2a70d8130aa9d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EAS7OJwvyInle8J7IIBbw.jpeg","isPro":false,"fullname":"Levi Sverdlov","user":"Sverd","type":"user"},{"_id":"653ac9869d53a3f8514b7874","avatarUrl":"/avatars/69d5406f2662f7668b71b59fb6e18e90.svg","isPro":false,"fullname":"Linkun","user":"hugg1ngfac3","type":"user"},{"_id":"6358edff3b3638bdac83f7ac","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1666772404424-noauth.jpeg","isPro":false,"fullname":"Pratyay Banerjee","user":"Neilblaze","type":"user"},{"_id":"655ac762cb17ec19ef82719b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655ac762cb17ec19ef82719b/1kDncYrGLYS_2SR8cNdAL.png","isPro":false,"fullname":"Welcome to matlok","user":"matlok","type":"user"},{"_id":"63a7422854f1d0225b075bfc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63a7422854f1d0225b075bfc/XGYAcDPZG5ZEsNBWG6guw.jpeg","isPro":true,"fullname":"lhl","user":"leonardlin","type":"user"},{"_id":"637f0eb22438d7485b8ef5d7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/637f0eb22438d7485b8ef5d7/70h7dekqj7LuBobOXckmJ.jpeg","isPro":false,"fullname":"Ming Li","user":"limingcv","type":"user"},{"_id":"6538119803519fddb4a17e10","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg","isPro":false,"fullname":"samusenps","user":"samusenps","type":"user"},{"_id":"60aef0fbee40717d1a8fa6a5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1624676266012-60aef0fbee40717d1a8fa6a5.png","isPro":false,"fullname":"Mayank Bhaskar","user":"cataluna84","type":"user"},{"_id":"60cb51422424f83d03935470","avatarUrl":"/avatars/f071939cfd6fdd08a8399bf9a7a96fa2.svg","isPro":false,"fullname":"Oleksiy Syvokon","user":"osyvokon","type":"user"},{"_id":"62ec1bfc7e6d199d7f785490","avatarUrl":"/avatars/c45a011c4cdd6dd46b2f6c354f0fb599.svg","isPro":false,"fullname":"Zoroa Strella","user":"ZoroaStrella","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2404.07839

RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

Published on Apr 11, 2024

· Submitted by

AK on Apr 12, 2024

Upvote

Authors:

Aleksandar Botev ,

Soham De ,

Anushan Fernando ,

George-Cristian Muraru ,

Ruba Haroun ,

Leonard Berrada ,

Razvan Pascanu ,

Robert Dadashi ,

Léonard Hussenot ,

Johan Ferret ,

Alek Andreev ,

Surya Bhupatiraju ,

Laurent Sifre ,

Abstract

RecurrentGemma, using Griffin architecture, achieves excellent language performance with efficient memory usage and fewer training tokens compared to Gemma-2B.

AI-generated summary

We introduce RecurrentGemma, an open language model which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide a pre-trained model with 2B non-embedding parameters, and an instruction tuned variant. Both models achieve comparable performance to Gemma-2B despite being trained on fewer tokens.