https://x.com/BlinkDL_AI/status/1898579674575552558\n","updatedAt":"2025-03-19T03:23:48.931Z","author":{"_id":"62b3d8d651b07307bd12b7f0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1655953609090-noauth.jpeg","fullname":"BlinkDL","name":"BlinkDL","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":936,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5948665142059326},"editors":["BlinkDL"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1655953609090-noauth.jpeg"],"reactions":[{"reaction":"🚀","users":["ZhangRC","shoumenchougou","BlinkDL","stereoplegic","saitejautpala","jadexlaw","luisgalan","hevok","dont-worry-1","howard-hou"],"count":10},{"reaction":"🔥","users":["howard-hou","ZhangRC"],"count":2}],"isReport":false,"parentCommentId":"67da317b11b6db69207a9db2"}},{"id":"67dbd0bd7d4649d04c6782be","author":{"_id":"651c240a37fecec1fe96c60b","avatarUrl":"/avatars/5af52af97b7907e138efecac0f20799b.svg","fullname":"S.F.","name":"search-facility","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2025-03-20T08:24:29.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"any chance for moving to BlueSky? BlueSky still lacks AI community for now, though...","html":"
any chance for moving to BlueSky? BlueSky still lacks AI community for now, though...
\n","updatedAt":"2025-03-20T08:24:29.331Z","author":{"_id":"651c240a37fecec1fe96c60b","avatarUrl":"/avatars/5af52af97b7907e138efecac0f20799b.svg","fullname":"S.F.","name":"search-facility","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9345461130142212},"editors":["search-facility"],"editorAvatarUrls":["/avatars/5af52af97b7907e138efecac0f20799b.svg"],"reactions":[],"isReport":false,"parentCommentId":"67da317b11b6db69207a9db2"}}]},{"id":"67da698a04c8e330ee189390","author":{"_id":"6797a449ece6b7910c0b385b","avatarUrl":"/avatars/d0e20ebca1d8aa31c3b8e745d0268a72.svg","fullname":"banma","name":"wuhanluojia","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2025-03-19T06:51:54.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"可并行训练+\n边推理边学习（推理速度显存都恒定）+\n无限上下文+\n无限CoT+\n想知道上述无限具体指的是什么，比如具体应用的时候无限上下文？无限cot？","html":"可并行训练+
边推理边学习（推理速度显存都恒定）+
无限上下文+
无限CoT+
想知道上述无限具体指的是什么，比如具体应用的时候无限上下文？无限cot？
\n\n可并行训练+
\n
边推理边学习（推理速度显存都恒定）+
无限上下文+
无限CoT+
想知道上述无限具体指的是什么，比如具体应用的时候无限上下文？无限cot？
This claim about \"infinite context\" and \"infinite CoT\" is slightly overclaimed. This isn't from the authors but probably from a product manager.
However, RWKV-7 as an RNN can accept \"indefinite\" context length. It can accept arbitrarily long input but still suffer from forgetting. This is different from \"infinite\", since RNNs may still forget certain information in a very lengthy context.
So, \"无限\" means indefinite.
For example, RWKV-7 models trained with context length 4k can retain perfect memory until around 8k-16k, after that, they may drastically forget previous information. If necessary, you can fine-tune RWKV-7 models with arbitrarily long context length. Please test specific use cases for clarity.
New paper just dropped 👌👌
\n","updatedAt":"2025-03-19T08:02:30.273Z","author":{"_id":"63cac1a50932c72f13995d6f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63cac1a50932c72f13995d6f/Bd9jEsCL9yWXdyAQCx61J.jpeg","fullname":"Nathan Wilce","name":"m8than","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":18,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9254369139671326},"editors":["m8than"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63cac1a50932c72f13995d6f/Bd9jEsCL9yWXdyAQCx61J.jpeg"],"reactions":[{"reaction":"🚀","users":["ZhangRC","hevok","search-facility"],"count":3}],"isReport":false}},{"id":"67da8bbeda32970c672a736e","author":{"_id":"6340fcb095c20b94473d1b6b","avatarUrl":"/avatars/5bb04f2fb6b2aaab9f9b5351a80794b6.svg","fullname":"Jellyfish042","name":"Jellyfish042","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9,"isUserFollowing":false},"createdAt":"2025-03-19T09:17:50.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Transformers are toast this time for real","html":"Transformers are toast this time for real
\n","updatedAt":"2025-03-19T09:17:50.975Z","author":{"_id":"6340fcb095c20b94473d1b6b","avatarUrl":"/avatars/5bb04f2fb6b2aaab9f9b5351a80794b6.svg","fullname":"Jellyfish042","name":"Jellyfish042","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.880553126335144},"editors":["Jellyfish042"],"editorAvatarUrls":["/avatars/5bb04f2fb6b2aaab9f9b5351a80794b6.svg"],"reactions":[{"reaction":"😎","users":["ZhangRC","saitejautpala","hevok"],"count":3}],"isReport":false}},{"id":"67dad52f7589cb037ad469ec","author":{"_id":"6418629fd13ffa408128d7ae","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1679319546731-noauth.png","fullname":"Zhang Ruichong","name":"ZhangRC","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":27,"isUserFollowing":false},"createdAt":"2025-03-19T14:31:11.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"The RWKV community features diverse backgrounds and expertise. This time, most of our authors come from fields outside traditional AI, including physics, mathematics, game development, material science, biochemistry and cybernetics. \nJoin our community at https://discord.com/invite/bDSBUMeFpc and have a nice journey!","html":"The RWKV community features diverse backgrounds and expertise. This time, most of our authors come from fields outside traditional AI, including physics, mathematics, game development, material science, biochemistry and cybernetics.
Join our community at https://discord.com/invite/bDSBUMeFpc and have a nice journey!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling (2025) \n
- Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models (2025) \n
- Adapting Large Language Models for Time Series Modeling via a Novel Parameter-efficient Adaptation Method (2025) \n
- UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs (2025) \n
- EmbBERT-Q: Breaking Memory Barriers in Embedded NLP (2025) \n
- Scaling Inference-Efficient Language Models (2025) \n
- Reasoning with Reinforced Functional Token Tuning (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
The RWKV-7 delay has been too long. It was open-sourced in December last year, and after three months, the paper is finally complete. It hasn't been easy.
\n","updatedAt":"2025-03-24T23:20:03.274Z","author":{"_id":"64414d0a66a62c605c94d14d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg","fullname":"howard-hou","name":"howard-hou","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":17,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9965810179710388},"editors":["howard-hou"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/64414d0a66a62c605c94d14d/KGjlf9tRwslr7Q503YLnJ.jpeg"],"reactions":[{"reaction":"👍","users":["search-facility","ZhangRC","SkyMind","hegderavin","bndp"],"count":5},{"reaction":"❤️","users":["ZhangRC","SkyMind"],"count":2}],"isReport":false}},{"id":"694aef8961707ccb9365ec5f","author":{"_id":"65243980050781c16f234f1f","avatarUrl":"/avatars/743a009681d5d554c27e04300db9f267.svg","fullname":"Avi","name":"avahal","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2025-12-23T19:37:45.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"arXiv lens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/rwkv-7-goose-with-expressive-dynamic-state-evolution-1949-e940c720\n- Executive Summary\n- Detailed Breakdown\n- Practical Applications","html":"arXiv lens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/rwkv-7-goose-with-expressive-dynamic-state-evolution-1949-e940c720
\n- \n
- Executive Summary \n
- Detailed Breakdown \n
- Practical Applications \n
Papers
arxiv:2503.14456
RWKV-7 "Goose" with Expressive Dynamic State Evolution
Published on Mar 18, 2025
Abstract
RWKV-7 "Goose" achieves state-of-the-art performance in multilingual tasks with optimal memory and inference efficiency, exceeding Transformer capabilities in complexity.
AI-generated summary
We present RWKV-7 "Goose", a new sequence modeling architecture, along with pre-trained language models that establish a new state-of-the-art in downstream performance at the 3 billion parameter scale on multilingual tasks, and match current SoTA English language performance despite being trained on dramatically fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only constant memory usage and constant inference time per token. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to TC^0. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at https://huggingface.co/RWKV, and our training and inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0 License.
Community
RWKV-7 paper is finally out!
cant wait to see this paired with CoT-type reasoning
可并行训练+
边推理边学习（推理速度显存都恒定）+
无限上下文+
无限CoT+
想知道上述无限具体指的是什么，比如具体应用的时候无限上下文？无限cot？
可并行训练+
边推理边学习（推理速度显存都恒定）+
无限上下文+
无限CoT+
想知道上述无限具体指的是什么，比如具体应用的时候无限上下文？无限cot？
This claim about "infinite context" and "infinite CoT" is slightly overclaimed. This isn't from the authors but probably from a product manager.
However, RWKV-7 as an RNN can accept "indefinite" context length. It can accept arbitrarily long input but still suffer from forgetting. This is different from "infinite", since RNNs may still forget certain information in a very lengthy context.
So, "无限" means indefinite.
For example, RWKV-7 models trained with context length 4k can retain perfect memory until around 8k-16k, after that, they may drastically forget previous information. If necessary, you can fine-tune RWKV-7 models with arbitrarily long context length. Please test specific use cases for clarity.
New paper just dropped 👌👌
Transformers are toast this time for real
The RWKV community features diverse backgrounds and expertise. This time, most of our authors come from fields outside traditional AI, including physics, mathematics, game development, material science, biochemistry and cybernetics.
Join our community at https://discord.com/invite/bDSBUMeFpc and have a nice journey!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling (2025)
- Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models (2025)
- Adapting Large Language Models for Time Series Modeling via a Novel Parameter-efficient Adaptation Method (2025)
- UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs (2025)
- EmbBERT-Q: Breaking Memory Barriers in Embedded NLP (2025)
- Scaling Inference-Efficient Language Models (2025)
- Reasoning with Reinforced Functional Token Tuning (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
The RWKV-7 delay has been too long. It was open-sourced in December last year, and after three months, the paper is finally complete. It hasn't been easy.
arXiv lens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/rwkv-7-goose-with-expressive-dynamic-state-evolution-1949-e940c720
- Executive Summary
- Detailed Breakdown
- Practical Applications