https://github.com/LCM-Lab/Elastic-Attention
Model Collection: https://modelscope.cn/collections/LCM_group/Elastic-Attention

\n","updatedAt":"2026-01-27T03:21:37.574Z","author":{"_id":"64096ef79e9f790c905b846d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg","fullname":"Zecheng Tang","name":"ZetangForward","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8259636759757996},"editors":["ZetangForward"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg"],"reactions":[],"isReport":false}},{"id":"69796885eba377c312b7a7bc","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2026-01-28T01:38:13.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [A Unified Sparse Attention via Multi-Granularity Compression](https://huggingface.co/papers/2512.14082) (2025)\n* [BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding](https://huggingface.co/papers/2512.12087) (2025)\n* [Accelerate Speculative Decoding with Sparse Computation in Verification](https://huggingface.co/papers/2512.21911) (2025)\n* [Training-free Context-adaptive Attention for Efficient Long Context Modeling](https://huggingface.co/papers/2512.09238) (2025)\n* [Block Sparse Flash Attention](https://huggingface.co/papers/2512.07011) (2025)\n* [VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse](https://huggingface.co/papers/2512.14531) (2025)\n* [PSA: Pyramid Sparse Attention for Efficient Video Understanding and Generation](https://huggingface.co/papers/2512.04025) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2026-01-28T01:38:13.329Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7002789974212646},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}},{"id":"697979ee3a43278be87360a8","author":{"_id":"65243980050781c16f234f1f","avatarUrl":"/avatars/743a009681d5d554c27e04300db9f267.svg","fullname":"Avi","name":"avahal","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false},"createdAt":"2026-01-28T02:52:30.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"arXivlens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/elastic-attention-test-time-adaptive-sparsity-ratios-for-efficient-transformers-8412-02d813dc\n\n- Executive Summary\n- Detailed Breakdown\n- Practical Applications","html":"

arXivlens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/elastic-attention-test-time-adaptive-sparsity-ratios-for-efficient-transformers-8412-02d813dc

Executive Summary
Detailed Breakdown
Practical Applications

\n","updatedAt":"2026-01-28T02:52:30.556Z","author":{"_id":"65243980050781c16f234f1f","avatarUrl":"/avatars/743a009681d5d554c27e04300db9f267.svg","fullname":"Avi","name":"avahal","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.637006938457489},"editors":["avahal"],"editorAvatarUrls":["/avatars/743a009681d5d554c27e04300db9f267.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2601.17367","authors":[{"_id":"6978295b026bdf0473116db5","user":{"_id":"64096ef79e9f790c905b846d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg","isPro":false,"fullname":"Zecheng Tang","user":"ZetangForward","type":"user"},"name":"Zecheng Tang","status":"claimed_verified","statusLastChangedAt":"2026-01-27T09:42:48.253Z","hidden":false},{"_id":"6978295b026bdf0473116db6","user":{"_id":"6732fb1d24b316be87acaafe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6732fb1d24b316be87acaafe/BzD8HL4vhh3mfeSF3rm_1.jpeg","isPro":false,"fullname":"Quantong Qiu","user":"QQTang1223","type":"user"},"name":"Quantong Qiu","status":"admin_assigned","statusLastChangedAt":"2026-01-27T14:27:08.590Z","hidden":false},{"_id":"6978295b026bdf0473116db7","name":"Yi Yang","hidden":false},{"_id":"6978295b026bdf0473116db8","name":"Zhiyi Hong","hidden":false},{"_id":"6978295b026bdf0473116db9","name":"Haiya Xiang","hidden":false},{"_id":"6978295b026bdf0473116dba","user":{"_id":"67e7e66d0d9434ad14575739","avatarUrl":"/avatars/5cf522a690519c22b65b959c9e0ca8d2.svg","isPro":false,"fullname":"Kebin Liu","user":"wanderhzz","type":"user"},"name":"Kebin Liu","status":"claimed_verified","statusLastChangedAt":"2026-01-28T11:32:06.877Z","hidden":false},{"_id":"6978295b026bdf0473116dbb","user":{"_id":"68cce9276e4618473d590342","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/UqBWbWZ58EPwJ5iRUsJsB.png","isPro":false,"fullname":"Qingqing Dang","user":"DaisyGrace","type":"user"},"name":"Qingqing Dang","status":"admin_assigned","statusLastChangedAt":"2026-01-27T14:27:50.734Z","hidden":false},{"_id":"6978295b026bdf0473116dbc","user":{"_id":"6670e285b0c03c4e9d6e0985","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6670e285b0c03c4e9d6e0985/j9Zr-lOtrRmpFz5f4x420.jpeg","isPro":false,"fullname":"Juntao Li","user":"douvleplus","type":"user"},"name":"Juntao Li","status":"admin_assigned","statusLastChangedAt":"2026-01-27T14:27:56.908Z","hidden":false},{"_id":"6978295b026bdf0473116dbd","name":"Min Zhang","hidden":false}],"publishedAt":"2026-01-24T08:22:07.000Z","submittedOnDailyAt":"2026-01-27T00:51:37.565Z","title":"Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers","submittedOnDailyBy":{"_id":"64096ef79e9f790c905b846d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg","isPro":false,"fullname":"Zecheng Tang","user":"ZetangForward","type":"user"},"summary":"The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method.","upvotes":33,"discussionId":"6978295b026bdf0473116dbe","projectPage":"https://github.com/LCM-Lab/Elastic-Attention","githubRepo":"https://github.com/LCM-Lab/Elastic-Attention","githubRepoAddedBy":"user","ai_summary":"Elastic Attention enables dynamic adjustment of attention sparsity during inference by integrating a lightweight Attention Router into pretrained models, achieving efficient long-context processing.","ai_keywords":["standard attention mechanisms","sparse attention","full attention","hybrid attention strategies","attention router","attention heads","long-context scenarios","pretrained models","computational efficiency"],"githubStars":16,"organization":{"_id":"61f8e653129c9ff1b911293d","name":"SUDA","fullname":"Soochow University","avatar":"https://cdn-uploads.huggingface.co/production/uploads/1643701821817-61f8e5934a8e5a275b2b3e5a.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64096ef79e9f790c905b846d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg","isPro":false,"fullname":"Zecheng Tang","user":"ZetangForward","type":"user"},{"_id":"69704f38cc45f8d5814a330e","avatarUrl":"/avatars/b07c5624ab47040d42d16e7a91420f70.svg","isPro":false,"fullname":"liang","user":"xiaoboooo","type":"user"},{"_id":"6732fb1d24b316be87acaafe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6732fb1d24b316be87acaafe/BzD8HL4vhh3mfeSF3rm_1.jpeg","isPro":false,"fullname":"Quantong Qiu","user":"QQTang1223","type":"user"},{"_id":"643101a8440a10209804cb24","avatarUrl":"/avatars/ec3c8612e3236689acec836b87f2a0be.svg","isPro":false,"fullname":"keyan zhou","user":"Jonaszky123","type":"user"},{"_id":"616fb788e2ad27af26561b1a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1675485317568-616fb788e2ad27af26561b1a.jpeg","isPro":false,"fullname":"Xiao Xu","user":"LooperXX","type":"user"},{"_id":"64e5e3b0b687eece0edd5d6b","avatarUrl":"/avatars/48129ccecee0211fc1f78bb48740aa6c.svg","isPro":false,"fullname":"youwangjie","user":"Moriarty0923","type":"user"},{"_id":"67e7e66d0d9434ad14575739","avatarUrl":"/avatars/5cf522a690519c22b65b959c9e0ca8d2.svg","isPro":false,"fullname":"Kebin Liu","user":"wanderhzz","type":"user"},{"_id":"6978311bdd0add0478cdf579","avatarUrl":"/avatars/4024938613bfd72f08e9340776152819.svg","isPro":false,"fullname":"qidong wang","user":"qidong0713","type":"user"},{"_id":"6702c62f3598d954c090a79f","avatarUrl":"/avatars/6c659d94ed0ffb5ae87af6fa15b73664.svg","isPro":false,"fullname":"ZikangZhang","user":"zzk2","type":"user"},{"_id":"697832a1e8e3701db89f0ff0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/697832a1e8e3701db89f0ff0/k1zpl_00HZs8X0j__HGD_.png","isPro":false,"fullname":"Dai Chenchen","user":"Cassie033","type":"user"},{"_id":"68144b68a706ff01feda036c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/68144b68a706ff01feda036c/vlQHsP2dG3ha_nrP9_WNr.jpeg","isPro":false,"fullname":"ymrl","user":"ymrl","type":"user"},{"_id":"697836676c76bc0b0d9cdf7f","avatarUrl":"/avatars/774ca1ab62d03b7f2f711e79894a939c.svg","isPro":false,"fullname":"guanzhuang","user":"guanzhuang","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0,"organization":{"_id":"61f8e653129c9ff1b911293d","name":"SUDA","fullname":"Soochow University","avatar":"https://cdn-uploads.huggingface.co/production/uploads/1643701821817-61f8e5934a8e5a275b2b3e5a.png"}}">

Papers

arxiv:2601.17367

Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers

Published on Jan 24

· Submitted by

Zecheng Tang on Jan 27

Soochow University

Upvote

Authors:

Zecheng Tang ,

Quantong Qiu ,

Kebin Liu ,

Qingqing Dang ,

Juntao Li ,

Abstract

Elastic Attention enables dynamic adjustment of attention sparsity during inference by integrating a lightweight Attention Router into pretrained models, achieving efficient long-context processing.

AI-generated summary

The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method.

View arXiv page View PDF Project page GitHub 16 Add to collection

Community

ZetangForward

Paper author Paper submitter 25 days ago

Elastic Attention enables models to achieve both strong performance and efficient inference by dynamically allocating computation modes (Full Attention or Sparse Attention) to each attention head through our designed Attention Router, adapting sparsity ratios based on input characteristics.

Code & Training data: https://github.com/LCM-Lab/Elastic-Attention
Model Collection: https://modelscope.cn/collections/LCM_group/Elastic-Attention