Notice: The wikitext perplexity numbers are wrong in V1 of this paper. We did not set the correct LM Eval Harness settings. We will correct them swiftly.
\n","updatedAt":"2025-05-01T03:10:21.883Z","author":{"_id":"60cf8a354061635e43b28f60","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg","fullname":"Zayd Muhammad Kawakibi Zuhri","name":"zaydzuhri","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9302736520767212},"editors":["zaydzuhri"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg"],"reactions":[],"isReport":false}},{"id":"6813180938bee5488196c587","author":{"_id":"62d1ddfac58f969c1528f1b5","avatarUrl":"/avatars/75c372a831cde3c7c6dce3bc875488a7.svg","fullname":"Kalle Hilsenbek","name":"Bachstelze","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false},"createdAt":"2025-05-01T06:43:21.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Predominant attention-sinks occur with models of billions of parameters. Can softpick mitigate them at that scale?","html":"Predominant attention-sinks occur with models of billions of parameters. Can softpick mitigate them at that scale?
\n","updatedAt":"2025-05-01T06:43:21.896Z","author":{"_id":"62d1ddfac58f969c1528f1b5","avatarUrl":"/avatars/75c372a831cde3c7c6dce3bc875488a7.svg","fullname":"Kalle Hilsenbek","name":"Bachstelze","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.940544605255127},"editors":["Bachstelze"],"editorAvatarUrls":["/avatars/75c372a831cde3c7c6dce3bc875488a7.svg"],"reactions":[],"isReport":false},"replies":[{"id":"681327b57d3ec38a1ae824e3","author":{"_id":"60cf8a354061635e43b28f60","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg","fullname":"Zayd Muhammad Kawakibi Zuhri","name":"zaydzuhri","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false},"createdAt":"2025-05-01T07:50:13.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an early preprint. We will be scaling up to 2B and 7B model scales soon and update the paper. Stay tuned 👍","html":"This is an early preprint. We will be scaling up to 2B and 7B model scales soon and update the paper. Stay tuned 👍
\n","updatedAt":"2025-05-01T07:50:13.073Z","author":{"_id":"60cf8a354061635e43b28f60","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg","fullname":"Zayd Muhammad Kawakibi Zuhri","name":"zaydzuhri","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8848195672035217},"editors":["zaydzuhri"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg"],"reactions":[{"reaction":"🤗","users":["Bachstelze","Erland"],"count":2}],"isReport":false,"parentCommentId":"6813180938bee5488196c587"}}]},{"id":"681421468092f4c646f999c4","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2025-05-02T01:35:02.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs](https://huggingface.co/papers/2504.18415) (2025)\n* [Accurate INT8 Training Through Dynamic Block-Level Fallback](https://huggingface.co/papers/2503.08040) (2025)\n* [CASP: Compression of Large Multimodal Models Based on Attention Sparsity](https://huggingface.co/papers/2503.05936) (2025)\n* [RSQ: Learning from Important Tokens Leads to Better Quantized LLMs](https://huggingface.co/papers/2503.01820) (2025)\n* [Using Attention Sinks to Identify and Evaluate Dormant Heads in Pretrained LLMs](https://huggingface.co/papers/2504.03889) (2025)\n* [A Refined Analysis of Massive Activations in LLMs](https://huggingface.co/papers/2503.22329) (2025)\n* [Forgetting Transformer: Softmax Attention with a Forget Gate](https://huggingface.co/papers/2503.02130) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs (2025) \n
- Accurate INT8 Training Through Dynamic Block-Level Fallback (2025) \n
- CASP: Compression of Large Multimodal Models Based on Attention Sparsity (2025) \n
- RSQ: Learning from Important Tokens Leads to Better Quantized LLMs (2025) \n
- Using Attention Sinks to Identify and Evaluate Dormant Heads in Pretrained LLMs (2025) \n
- A Refined Analysis of Massive Activations in LLMs (2025) \n
- Forgetting Transformer: Softmax Attention with a Forget Gate (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
Abstract
Softpick is a drop-in replacement for softmax in transformer attention mechanisms that improves performance, reduces activation sinks, and achieves sparsity, particularly beneficial for quantization and low-precision training.
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our experiments with 340M parameter models demonstrate that softpick maintains performance parity with softmax on standard benchmarks while achieving 0% sink rate. The softpick transformer produces hidden states with significantly lower kurtosis (340 vs 33,510) and creates sparse attention maps (46.97% sparsity). Models using softpick consistently outperform softmax when quantized, with particularly pronounced advantages at lower bit precisions. Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability. Our code is available at https://github.com/zaydzuhri/softpick-attention.
Community
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our experiments with 340M parameter models demonstrate that softpick maintains performance parity with softmax on standard benchmarks while achieving 0% sink rate. The softpick transformer produces hidden states with significantly lower kurtosis (340 vs 33,510) and creates sparse attention maps (46.97% sparsity). Models using softpick consistently outperform softmax when quantized, with particularly pronounced advantages at lower bit precisions. Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability. Our code is available at https://github.com/zaydzuhri/softpick-attention.
Notice: The wikitext perplexity numbers are wrong in V1 of this paper. We did not set the correct LM Eval Harness settings. We will correct them swiftly.
Predominant attention-sinks occur with models of billions of parameters. Can softpick mitigate them at that scale?
This is an early preprint. We will be scaling up to 2B and 7B model scales soon and update the paper. Stay tuned 👍
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs (2025)
- Accurate INT8 Training Through Dynamic Block-Level Fallback (2025)
- CASP: Compression of Large Multimodal Models Based on Attention Sparsity (2025)
- RSQ: Learning from Important Tokens Leads to Better Quantized LLMs (2025)
- Using Attention Sinks to Identify and Evaluate Dormant Heads in Pretrained LLMs (2025)
- A Refined Analysis of Massive Activations in LLMs (2025)
- Forgetting Transformer: Softmax Attention with a Forget Gate (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 6
Browse 6 models citing this paperDatasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper