https://huggingface.co/papers/2507.22448), this work extends the idea to learnable matrix-, row-, and column-wise scaling. We show that the weight-norm equilibrium induced by weight decay and gradient noise is suboptimal, and that freeing these scale constraints yields consistent gains, generalizes μP, and improves downstream performance with both Adam and Muon optimizers.\n","updatedAt":"2026-01-09T05:25:50.946Z","author":{"_id":"6460c3811db65f878513bcaf","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6460c3811db65f878513bcaf/CRdJ8lXixDku3k8Rm5Stn.jpeg","fullname":"Jingwei Zuo","name":"JingweiZuo","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":39,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8441076874732971},"editors":["JingweiZuo"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6460c3811db65f878513bcaf/CRdJ8lXixDku3k8Rm5Stn.jpeg"],"reactions":[],"isReport":false}},{"id":"6961ad8800600e42ae401063","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2026-01-10T01:38:16.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Controlling changes to attention logits](https://huggingface.co/papers/2511.21377) (2025)\n* [Understanding the Mechanisms of Fast Hyperparameter Transfer](https://huggingface.co/papers/2512.22768) (2025)\n* [The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss](https://huggingface.co/papers/2512.08374) (2025)\n* [Data-Free Pruning of Self-Attention Layers in LLMs](https://huggingface.co/papers/2512.20636) (2025)\n* [Scaling Behavior of Discrete Diffusion Language Models](https://huggingface.co/papers/2512.10858) (2025)\n* [Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings](https://huggingface.co/papers/2512.12167) (2025)\n* [Correction of Decoupled Weight Decay](https://huggingface.co/papers/2512.08217) (2025)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Controlling changes to attention logits (2025) \n
- Understanding the Mechanisms of Fast Hyperparameter Transfer (2025) \n
- The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss (2025) \n
- Data-Free Pruning of Self-Attention Layers in LLMs (2025) \n
- Scaling Behavior of Discrete Diffusion Language Models (2025) \n
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings (2025) \n
- Correction of Decoupled Weight Decay (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
arXivlens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/learnable-multipliers-freeing-the-scale-of-language-model-matrix-layers-4763-7a709001
\n- \n
- Executive Summary \n
- Detailed Breakdown \n
- Practical Applications \n
Papers
arxiv:2601.04890
Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers
Published on Jan 8
#3 Paper of the day
Abstract
Learnable multipliers are introduced to address weight decay-induced normalization artifacts in large language model training, outperforming traditional methods while reducing computational overhead.
AI-generated summary
Applying weight decay (WD) to matrix layers is standard practice in large-language-model pretraining. Prior work suggests that stochastic gradient noise induces a Brownian-like expansion of the weight matrices W, whose growth is counteracted by WD, leading to a WD-noise equilibrium with a certain weight norm ||W||. In this work, we view the equilibrium norm as a harmful artifact of the training procedure, and address it by introducing learnable multipliers to learn the optimal scale. First, we attach a learnable scalar multiplier to W and confirm that the WD-noise equilibrium norm is suboptimal: the learned scale adapts to data and improves performance. We then argue that individual row and column norms are similarly constrained, and free their scale by introducing learnable per-row and per-column multipliers. Our method can be viewed as a learnable, more expressive generalization of muP multipliers. It outperforms a well-tuned muP baseline, reduces the computational overhead of multiplier tuning, and surfaces practical questions such as forward-pass symmetries and the width-scaling of the learned multipliers. Finally, we validate learnable multipliers with both Adam and Muon optimizers, where it shows improvement in downstream evaluations matching the improvement of the switching from Adam to Muon.
Community
Building on the μP multipliers applied in Falcon-H1 pretraining (https://huggingface.co/papers/2507.22448), this work extends the idea to learnable matrix-, row-, and column-wise scaling. We show that the weight-norm equilibrium induced by weight decay and gradient noise is suboptimal, and that freeing these scale constraints yields consistent gains, generalizes μP, and improves downstream performance with both Adam and Muon optimizers.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Controlling changes to attention logits (2025)
- Understanding the Mechanisms of Fast Hyperparameter Transfer (2025)
- The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss (2025)
- Data-Free Pruning of Self-Attention Layers in LLMs (2025)
- Scaling Behavior of Discrete Diffusion Language Models (2025)
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings (2025)
- Correction of Decoupled Weight Decay (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
arXivlens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/learnable-multipliers-freeing-the-scale-of-language-model-matrix-layers-4763-7a709001
- Executive Summary
- Detailed Breakdown
- Practical Applications
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2601.04890 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2601.04890 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2601.04890 in a Space README.md to link it from this page.