https://arxivlens.com/PaperView/Details/scaling-embeddings-outperforms-scaling-experts-in-language-models-8780-2f224bf1\n
- \n
- Executive Summary \n
- Detailed Breakdown \n
- Practical Applications \n
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (2026) \n
- Sigma-MoE-Tiny Technical Report (2025) \n
- VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse (2025) \n
- STEM: Scaling Transformers with Embedding Modules (2026) \n
- LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (2026) \n
- MIDUS: Memory-Infused Depth Up-Scaling (2025) \n
- A.X K1 Technical Report (2026) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Papers
arxiv:2601.21204
Scaling Embeddings Outperforms Scaling Experts in Language Models
Abstract
Embedding scaling offers superior sparsity scaling compared to expert scaling in large language models, enabling efficient inference through system optimizations and speculative decoding.
AI-generated summary
While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.
Community
Embedding scaling can outperform mixture of experts for sparse language models, aided by system optimizations and speculative decoding, with LongCat-Flash-Lite achieving strong competitiveness.
arXivLens breakdown of this paper 👉 https://arxivlens.com/PaperView/Details/scaling-embeddings-outperforms-scaling-experts-in-language-models-8780-2f224bf1
- Executive Summary
- Detailed Breakdown
- Practical Applications
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (2026)
- Sigma-MoE-Tiny Technical Report (2025)
- VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse (2025)
- STEM: Scaling Transformers with Embedding Modules (2026)
- LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts (2026)
- MIDUS: Memory-Infused Depth Up-Scaling (2025)
- A.X K1 Technical Report (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 2
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2601.21204 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2601.21204 in a Space README.md to link it from this page.