The following papers were recommended by the Semantic Scholar API
\n- \n
- Channel Merging: Preserving Specialization for Merged Experts (2024) \n
- MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025) \n
- FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models (2025) \n
- FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing (2024) \n
- Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (2025) \n
- ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning (2025) \n
- Training Language Models to Reason Efficiently (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
Abstract
A method for merging Large Language Models leverages layer-wise averaging and task-level expert routing to improve performance and reduce system cost.
Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one. However, parameter conflicts between models leads to performance degradation in averaging. While model routing addresses this issue by selecting individual models during inference, it imposes excessive storage and compute costs, and fails to leverage the common knowledge from different models. In this work, we observe that different layers exhibit varying levels of parameter conflicts. Building on this insight, we average layers with minimal parameter conflicts and use a novel task-level expert routing for layers with significant conflicts. To further reduce storage costs, inspired by task arithmetic sparsity, we decouple multiple fine-tuned experts into a dense expert and several sparse experts. Considering the out-of-distribution samples, we select and merge appropriate experts based on the task uncertainty of the input data. We conduct extensive experiments on both LLaMA and Qwen with varying parameter scales, and evaluate on real-world reasoning tasks. Results demonstrate that our method consistently achieves significant performance improvements while requiring less system cost compared to existing methods.
Community
Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one. However, parameter conflicts between models leads to performance degradation in averaging. While model routing addresses this issue by selecting individual models during inference, it imposes excessive storage and compute costs, and fails to leverage the common knowledge from different models. In this work, we observe that different layers exhibit varying levels of parameter conflicts. Building on this insight, we average layers with minimal parameter conflicts and use a novel task-level expert routing for layers with significant conflicts. To further reduce storage costs, inspired by task arithmetic sparsity, we decouple multiple fine-tuned experts into a dense expert and several sparse experts. Considering the out-of-distribution samples, we select and merge appropriate experts based on the task uncertainty of the input data. We conduct extensive experiments on both LLaMA and Qwen with varying parameter scales, and evaluate on real-world reasoning tasks. Results demonstrate that our method consistently achieves significant performance improvements while requiring less system cost compared to existing methods.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Channel Merging: Preserving Specialization for Merged Experts (2024)
- MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs (2025)
- FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models (2025)
- FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing (2024)
- Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging (2025)
- ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning (2025)
- Training Language Models to Reason Efficiently (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper