Librarian Bot. I found the following papers similar to this paper. \n
The following papers were recommended by the Semantic Scholar API
\n- \n
- Punica: Multi-Tenant LoRA Serving (2023) \n
- NOLA: Networks as Linear Combination of Low Rank Random Basis (2023) \n
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (2023) \n
- LoRA ensembles for large language model fine-tuning (2023) \n
- Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling (2023) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n","updatedAt":"2023-11-08T10:31:21.278Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7223953604698181},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[{"reaction":"👍","users":["stevaras2","xu1998hz"],"count":2}],"isReport":false}},{"id":"666557b9da1e6adefeca135b","author":{"_id":"6186ddf6a7717cb375090c01","avatarUrl":"/avatars/716b6a7d1094c8036b2a8a7b9063e8aa.svg","fullname":"Julien BLANCHON","name":"blanchon","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":176,"isUserFollowing":false},"createdAt":"2024-06-09T07:20:25.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"# S-LoRA: Efficiently Serving Thousands of LoRA-Adaptive Models!\n\nhttps://cdn-uploads.huggingface.co/production/uploads/6186ddf6a7717cb375090c01/g7pUN8Lle97b4AEw-4tR7.mp4 \n\n## Links 🔗:\n👉 Subscribe: https://www.youtube.com/@Arxflix\n👉 Twitter: https://x.com/arxflix\n👉 LMNT (Partner): https://lmnt.com/\n\n\nBy Arxflix\n","html":"\n\t\n\t\t\n\t\n\t\n\t\tS-LoRA: Efficiently Serving Thousands of LoRA-Adaptive Models!\n\t\n
\n \n\n\n\t\n\t\t\n\t\n\t\n\t\tLinks 🔗:\n\t\n
\n👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/
By Arxflix
Papers
arxiv:2311.03285
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
Published on Nov 6, 2023
Authors:
Abstract
S-LoRA is a system that allows for efficient and scalable serving of numerous LoRA adapters using a unified memory pool, tensor parallelism, and custom CUDA kernels.
AI-generated summary
The "pretrain-then-finetune" paradigm is commonly adopted in the deployment of large language models. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks, resulting in a substantial collection of LoRA adapters derived from one base model. We observe that this paradigm presents significant opportunities for batched inference during serving. To capitalize on these opportunities, we present S-LoRA, a system designed for the scalable serving of many LoRA adapters. S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead. Compared to state-of-the-art libraries such as HuggingFace PEFT and vLLM (with naive support of LoRA serving), S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude. As a result, S-LoRA enables scalable serving of many task-specific fine-tuned models and offers the potential for large-scale customized fine-tuning services.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Punica: Multi-Tenant LoRA Serving (2023)
- NOLA: Networks as Linear Combination of Low Rank Random Basis (2023)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (2023)
- LoRA ensembles for large language model fine-tuning (2023)
- Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
S-LoRA: Efficiently Serving Thousands of LoRA-Adaptive Models!
Links 🔗:
👉 Subscribe: https://www.youtube.com/@Arxflix
👉 Twitter: https://x.com/arxflix
👉 LMNT (Partner): https://lmnt.com/
By Arxflix
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2311.03285 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2311.03285 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2311.03285 in a Space README.md to link it from this page.