@librarian-bot\n\t recommend

\n","updatedAt":"2024-09-24T18:05:41.106Z","author":{"_id":"646b8e6f31968a60a0201a12","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/646b8e6f31968a60a0201a12/SU2Gs1NPuk1zoXHwFHl0U.jpeg","fullname":")))?!?(((","name":"stereoplegic","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3927,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7918877601623535},"editors":["stereoplegic"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/646b8e6f31968a60a0201a12/SU2Gs1NPuk1zoXHwFHl0U.jpeg"],"reactions":[],"isReport":false},"replies":[{"id":"66f2ff7e455a33c11e2828a2","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false},"createdAt":"2024-09-24T18:05:50.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding](https://huggingface.co/papers/2408.11049) (2024)\n* [Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding](https://huggingface.co/papers/2408.00264) (2024)\n* [Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling](https://huggingface.co/papers/2408.08696) (2024)\n* [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://huggingface.co/papers/2408.12757) (2024)\n* [InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference](https://huggingface.co/papers/2409.04992) (2024)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-09-24T18:05:50.640Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7582108378410339},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"66f2ff75cbe35a1ea95a5773"}}]}],"primaryEmailConfirmed":false,"paper":{"id":"2407.11798","authors":[{"_id":"66a0520f7dd442476beeea8d","name":"Branden Butler","hidden":false},{"_id":"66a0520f7dd442476beeea8e","name":"Sixing Yu","hidden":false},{"_id":"66a0520f7dd442476beeea8f","name":"Arya Mazaheri","hidden":false},{"_id":"66a0520f7dd442476beeea90","name":"Ali Jannesari","hidden":false}],"publishedAt":"2024-07-16T14:52:02.000Z","title":"PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined\n Speculation","summary":"Inference of Large Language Models (LLMs) across computer clusters has become\na focal point of research in recent times, with many acceleration techniques\ntaking inspiration from CPU speculative execution. These techniques reduce\nbottlenecks associated with memory bandwidth, but also increase end-to-end\nlatency per inference run, requiring high speculation acceptance rates to\nimprove performance. Combined with a variable rate of acceptance across tasks,\nspeculative inference techniques can result in reduced performance.\nAdditionally, pipeline-parallel designs require many user requests to maintain\nmaximum utilization. As a remedy, we propose PipeInfer, a pipelined speculative\nacceleration technique to reduce inter-token latency and improve system\nutilization for single-request scenarios while also improving tolerance to low\nspeculation acceptance rates and low-bandwidth interconnects. PipeInfer\nexhibits up to a 2.15times improvement in generation speed over standard\nspeculative inference. PipeInfer achieves its improvement through Continuous\nAsynchronous Speculation and Early Inference Cancellation, the former improving\nlatency and generation speed by running single-token inference simultaneously\nwith several speculative runs, while the latter improves speed and latency by\nskipping the computation of invalidated runs, even in the middle of inference.","upvotes":1,"discussionId":"66a052107dd442476beeeadf","ai_summary":"PipeInfer improves inference speed and latency for large language models using continuous asynchronous speculation and early cancellation in a pipelined design.","ai_keywords":["Large Language Models","LLMs","speculative execution","pipeline-parallel designs","Continuous Asynchronous Speculation","Early Inference Cancellation"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63869d1e81fe8c678a3a9422","avatarUrl":"/avatars/3bb8728057fa2ba0e24f5ceb1600068d.svg","isPro":true,"fullname":"Zach Mustafa","user":"Zmu","type":"user"}],"acceptLanguages":["*"]}">

Papers

arxiv:2407.11798

PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation

Published on Jul 16, 2024

Upvote

Authors:

Abstract

PipeInfer improves inference speed and latency for large language models using continuous asynchronous speculation and early cancellation in a pipelined design.

AI-generated summary

Inference of Large Language Models (LLMs) across computer clusters has become a focal point of research in recent times, with many acceleration techniques taking inspiration from CPU speculative execution. These techniques reduce bottlenecks associated with memory bandwidth, but also increase end-to-end latency per inference run, requiring high speculation acceptance rates to improve performance. Combined with a variable rate of acceptance across tasks, speculative inference techniques can result in reduced performance. Additionally, pipeline-parallel designs require many user requests to maintain maximum utilization. As a remedy, we propose PipeInfer, a pipelined speculative acceleration technique to reduce inter-token latency and improve system utilization for single-request scenarios while also improving tolerance to low speculation acceptance rates and low-bandwidth interconnects. PipeInfer exhibits up to a 2.15times improvement in generation speed over standard speculative inference. PipeInfer achieves its improvement through Continuous Asynchronous Speculation and Early Inference Cancellation, the former improving latency and generation speed by running single-token inference simultaneously with several speculative runs, while the latter improves speed and latency by skipping the computation of invalidated runs, even in the middle of inference.

View arXiv page View PDF Add to collection