Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend

\n","updatedAt":"2024-08-27T01:33:11.119Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":318,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7344258427619934},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2408.13233","authors":[{"_id":"66cc05aed20991c2a015a6bc","name":"Yingyu Liang","hidden":false},{"_id":"66cc05aed20991c2a015a6bd","user":{"_id":"6335604ea01bd734f72316b0","avatarUrl":"/avatars/4c6611dabd492106ffb2e82fd680d983.svg","isPro":false,"fullname":"Zhizhou Sha","user":"JamesSand","type":"user"},"name":"Zhizhou Sha","status":"claimed_verified","statusLastChangedAt":"2024-08-27T07:42:34.868Z","hidden":false},{"_id":"66cc05aed20991c2a015a6be","user":{"_id":"64b769d7fa7eabaae5fb7f2f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64b769d7fa7eabaae5fb7f2f/AbxVphanApZDfj3cOI1tb.jpeg","isPro":false,"fullname":"Zhenmei Shi","user":"Zhenmei","type":"user"},"name":"Zhenmei Shi","status":"claimed_verified","statusLastChangedAt":"2024-09-30T08:07:40.297Z","hidden":false},{"_id":"66cc05aed20991c2a015a6bf","name":"Zhao Song","hidden":false},{"_id":"66cc05aed20991c2a015a6c0","user":{"_id":"658ab894c4b2004663dff3ae","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658ab894c4b2004663dff3ae/oPRnFuW2Imaa2KkYWNbSf.jpeg","isPro":false,"fullname":"YUFA ZHOU","user":"MasterZhou","type":"user"},"name":"Yufa Zhou","status":"claimed_verified","statusLastChangedAt":"2024-08-27T07:42:27.200Z","hidden":false}],"publishedAt":"2024-08-23T17:16:43.000Z","submittedOnDailyAt":"2024-08-26T03:13:28.024Z","title":"Multi-Layer Transformers Gradient Can be Approximated in Almost Linear\n Time","submittedOnDailyBy":{"_id":"6335604ea01bd734f72316b0","avatarUrl":"/avatars/4c6611dabd492106ffb2e82fd680d983.svg","isPro":false,"fullname":"Zhizhou Sha","user":"JamesSand","type":"user"},"summary":"The quadratic computational complexity in the self-attention mechanism of\npopular transformer architectures poses significant challenges for training and\ninference, particularly in terms of efficiency and memory requirements. Towards\naddressing these challenges, this paper introduces a novel fast computation\nmethod for gradient calculation in multi-layer transformer models. Our approach\nenables the computation of gradients for the entire multi-layer transformer\nmodel in almost linear time n^{1+o(1)}, where n is the input sequence\nlength. This breakthrough significantly reduces the computational bottleneck\nassociated with the traditional quadratic time complexity. Our theory holds for\nany loss function and maintains a bounded approximation error across the entire\nmodel. Furthermore, our analysis can hold when the multi-layer transformer\nmodel contains many practical sub-modules, such as residual connection, casual\nmask, and multi-head attention. By improving the efficiency of gradient\ncomputation in large language models, we hope that our work will facilitate the\nmore effective training and deployment of long-context language models based on\nour theoretical results.","upvotes":23,"discussionId":"66cc05aed20991c2a015a6ed","ai_summary":"A novel fast computation method for gradient calculation in multi-layer transformer models reduces computational complexity from quadratic to almost linear, improving efficiency and enabling better handling of long-context language models.","ai_keywords":["self-attention mechanism","transformer architectures","gradient calculation","multi-layer transformer models","residual connection","casual mask","multi-head attention"]},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6335604ea01bd734f72316b0","avatarUrl":"/avatars/4c6611dabd492106ffb2e82fd680d983.svg","isPro":false,"fullname":"Zhizhou Sha","user":"JamesSand","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"},{"_id":"625026749d39e8be3166132f","avatarUrl":"/avatars/f32291df2054c1bb4a01889d1b41c0d5.svg","isPro":false,"fullname":"Christopher Schröder","user":"cschroeder","type":"user"},{"_id":"644e1b1d9b4e87c31bab0a14","avatarUrl":"/avatars/88bb4c4a67dc8958069e9014f5e73a0b.svg","isPro":false,"fullname":"Michael Barry","user":"MichaelBarryUK","type":"user"},{"_id":"64ace20409d1b58891e55108","avatarUrl":"/avatars/deee6b3a42898b525d5efb5851d02d64.svg","isPro":false,"fullname":"Ethan Tracy","user":"EthanTracy","type":"user"},{"_id":"648eb1eb59c4e5c87dc116e0","avatarUrl":"/avatars/c636cea39c2c0937f01398c94ead5dad.svg","isPro":false,"fullname":"fdsqefsgergd","user":"T-representer","type":"user"},{"_id":"6270324ebecab9e2dcf245de","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6270324ebecab9e2dcf245de/cMbtWSasyNlYc9hvsEEzt.jpeg","isPro":false,"fullname":"Kye Gomez","user":"kye","type":"user"},{"_id":"62deb6c3520a9fae78bb9bc3","avatarUrl":"/avatars/5d75fffa9bad36d20adb8f47141d1f0b.svg","isPro":false,"fullname":"Literate Goggles","user":"literate-goggles","type":"user"},{"_id":"658ab894c4b2004663dff3ae","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/658ab894c4b2004663dff3ae/oPRnFuW2Imaa2KkYWNbSf.jpeg","isPro":false,"fullname":"YUFA ZHOU","user":"MasterZhou","type":"user"},{"_id":"65ba471ad88a65abb9328ee2","avatarUrl":"/avatars/956238ce5034091e64d026b0272c4400.svg","isPro":false,"fullname":"Dazhi Jiang","user":"thuzhizhi","type":"user"},{"_id":"654ad0fa7c2fd9829f597fa9","avatarUrl":"/avatars/54eb07d25f122299a9eb18e72bf08651.svg","isPro":false,"fullname":"Arjuna","user":"arjunaaqa1","type":"user"},{"_id":"654ed9048d8c3bf4ddb1fa90","avatarUrl":"/avatars/688348e816e74257bd5c65fe431c4589.svg","isPro":false,"fullname":"Ben Pope","user":"realbenpope","type":"user"}],"acceptLanguages":["*"],"dailyPaperRank":0}">

Papers

arxiv:2408.13233

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time

Published on Aug 23, 2024

· Submitted by

Zhizhou Sha on Aug 26, 2024

Upvote

Authors:

Zhizhou Sha ,

Zhenmei Shi ,

Yufa Zhou

Abstract

A novel fast computation method for gradient calculation in multi-layer transformer models reduces computational complexity from quadratic to almost linear, improving efficiency and enabling better handling of long-context language models.

AI-generated summary

The quadratic computational complexity in the self-attention mechanism of popular transformer architectures poses significant challenges for training and inference, particularly in terms of efficiency and memory requirements. Towards addressing these challenges, this paper introduces a novel fast computation method for gradient calculation in multi-layer transformer models. Our approach enables the computation of gradients for the entire multi-layer transformer model in almost linear time n^{1+o(1)}, where n is the input sequence length. This breakthrough significantly reduces the computational bottleneck associated with the traditional quadratic time complexity. Our theory holds for any loss function and maintains a bounded approximation error across the entire model. Furthermore, our analysis can hold when the multi-layer transformer model contains many practical sub-modules, such as residual connection, casual mask, and multi-head attention. By improving the efficiency of gradient computation in large language models, we hope that our work will facilitate the more effective training and deployment of long-context language models based on our theoretical results.

View arXiv page View PDF Add to collection

Community

JamesSand

Paper author Paper submitter Aug 26, 2024

Really excited to introduce this new work. It applies polynomial kernel approximation [AS23, AS24a] to solve the forward and backward computations of multi-layer transformer in almost linear time $n^{1+o(1)}$.

literate-goggles

Aug 26, 2024

Thank you for the excellent work! I find the current structure of the paper a bit challenging to follow. It would be helpful to see more concise equations that better capture the core of the approximation method. I look forward to a tutorial on this in the future.

JamesSand

Paper author Aug 26, 2024

Thank you for your interest to our work and your nice suggestions. We will try to provide more visualizations for the polynomial approximation method in the future, to make it easier to understand.